climateprediction.net home page
Task 22305870

Task 22305870

Name oifs_43r3_ps_0404_1999050100_123_968_12185048_1
Workunit 12185048
Created 5 Feb 2023, 18:44:09 UTC
Sent 5 Feb 2023, 18:45:26 UTC
Report deadline 6 Apr 2023, 18:45:26 UTC
Received 27 Feb 2023, 12:39:51 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1509518
Run time 13 hours 57 min 37 sec
CPU time 6 hours 2 min 15 sec
Validate state Invalid
Credit 7,136.64
Device peak FLOPS 4.20 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.80 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,294.97 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
46
  01:35:57 STEP 1266 H=1266:00 +CPU= 30.972
  01:36:16 STEP 1267 H=1267:00 +CPU= 19.257
  01:36:36 STEP 1268 H=1268:00 +CPU= 19.607
  01:37:06 STEP 1269 H=1269:00 +CPU= 29.736
  01:37:25 STEP 1270 H=1270:00 +CPU= 19.744
  01:37:45 STEP 1271 H=1271:00 +CPU= 19.665
  01:38:20 STEP 1272 H=1272:00 +CPU= 35.245
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_22.zip
Uploading the intermediate file: upload_file_22.zip
  01:38:41 STEP 1273 H=1273:00 +CPU= 20.420
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:39:29 STEP 1274 H=1274:00 +CPU= 25.915
  01:40:04 STEP 1275 H=1275:00 +CPU= 34.145
  01:40:25 STEP 1276 H=1276:00 +CPU= 20.653
  01:40:49 STEP 1277 H=1277:00 +CPU= 23.499
  01:41:30 STEP 1278 H=1278:00 +CPU= 38.739
  01:41:56 STEP 1279 H=1279:00 +CPU= 25.558
  01:42:23 STEP 1280 H=1280:00 +CPU= 25.876
  01:42:59 STEP 1281 H=1281:00 +CPU= 35.138
  01:43:21 STEP 1282 H=1282:00 +CPU= 21.586
  01:43:41 STEP 1283 H=1283:00 +CPU= 20.098
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:44:39 STEP 1284 H=1284:00 +CPU= 43.307
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001284
  01:45:04 STEP 1285 H=1285:00 +CPU= 24.412
  01:45:32 STEP 1286 H=1286:00 +CPU= 26.655
  01:46:07 STEP 1287 H=1287:00 +CPU= 34.006
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:47:31 STEP 1288 H=1288:00 +CPU= 29.419
  01:47:54 STEP 1289 H=1289:00 +CPU= 22.777
  01:48:30 STEP 1290 H=1290:00 +CPU= 34.782
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:49:18 STEP 1291 H=1291:00 +CPU= 25.254
  01:49:39 STEP 1292 H=1292:00 +CPU= 21.142
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:50:29 STEP 1293 H=1293:00 +CPU= 37.524
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:51:05 STEP 1294 H=1294:00 +CPU= 24.624
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:52:30 STEP 1295 H=1295:00 +CPU= 29.938
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:53:30 STEP 1296 H=1296:00 +CPU= 38.172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_23.zip
Uploading the intermediate file: upload_file_23.zip
  01:53:49 STEP 1297 H=1297:00 +CPU= 19.226
  01:54:09 STEP 1298 H=1298:00 +CPU= 19.767
  01:54:39 STEP 1299 H=1299:00 +CPU= 29.862
  01:54:59 STEP 1300 H=1300:00 +CPU= 19.697
  01:55:18 STEP 1301 H=1301:00 +CPU= 19.179
  01:55:48 STEP 1302 H=1302:00 +CPU= 30.497
  01:56:08 STEP 1303 H=1303:00 +CPU= 19.113
  01:56:27 STEP 1304 H=1304:00 +CPU= 19.721
  01:56:57 STEP 1305 H=1305:00 +CPU= 29.823
  01:57:17 STEP 1306 H=1306:00 +CPU= 19.751
  01:57:36 STEP 1307 H=1307:00 +CPU= 19.158
  01:58:09 STEP 1308 H=1308:00 +CPU= 33.189
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001308
  01:58:28 STEP 1309 H=1309:00 +CPU= 19.202
  01:58:48 STEP 1310 H=1310:00 +CPU= 19.865
  01:59:18 STEP 1311 H=1311:00 +CPU= 29.759
  01:59:38 STEP 1312 H=1312:00 +CPU= 19.698
  01:59:57 STEP 1313 H=1313:00 +CPU= 19.092
  02:00:27 STEP 1314 H=1314:00 +CPU= 30.402
  02:00:46 STEP 1315 H=1315:00 +CPU= 19.087
  02:01:06 STEP 1316 H=1316:00 +CPU= 19.649
  02:01:36 STEP 1317 H=1317:00 +CPU= 29.770
  02:01:55 STEP 1318 H=1318:00 +CPU= 19.614
  02:02:15 STEP 1319 H=1319:00 +CPU= 19.728
  02:02:51 STEP 1320 H=1320:00 +CPU= 35.331
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_24.zip
Uploading the intermediate file: upload_file_24.zip
  02:03:10 STEP 1321 H=1321:00 +CPU= 19.203
  02:03:29 STEP 1322 H=1322:00 +CPU= 19.709
  02:03:59 STEP 1323 H=1323:00 +CPU= 29.791
  02:04:19 STEP 1324 H=1324:00 +CPU= 19.695
  02:04:38 STEP 1325 H=1325:00 +CPU= 19.090
  02:05:09 STEP 1326 H=1326:00 +CPU= 30.483
  02:05:28 STEP 1327 H=1327:00 +CPU= 19.181
  02:05:47 STEP 1328 H=1328:00 +CPU= 19.715
  02:06:17 STEP 1329 H=1329:00 +CPU= 29.803
  02:06:37 STEP 1330 H=1330:00 +CPU= 19.710
  02:06:56 STEP 1331 H=1331:00 +CPU= 19.145
  02:07:29 STEP 1332 H=1332:00 +CPU= 33.151
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001332
  02:07:48 STEP 1333 H=1333:00 +CPU= 19.135
  02:08:08 STEP 1334 H=1334:00 +CPU= 19.701
  02:08:38 STEP 1335 H=1335:00 +CPU= 29.889
  02:08:58 STEP 1336 H=1336:00 +CPU= 19.681
  02:09:17 STEP 1337 H=1337:00 +CPU= 19.107
  02:09:47 STEP 1338 H=1338:00 +CPU= 30.432
  02:10:06 STEP 1339 H=1339:00 +CPU= 19.151
  02:10:26 STEP 1340 H=1340:00 +CPU= 19.724
  02:10:56 STEP 1341 H=1341:00 +CPU= 29.787
  02:11:16 STEP 1342 H=1342:00 +CPU= 19.698
  02:11:35 STEP 1343 H=1343:00 +CPU= 19.681
  02:12:11 STEP 1344 H=1344:00 +CPU= 35.239
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_25.zip
Uploading the intermediate file: upload_file_25.zip
  02:12:30 STEP 1345 H=1345:00 +CPU= 19.186
  02:12:49 STEP 1346 H=1346:00 +CPU= 19.690
  02:13:19 STEP 1347 H=1347:00 +CPU= 29.795
  02:13:39 STEP 1348 H=1348:00 +CPU= 19.976
  02:13:58 STEP 1349 H=1349:00 +CPU= 19.144
  02:14:29 STEP 1350 H=1350:00 +CPU= 30.397
  02:14:48 STEP 1351 H=1351:00 +CPU= 19.115
  02:15:08 STEP 1352 H=1352:00 +CPU= 19.693
  02:15:37 STEP 1353 H=1353:00 +CPU= 29.779
  02:15:57 STEP 1354 H=1354:00 +CPU= 19.674
  02:16:16 STEP 1355 H=1355:00 +CPU= 19.176
  02:16:49 STEP 1356 H=1356:00 +CPU= 33.156
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001356
  02:17:09 STEP 1357 H=1357:00 +CPU= 19.136
  02:17:28 STEP 1358 H=1358:00 +CPU= 19.690
  02:17:58 STEP 1359 H=1359:00 +CPU= 29.789
  02:18:18 STEP 1360 H=1360:00 +CPU= 19.751
  02:18:37 STEP 1361 H=1361:00 +CPU= 19.235
  02:19:08 STEP 1362 H=1362:00 +CPU= 30.421
  02:19:27 STEP 1363 H=1363:00 +CPU= 19.104
  02:19:46 STEP 1364 H=1364:00 +CPU= 19.693
  02:20:16 STEP 1365 H=1365:00 +CPU= 29.756
  02:20:36 STEP 1366 H=1366:00 +CPU= 19.663
  02:20:56 STEP 1367 H=1367:00 +CPU= 19.765
  02:21:31 STEP 1368 H=1368:00 +CPU= 35.231
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_26.zip
Uploading the intermediate file: upload_file_26.zip
  02:21:50 STEP 1369 H=1369:00 +CPU= 19.214
  02:22:10 STEP 1370 H=1370:00 +CPU= 19.726
  02:22:40 STEP 1371 H=1371:00 +CPU= 29.789
  02:22:59 STEP 1372 H=1372:00 +CPU= 19.697
  02:23:18 STEP 1373 H=1373:00 +CPU= 19.212
  02:23:52 STEP 1374 H=1374:00 +CPU= 33.122
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:24:25 STEP 1375 H=1375:00 +CPU= 21.930
  02:24:51 STEP 1376 H=1376:00 +CPU= 24.719
  02:25:26 STEP 1377 H=1377:00 +CPU= 34.325
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:26:26 STEP 1378 H=1378:00 +CPU= 26.117
  02:26:50 STEP 1379 H=1379:00 +CPU= 23.078
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:27:55 STEP 1380 H=1380:00 +CPU= 42.291
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001380
  02:28:18 STEP 1381 H=1381:00 +CPU= 22.341
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:28:57 STEP 1382 H=1382:00 +CPU= 27.051
  02:29:31 STEP 1383 H=1383:00 +CPU= 33.303
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:30:08 STEP 1384 H=1384:00 +CPU= 24.946
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:31:37 STEP 1385 H=1385:00 +CPU= 26.240
  02:32:12 STEP 1386 H=1386:00 +CPU= 33.709
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:32:46 STEP 1387 H=1387:00 +CPU= 22.661
  02:33:08 STEP 1388 H=1388:00 +CPU= 21.321
  02:33:40 STEP 1389 H=1389:00 +CPU= 31.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:34:14 STEP 1390 H=1390:00 +CPU= 22.787
  02:34:37 STEP 1391 H=1391:00 +CPU= 22.194
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:35:28 STEP 1392 H=1392:00 +CPU= 39.773
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_27.zip
Uploading the intermediate file: upload_file_27.zip
  02:35:53 STEP 1393 H=1393:00 +CPU= 23.730
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:36:40 STEP 1394 H=1394:00 +CPU= 24.271
  02:37:11 STEP 1395 H=1395:00 +CPU= 30.935
  02:37:31 STEP 1396 H=1396:00 +CPU= 19.833
  02:37:53 STEP 1397 H=1397:00 +CPU= 21.625
  02:38:26 STEP 1398 H=1398:00 +CPU= 32.229
  02:38:47 STEP 1399 H=1399:00 +CPU= 20.544
  02:39:09 STEP 1400 H=1400:00 +CPU= 21.450
  02:39:39 STEP 1401 H=1401:00 +CPU= 30.021
  02:40:00 STEP 1402 H=1402:00 +CPU= 20.735
  02:40:20 STEP 1403 H=1403:00 +CPU= 20.238
  02:40:57 STEP 1404 H=1404:00 +CPU= 36.093
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001404
  02:41:21 STEP 1405 H=1405:00 +CPU= 23.570
  02:41:44 STEP 1406 H=1406:00 +CPU= 22.428
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:42:42 STEP 1407 H=1407:00 +CPU= 34.973
  02:43:04 STEP 1408 H=1408:00 +CPU= 21.575
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:43:42 STEP 1409 H=1409:00 +CPU= 25.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:44:43 STEP 1410 H=1410:00 +CPU= 38.511
  02:45:05 STEP 1411 H=1411:00 +CPU= 21.550
  02:45:28 STEP 1412 H=1412:00 +CPU= 22.226
  02:46:02 STEP 1413 H=1413:00 +CPU= 33.594
  02:46:27 STEP 1414 H=1414:00 +CPU= 24.089
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:47:14 STEP 1415 H=1415:00 +CPU= 23.815
  02:47:53 STEP 1416 H=1416:00 +CPU= 38.805
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_28.zip
Uploading the intermediate file: upload_file_28.zip
  02:48:14 STEP 1417 H=1417:00 +CPU= 20.662
Uploading trickle at timestep: 5097600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:48:53 STEP 1418 H=1418:00 +CPU= 26.312
  02:49:24 STEP 1419 H=1419:00 +CPU= 31.052
  02:49:48 STEP 1420 H=1420:00 +CPU= 22.797
  02:50:09 STEP 1421 H=1421:00 +CPU= 21.122
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:50:57 STEP 1422 H=1422:00 +CPU= 35.969
  02:51:19 STEP 1423 H=1423:00 +CPU= 21.538
  02:51:42 STEP 1424 H=1424:00 +CPU= 21.854
  02:52:14 STEP 1425 H=1425:00 +CPU= 31.598
  02:52:35 STEP 1426 H=1426:00 +CPU= 21.533
  02:53:01 STEP 1427 H=1427:00 +CPU= 24.566
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:53:52 STEP 1428 H=1428:00 +CPU= 38.867
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001428
  02:54:16 STEP 1429 H=1429:00 +CPU= 22.742
  02:54:40 STEP 1430 H=1430:00 +CPU= 23.869
  02:55:13 STEP 1431 H=1431:00 +CPU= 31.959
  02:55:33 STEP 1432 H=1432:00 +CPU= 20.631
  02:55:58 STEP 1433 H=1433:00 +CPU= 23.476
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:56:49 STEP 1434 H=1434:00 +CPU= 37.859
  02:57:15 STEP 1435 H=1435:00 +CPU= 24.690
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:58:25 STEP 1436 H=1436:00 +CPU= 27.359
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:59:20 STEP 1437 H=1437:00 +CPU= 41.079
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:00:09 STEP 1438 H=1438:00 +CPU= 26.939
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:01:00 STEP 1439 H=1439:00 +CPU= 27.297
  03:01:41 STEP 1440 H=1440:00 +CPU= 40.235
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_29.zip
Uploading the intermediate file: upload_file_29.zip
  03:02:02 STEP 1441 H=1441:00 +CPU= 20.581
  03:02:23 STEP 1442 H=1442:00 +CPU= 20.801
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:03:16 STEP 1443 H=1443:00 +CPU= 39.704
  03:03:42 STEP 1444 H=1444:00 +CPU= 25.224
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030359:1677323039:36338.782] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030400:1677323040:36339.784] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1920:1920] [20230225:030400:1677323040:36339.784] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001038431  cloudsc_                  727  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F1A790  callpar_                 1223  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0404
(argv4) batchid: 968
(argv5) wuid: 12185048
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_225b339456f5bf3a9c69f53378c00e4f to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0404_1999050100_123_968_12185048.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0404_1999050100_123_968_12185048.zip
ic_ancil_file: ic_ancil_12185048
ifsdata_file: ifsdata_12185048
climate_data_file: clim_data_12185048
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b1bd23a97400b970c5860cd01c824e8d to: /var/lib/boinc-client/slots/1/ic_ancil_12185048.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12185048.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_684e9d374803bdcac30fb7de32ac0032 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185048.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185048.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185048.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185048.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12185048.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12185048.xml
-- Model is restarting --
Adjusting last_iter, 1445, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12185048.xml
last_cpu_time: 19790
upload_file_number: 30
last_iter: 1439
last_upload: 5184000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857
The child process has been launched with process id: 1902
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230225:185002:1677379802:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:50:42 STEP 1440 H=1440:00 +CPU= 34.216
  02:51:01 STEP 1441 H=1441:00 +CPU= 19.067
  02:51:24 STEP 1442 H=1442:00 +CPU= 21.675
  02:51:54 STEP 1443 H=1443:00 +CPU= 30.500
  02:52:14 STEP 1444 H=1444:00 +CPU= 19.760
  02:52:33 STEP 1445 H=1445:00 +CPU= 19.066
  02:53:09 STEP 1446 H=1446:00 +CPU= 35.550
  02:53:37 STEP 1447 H=1447:00 +CPU= 27.381
  02:54:00 STEP 1448 H=1448:00 +CPU= 22.783
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:57:06 STEP 1449 H=1449:00 +CPU= 49.037
  02:57:36 STEP 1450 H=1450:00 +CPU= 28.463
  02:58:05 STEP 1451 H=1451:00 +CPU= 27.516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:59:20 STEP 1452 H=1452:00 +CPU= 49.815
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001452
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:00:53 STEP 1453 H=1453:00 +CPU= 35.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:02:03 STEP 1454 H=1454:00 +CPU= 26.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:03:00 STEP 1455 H=1455:00 +CPU= 35.238
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:03:57 STEP 1456 H=1456:00 +CPU= 25.064
  03:04:25 STEP 1457 H=1457:00 +CPU= 26.342
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:05:37 STEP 1458 H=1458:00 +CPU= 38.658
  03:05:58 STEP 1459 H=1459:00 +CPU= 21.072
  03:06:19 STEP 1460 H=1460:00 +CPU= 20.402
  03:06:49 STEP 1461 H=1461:00 +CPU= 30.127
  03:07:09 STEP 1462 H=1462:00 +CPU= 20.152
  03:07:29 STEP 1463 H=1463:00 +CPU= 19.999
  03:08:07 STEP 1464 H=1464:00 +CPU= 36.843
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_30.zip
Uploading the intermediate file: upload_file_30.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:08:55 STEP 1465 H=1465:00 +CPU= 27.689
  03:09:20 STEP 1466 H=1466:00 +CPU= 23.775
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:10:23 STEP 1467 H=1467:00 +CPU= 40.231
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:11:01 STEP 1468 H=1468:00 +CPU= 25.124
  03:11:25 STEP 1469 H=1469:00 +CPU= 23.239
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:13:13 STEP 1470 H=1470:00 +CPU= 42.625
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:13:53 STEP 1471 H=1471:00 +CPU= 27.431
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:14:44 STEP 1472 H=1472:00 +CPU= 28.078
  03:15:23 STEP 1473 H=1473:00 +CPU= 38.005
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:16:15 STEP 1474 H=1474:00 +CPU= 28.098
  03:16:35 STEP 1475 H=1475:00 +CPU= 19.955
  03:17:10 STEP 1476 H=1476:00 +CPU= 34.761
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001476
  03:17:30 STEP 1477 H=1477:00 +CPU= 19.749
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:18:05 STEP 1478 H=1478:00 +CPU= 23.611
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.688] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.688] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031826:1677410306:30503.689] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  CUADJTQ 
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031827:1677410307:30504.689] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1902:1902] [20230226:031827:1677410307:30504.689] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CBB804  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FBCB79  cuadjtq_                  299  cuadjtq.F90
oifs_43r3_model.e  00000000010255EE  cloudsc_                 1522  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F10C1B  callpar_                 1011  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0404
(argv4) batchid: 968
(argv5) wuid: 12185048
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_225b339456f5bf3a9c69f53378c00e4f to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0404_1999050100_123_968_12185048.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0404_1999050100_123_968_12185048.zip
ic_ancil_file: ic_ancil_12185048
ifsdata_file: ifsdata_12185048
climate_data_file: clim_data_12185048
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b1bd23a97400b970c5860cd01c824e8d to: /var/lib/boinc-client/slots/1/ic_ancil_12185048.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12185048.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_684e9d374803bdcac30fb7de32ac0032 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185048.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185048.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185048.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185048.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12185048.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12185048.xml
-- Model is restarting --
Adjusting last_iter, 1479, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12185048.xml
last_cpu_time: 20928
upload_file_number: 31
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857
The child process has been launched with process id: 1897
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1897:1897] [20230226:193438:1677468878:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:37:16 STEP 1464 H=1464:00 +CPU= 45.459
  03:37:46 STEP 1465 H=1465:00 +CPU= 28.325
  03:38:09 STEP 1466 H=1466:00 +CPU= 22.425
  03:38:46 STEP 1467 H=1467:00 +CPU= 35.896
  03:39:11 STEP 1468 H=1468:00 +CPU= 23.990
  03:39:34 STEP 1469 H=1469:00 +CPU= 22.065
  03:40:07 STEP 1470 H=1470:00 +CPU= 32.720
  03:40:27 STEP 1471 H=1471:00 +CPU= 20.380
  03:40:49 STEP 1472 H=1472:00 +CPU= 21.125
  03:41:25 STEP 1473 H=1473:00 +CPU= 35.373
  03:41:47 STEP 1474 H=1474:00 +CPU= 21.177
  03:42:08 STEP 1475 H=1475:00 +CPU= 20.988
  03:42:47 STEP 1476 H=1476:00 +CPU= 37.493
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001476
  03:43:06 STEP 1477 H=1477:00 +CPU= 19.469
  03:43:26 STEP 1478 H=1478:00 +CPU= 19.755
  03:43:57 STEP 1479 H=1479:00 +CPU= 31.066
  03:44:17 STEP 1480 H=1480:00 +CPU= 19.818
  03:44:37 STEP 1481 H=1481:00 +CPU= 19.350
  03:45:09 STEP 1482 H=1482:00 +CPU= 31.708
  03:45:28 STEP 1483 H=1483:00 +CPU= 19.588
  03:45:52 STEP 1484 H=1484:00 +CPU= 23.454
  03:46:24 STEP 1485 H=1485:00 +CPU= 31.095
  03:46:43 STEP 1486 H=1486:00 +CPU= 19.690
  03:47:03 STEP 1487 H=1487:00 +CPU= 19.726
  03:47:39 STEP 1488 H=1488:00 +CPU= 35.207
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185048/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0404_1999050100_123_968_12185048_1_r578666857_31.zip
Uploading the intermediate file: upload_file_31.zip
  03:47:58 STEP 1489 H=1489:00 +CPU= 19.152
  03:48:17 STEP 1490 H=1490:00 +CPU= 19.701
  03:48:47 STEP 1491 H=1491:00 +CPU= 29.876
  03:49:07 STEP 1492 H=1492:00 +CPU= 19.715
  03:49:26 STEP 1493 H=1493:00 +CPU= 19.149
  03:49:57 STEP 1494 H=1494:00 +CPU= 30.579
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:53:32 1509518 22305870 oifs_43r3_ps_0404_1999050100_123_968_12185048_1 5,097,600 19,011 0.0037
09 May 2023 16:52:33 1509518 22305870 oifs_43r3_ps_0404_1999050100_123_968_12185048_1 4,147,200 11,714 0.0028
09 May 2023 16:48:25 1509518 22305870 oifs_43r3_ps_0404_1999050100_123_968_12185048_1 1,728,000 13,171 0.0076
09 May 2023 16:48:23 1509518 22305870 oifs_43r3_ps_0404_1999050100_123_968_12185048_1 864,000 6,617 0.0077


©2024 cpdn.org