climateprediction.net home page
Task 22290511

Task 22290511

Name oifs_43r3_ps_0383_2018050100_123_987_12204027_0
Workunit 12204027
Created 24 Dec 2022, 20:30:29 UTC
Sent 8 Jan 2023, 1:05:32 UTC
Report deadline 7 Feb 2023, 1:05:32 UTC
Received 9 Jan 2023, 6:20:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1493665
Run time 16 hours 46 min 12 sec
CPU time 15 hours 14 min 58 sec
Validate state Invalid
Credit 6,168.96
Device peak FLOPS 4.47 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,523.63 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,833.36 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
eprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_53.zip
Uploading the intermediate file: upload_file_53.zip
  13:14:18 STEP 1297 H=1297:00 +CPU= 26.653
  13:14:50 STEP 1298 H=1298:00 +CPU= 27.588
  13:15:38 STEP 1299 H=1299:00 +CPU= 42.120
  13:16:09 STEP 1300 H=1300:00 +CPU= 27.895
  13:16:40 STEP 1301 H=1301:00 +CPU= 26.715
  13:17:30 STEP 1302 H=1302:00 +CPU= 42.961
  13:18:00 STEP 1303 H=1303:00 +CPU= 26.475
  13:18:32 STEP 1304 H=1304:00 +CPU= 27.419
  13:19:19 STEP 1305 H=1305:00 +CPU= 41.807
  13:19:51 STEP 1306 H=1306:00 +CPU= 27.461
  13:20:22 STEP 1307 H=1307:00 +CPU= 26.533
  13:21:15 STEP 1308 H=1308:00 +CPU= 46.466
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001308
  13:21:46 STEP 1309 H=1309:00 +CPU= 26.546
  13:22:18 STEP 1310 H=1310:00 +CPU= 27.296
  13:23:06 STEP 1311 H=1311:00 +CPU= 41.890
  13:23:38 STEP 1312 H=1312:00 +CPU= 27.324
  13:24:08 STEP 1313 H=1313:00 +CPU= 26.720
  13:24:57 STEP 1314 H=1314:00 +CPU= 42.726
  13:25:28 STEP 1315 H=1315:00 +CPU= 26.624
  13:25:59 STEP 1316 H=1316:00 +CPU= 27.715
  13:26:47 STEP 1317 H=1317:00 +CPU= 41.769
  13:27:18 STEP 1318 H=1318:00 +CPU= 27.383
  13:27:50 STEP 1319 H=1319:00 +CPU= 27.381
  13:28:47 STEP 1320 H=1320:00 +CPU= 49.406
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_54.zip
Uploading the intermediate file: upload_file_54.zip
  13:29:17 STEP 1321 H=1321:00 +CPU= 26.318
  13:29:49 STEP 1322 H=1322:00 +CPU= 27.635
  13:30:37 STEP 1323 H=1323:00 +CPU= 41.593
  13:31:08 STEP 1324 H=1324:00 +CPU= 27.201
  13:31:38 STEP 1325 H=1325:00 +CPU= 26.078
  13:32:27 STEP 1326 H=1326:00 +CPU= 42.485
  13:32:57 STEP 1327 H=1327:00 +CPU= 26.913
  13:33:29 STEP 1328 H=1328:00 +CPU= 27.287
  13:34:17 STEP 1329 H=1329:00 +CPU= 41.780
  13:34:48 STEP 1330 H=1330:00 +CPU= 27.135
  13:35:18 STEP 1331 H=1331:00 +CPU= 26.243
  13:36:11 STEP 1332 H=1332:00 +CPU= 45.997
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001332
  13:36:41 STEP 1333 H=1333:00 +CPU= 26.524
  13:37:13 STEP 1334 H=1334:00 +CPU= 27.480
  13:38:01 STEP 1335 H=1335:00 +CPU= 41.306
  13:38:32 STEP 1336 H=1336:00 +CPU= 27.146
  13:39:02 STEP 1337 H=1337:00 +CPU= 26.371
  13:39:51 STEP 1338 H=1338:00 +CPU= 42.708
  13:40:22 STEP 1339 H=1339:00 +CPU= 26.442
  13:40:53 STEP 1340 H=1340:00 +CPU= 27.571
  13:41:42 STEP 1341 H=1341:00 +CPU= 41.881
  13:42:13 STEP 1342 H=1342:00 +CPU= 27.496
  13:42:46 STEP 1343 H=1343:00 +CPU= 27.483
  13:43:44 STEP 1344 H=1344:00 +CPU= 48.957
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_55.zip
Uploading the intermediate file: upload_file_55.zip
  13:44:11 STEP 1345 H=1345:00 +CPU= 25.933
  13:44:40 STEP 1346 H=1346:00 +CPU= 26.747
  13:45:24 STEP 1347 H=1347:00 +CPU= 41.547
  13:45:52 STEP 1348 H=1348:00 +CPU= 26.829
  13:46:19 STEP 1349 H=1349:00 +CPU= 25.723
  13:47:04 STEP 1350 H=1350:00 +CPU= 42.223
  13:47:32 STEP 1351 H=1351:00 +CPU= 26.012
  13:48:00 STEP 1352 H=1352:00 +CPU= 26.967
  13:48:44 STEP 1353 H=1353:00 +CPU= 41.622
  13:49:12 STEP 1354 H=1354:00 +CPU= 26.659
  13:49:39 STEP 1355 H=1355:00 +CPU= 26.125
  13:50:28 STEP 1356 H=1356:00 +CPU= 45.702
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001356
  13:50:56 STEP 1357 H=1357:00 +CPU= 26.202
  13:51:24 STEP 1358 H=1358:00 +CPU= 26.918
  13:52:07 STEP 1359 H=1359:00 +CPU= 41.009
  13:52:36 STEP 1360 H=1360:00 +CPU= 26.895
  13:53:04 STEP 1361 H=1361:00 +CPU= 26.404
  13:53:48 STEP 1362 H=1362:00 +CPU= 42.060
  13:54:16 STEP 1363 H=1363:00 +CPU= 26.007
  13:54:45 STEP 1364 H=1364:00 +CPU= 27.269
  13:55:29 STEP 1365 H=1365:00 +CPU= 41.372
  13:55:57 STEP 1366 H=1366:00 +CPU= 27.096
  13:56:26 STEP 1367 H=1367:00 +CPU= 27.046
  13:57:17 STEP 1368 H=1368:00 +CPU= 48.132
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_56.zip
Uploading the intermediate file: upload_file_56.zip
  13:57:44 STEP 1369 H=1369:00 +CPU= 25.795
  13:58:12 STEP 1370 H=1370:00 +CPU= 26.738
  13:58:58 STEP 1371 H=1371:00 +CPU= 41.448
  13:59:26 STEP 1372 H=1372:00 +CPU= 26.965
  13:59:54 STEP 1373 H=1373:00 +CPU= 26.306
  14:00:39 STEP 1374 H=1374:00 +CPU= 42.439
  14:01:06 STEP 1375 H=1375:00 +CPU= 25.925
  14:01:34 STEP 1376 H=1376:00 +CPU= 27.019
  14:02:18 STEP 1377 H=1377:00 +CPU= 41.331
  14:02:46 STEP 1378 H=1378:00 +CPU= 26.742
  14:03:13 STEP 1379 H=1379:00 +CPU= 25.867
  14:04:02 STEP 1380 H=1380:00 +CPU= 45.989
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001380
  14:04:30 STEP 1381 H=1381:00 +CPU= 26.249
  14:04:59 STEP 1382 H=1382:00 +CPU= 27.205
  14:05:43 STEP 1383 H=1383:00 +CPU= 41.135
  14:06:11 STEP 1384 H=1384:00 +CPU= 26.682
  14:06:38 STEP 1385 H=1385:00 +CPU= 26.037
  14:07:23 STEP 1386 H=1386:00 +CPU= 42.418
  14:07:50 STEP 1387 H=1387:00 +CPU= 26.113
  14:08:19 STEP 1388 H=1388:00 +CPU= 27.054
  14:09:03 STEP 1389 H=1389:00 +CPU= 41.545
  14:09:31 STEP 1390 H=1390:00 +CPU= 27.473
  14:10:01 STEP 1391 H=1391:00 +CPU= 27.573
  14:10:52 STEP 1392 H=1392:00 +CPU= 48.477
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_57.zip
Uploading the intermediate file: upload_file_57.zip
  14:11:20 STEP 1393 H=1393:00 +CPU= 26.047
  14:11:49 STEP 1394 H=1394:00 +CPU= 27.110
  14:12:36 STEP 1395 H=1395:00 +CPU= 42.005
  14:13:08 STEP 1396 H=1396:00 +CPU= 27.376
  14:13:39 STEP 1397 H=1397:00 +CPU= 26.540
  14:14:26 STEP 1398 H=1398:00 +CPU= 42.328
  14:14:54 STEP 1399 H=1399:00 +CPU= 26.711
  14:15:27 STEP 1400 H=1400:00 +CPU= 27.771
  14:16:15 STEP 1401 H=1401:00 +CPU= 41.894
  14:16:45 STEP 1402 H=1402:00 +CPU= 27.255
  14:17:14 STEP 1403 H=1403:00 +CPU= 26.129
  14:18:05 STEP 1404 H=1404:00 +CPU= 46.012
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001404
  14:18:34 STEP 1405 H=1405:00 +CPU= 26.402
  14:19:05 STEP 1406 H=1406:00 +CPU= 27.392
  14:19:55 STEP 1407 H=1407:00 +CPU= 42.148
  14:20:27 STEP 1408 H=1408:00 +CPU= 27.498
  14:20:58 STEP 1409 H=1409:00 +CPU= 26.697
  14:21:44 STEP 1410 H=1410:00 +CPU= 42.643
  14:22:12 STEP 1411 H=1411:00 +CPU= 26.314
  14:22:40 STEP 1412 H=1412:00 +CPU= 26.883
  14:23:24 STEP 1413 H=1413:00 +CPU= 41.330
  14:23:53 STEP 1414 H=1414:00 +CPU= 27.317
  14:24:22 STEP 1415 H=1415:00 +CPU= 27.134
  14:25:13 STEP 1416 H=1416:00 +CPU= 48.551
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_58.zip
Uploading the intermediate file: upload_file_58.zip
  14:25:41 STEP 1417 H=1417:00 +CPU= 26.246
  14:26:10 STEP 1418 H=1418:00 +CPU= 27.174
  14:26:55 STEP 1419 H=1419:00 +CPU= 41.598
  14:27:23 STEP 1420 H=1420:00 +CPU= 26.943
  14:27:51 STEP 1421 H=1421:00 +CPU= 26.345
  14:28:36 STEP 1422 H=1422:00 +CPU= 42.582
  14:29:04 STEP 1423 H=1423:00 +CPU= 26.283
  14:29:33 STEP 1424 H=1424:00 +CPU= 27.266
  14:30:17 STEP 1425 H=1425:00 +CPU= 41.467
  14:30:45 STEP 1426 H=1426:00 +CPU= 27.014
  14:31:13 STEP 1427 H=1427:00 +CPU= 26.124
  14:32:02 STEP 1428 H=1428:00 +CPU= 46.060
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001428
  14:32:29 STEP 1429 H=1429:00 +CPU= 25.864
  14:32:58 STEP 1430 H=1430:00 +CPU= 26.913
  14:33:42 STEP 1431 H=1431:00 +CPU= 41.543
  14:34:11 STEP 1432 H=1432:00 +CPU= 27.184
  14:34:39 STEP 1433 H=1433:00 +CPU= 26.007
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0383
(argv4) batchid: 987
(argv5) wuid: 12204027
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/12
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5996e67eecd9bb085ef8bcb363c0348f to: /var/lib/boinc/slots/12/oifs_43r3_ps_0383_2018050100_123_987_12204027.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/12/oifs_43r3_ps_0383_2018050100_123_987_12204027.zip
ic_ancil_file: ic_ancil_12204027
ifsdata_file: ifsdata_12204027
climate_data_file: clim_data_12204027
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f9985028ebb54c9c94ecc6b9bdac601b to: /var/lib/boinc/slots/12/ic_ancil_12204027.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/12/ic_ancil_12204027.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f843fae7d1b7a9b729a9cc3a53ba4cd to: /var/lib/boinc/slots/12/ifsdata/ifsdata_12204027.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/12/ifsdata/ifsdata_12204027.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/12/159l_2/clim_data_12204027.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/12/159l_2/clim_data_12204027.zip
Checking for progress XML file: /var/lib/boinc/slots/12/progress_file_12204027.xml
Opened progress file ok : /var/lib/boinc/slots/12/progress_file_12204027.xml
-- Model is restarting --
Adjusting last_iter, 1434, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/12/progress_file_12204027.xml
last_cpu_time: 46948
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452
The child process has been launched with process id: 3847
Executing the command: /var/lib/boinc/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:3847:3847] [20230108:145245:1673214765:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:53:38 STEP 1416 H=1416:00 +CPU= 45.739
  14:54:06 STEP 1417 H=1417:00 +CPU= 26.033
  14:54:34 STEP 1418 H=1418:00 +CPU= 26.265
  14:55:16 STEP 1419 H=1419:00 +CPU= 40.230
  14:55:44 STEP 1420 H=1420:00 +CPU= 26.496
  14:56:12 STEP 1421 H=1421:00 +CPU= 25.988
  14:56:56 STEP 1422 H=1422:00 +CPU= 42.019
  14:57:23 STEP 1423 H=1423:00 +CPU= 25.703
  14:57:51 STEP 1424 H=1424:00 +CPU= 26.520
  14:58:35 STEP 1425 H=1425:00 +CPU= 40.809
  14:59:03 STEP 1426 H=1426:00 +CPU= 27.064
  14:59:31 STEP 1427 H=1427:00 +CPU= 25.754
  15:00:19 STEP 1428 H=1428:00 +CPU= 45.377
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001428
  15:00:46 STEP 1429 H=1429:00 +CPU= 25.680
  15:01:14 STEP 1430 H=1430:00 +CPU= 26.465
  15:01:57 STEP 1431 H=1431:00 +CPU= 41.119
  15:02:25 STEP 1432 H=1432:00 +CPU= 26.813
  15:02:52 STEP 1433 H=1433:00 +CPU= 25.953
  15:03:36 STEP 1434 H=1434:00 +CPU= 41.842
  15:04:10 STEP 1435 H=1435:00 +CPU= 26.526
  15:04:37 STEP 1436 H=1436:00 +CPU= 26.581
  15:05:20 STEP 1437 H=1437:00 +CPU= 40.573
  15:05:48 STEP 1438 H=1438:00 +CPU= 26.276
  15:06:16 STEP 1439 H=1439:00 +CPU= 26.781
  15:07:07 STEP 1440 H=1440:00 +CPU= 48.279
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_59.zip
Uploading the intermediate file: upload_file_59.zip
  15:07:34 STEP 1441 H=1441:00 +CPU= 25.773
  15:08:02 STEP 1442 H=1442:00 +CPU= 26.665
  15:08:45 STEP 1443 H=1443:00 +CPU= 40.765
  15:09:13 STEP 1444 H=1444:00 +CPU= 26.580
  15:09:40 STEP 1445 H=1445:00 +CPU= 25.978
  15:10:24 STEP 1446 H=1446:00 +CPU= 42.219
  15:10:51 STEP 1447 H=1447:00 +CPU= 25.529
  15:11:19 STEP 1448 H=1448:00 +CPU= 26.514
  15:12:02 STEP 1449 H=1449:00 +CPU= 40.719
  15:12:30 STEP 1450 H=1450:00 +CPU= 26.520
  15:12:57 STEP 1451 H=1451:00 +CPU= 25.747
  15:13:45 STEP 1452 H=1452:00 +CPU= 45.331
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001452
  15:14:12 STEP 1453 H=1453:00 +CPU= 25.824
  15:14:40 STEP 1454 H=1454:00 +CPU= 26.549
  15:15:23 STEP 1455 H=1455:00 +CPU= 41.266
  15:15:52 STEP 1456 H=1456:00 +CPU= 26.983
  15:16:19 STEP 1457 H=1457:00 +CPU= 25.787
  15:17:03 STEP 1458 H=1458:00 +CPU= 41.671
  15:17:30 STEP 1459 H=1459:00 +CPU= 25.492
  15:17:57 STEP 1460 H=1460:00 +CPU= 26.734
  15:18:40 STEP 1461 H=1461:00 +CPU= 40.882
  15:19:10 STEP 1462 H=1462:00 +CPU= 26.945
  15:19:42 STEP 1463 H=1463:00 +CPU= 27.401
  15:20:35 STEP 1464 H=1464:00 +CPU= 49.592
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_60.zip
Uploading the intermediate file: upload_file_60.zip
  15:21:06 STEP 1465 H=1465:00 +CPU= 26.834
  15:21:35 STEP 1466 H=1466:00 +CPU= 27.163
  15:22:19 STEP 1467 H=1467:00 +CPU= 41.902
  15:22:48 STEP 1468 H=1468:00 +CPU= 27.040
  15:23:16 STEP 1469 H=1469:00 +CPU= 26.626
  15:24:01 STEP 1470 H=1470:00 +CPU= 42.920
  15:24:30 STEP 1471 H=1471:00 +CPU= 26.678
  15:24:59 STEP 1472 H=1472:00 +CPU= 27.397
  15:25:44 STEP 1473 H=1473:00 +CPU= 41.606
  15:26:13 STEP 1474 H=1474:00 +CPU= 27.355
  15:26:41 STEP 1475 H=1475:00 +CPU= 26.357
  15:27:31 STEP 1476 H=1476:00 +CPU= 46.958
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001476
  15:27:59 STEP 1477 H=1477:00 +CPU= 26.584
  15:28:28 STEP 1478 H=1478:00 +CPU= 27.291
  15:29:13 STEP 1479 H=1479:00 +CPU= 41.533
  15:29:41 STEP 1480 H=1480:00 +CPU= 27.065
  15:30:09 STEP 1481 H=1481:00 +CPU= 26.032
  15:30:54 STEP 1482 H=1482:00 +CPU= 42.412
  15:31:22 STEP 1483 H=1483:00 +CPU= 26.364
  15:31:50 STEP 1484 H=1484:00 +CPU= 26.956
  15:32:35 STEP 1485 H=1485:00 +CPU= 42.057
  15:33:04 STEP 1486 H=1486:00 +CPU= 27.347
  15:33:34 STEP 1487 H=1487:00 +CPU= 27.436
  15:34:27 STEP 1488 H=1488:00 +CPU= 49.358
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_61.zip
Uploading the intermediate file: upload_file_61.zip
  15:34:55 STEP 1489 H=1489:00 +CPU= 26.433
  15:35:24 STEP 1490 H=1490:00 +CPU= 27.403
  15:36:08 STEP 1491 H=1491:00 +CPU= 41.497
  15:36:37 STEP 1492 H=1492:00 +CPU= 27.489
  15:37:06 STEP 1493 H=1493:00 +CPU= 26.710
  15:37:56 STEP 1494 H=1494:00 +CPU= 43.056
  15:38:24 STEP 1495 H=1495:00 +CPU= 26.404
  15:38:54 STEP 1496 H=1496:00 +CPU= 27.802
  15:39:38 STEP 1497 H=1497:00 +CPU= 42.126
  15:40:08 STEP 1498 H=1498:00 +CPU= 27.293
  15:40:39 STEP 1499 H=1499:00 +CPU= 26.815
  15:41:32 STEP 1500 H=1500:00 +CPU= 46.927
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001500
  15:42:02 STEP 1501 H=1501:00 +CPU= 26.781
  15:42:31 STEP 1502 H=1502:00 +CPU= 27.276
  15:43:16 STEP 1503 H=1503:00 +CPU= 41.950
  15:43:45 STEP 1504 H=1504:00 +CPU= 27.409
  15:44:13 STEP 1505 H=1505:00 +CPU= 26.506
  15:45:03 STEP 1506 H=1506:00 +CPU= 43.381
  15:45:34 STEP 1507 H=1507:00 +CPU= 27.087
  15:46:03 STEP 1508 H=1508:00 +CPU= 27.409
  15:46:49 STEP 1509 H=1509:00 +CPU= 41.997
  15:47:18 STEP 1510 H=1510:00 +CPU= 27.197
  15:47:46 STEP 1511 H=1511:00 +CPU= 26.885
  15:48:38 STEP 1512 H=1512:00 +CPU= 48.731
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_62.zip
Uploading the intermediate file: upload_file_62.zip
  15:49:06 STEP 1513 H=1513:00 +CPU= 26.308
  15:49:35 STEP 1514 H=1514:00 +CPU= 27.322
  15:50:20 STEP 1515 H=1515:00 +CPU= 42.107
  15:50:49 STEP 1516 H=1516:00 +CPU= 27.286
  15:51:18 STEP 1517 H=1517:00 +CPU= 26.866
  15:52:03 STEP 1518 H=1518:00 +CPU= 42.975
  15:52:31 STEP 1519 H=1519:00 +CPU= 26.325
  15:53:00 STEP 1520 H=1520:00 +CPU= 27.253
  15:53:44 STEP 1521 H=1521:00 +CPU= 41.793
  15:54:13 STEP 1522 H=1522:00 +CPU= 27.179
  15:54:41 STEP 1523 H=1523:00 +CPU= 26.175
  15:55:30 STEP 1524 H=1524:00 +CPU= 46.374
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001524
  15:55:59 STEP 1525 H=1525:00 +CPU= 26.397
  15:56:30 STEP 1526 H=1526:00 +CPU= 27.717
  15:57:15 STEP 1527 H=1527:00 +CPU= 42.267
  15:57:45 STEP 1528 H=1528:00 +CPU= 27.733
  15:58:14 STEP 1529 H=1529:00 +CPU= 26.401
  15:59:02 STEP 1530 H=1530:00 +CPU= 42.508
  15:59:30 STEP 1531 H=1531:00 +CPU= 26.194
  15:59:58 STEP 1532 H=1532:00 +CPU= 27.044
  16:00:41 STEP 1533 H=1533:00 +CPU= 41.284
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0383
(argv4) batchid: 987
(argv5) wuid: 12204027
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/12
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5996e67eecd9bb085ef8bcb363c0348f to: /var/lib/boinc/slots/12/oifs_43r3_ps_0383_2018050100_123_987_12204027.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/12/oifs_43r3_ps_0383_2018050100_123_987_12204027.zip
ic_ancil_file: ic_ancil_12204027
ifsdata_file: ifsdata_12204027
climate_data_file: clim_data_12204027
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f9985028ebb54c9c94ecc6b9bdac601b to: /var/lib/boinc/slots/12/ic_ancil_12204027.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/12/ic_ancil_12204027.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f843fae7d1b7a9b729a9cc3a53ba4cd to: /var/lib/boinc/slots/12/ifsdata/ifsdata_12204027.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/12/ifsdata/ifsdata_12204027.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/12/159l_2/clim_data_12204027.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/12/159l_2/clim_data_12204027.zip
Checking for progress XML file: /var/lib/boinc/slots/12/progress_file_12204027.xml
Opened progress file ok : /var/lib/boinc/slots/12/progress_file_12204027.xml
-- Model is restarting --
Adjusting last_iter, 1534, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/12/progress_file_12204027.xml
last_cpu_time: 50772
upload_file_number: 63
last_iter: 1511
last_upload: 5443200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452
The child process has been launched with process id: 19554
Executing the command: /var/lib/boinc/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:19554:19554] [20230108:172637:1673223997:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:27:31 STEP 1512 H=1512:00 +CPU= 45.639
  17:27:58 STEP 1513 H=1513:00 +CPU= 25.796
  17:28:25 STEP 1514 H=1514:00 +CPU= 26.392
  17:29:07 STEP 1515 H=1515:00 +CPU= 40.942
  17:29:35 STEP 1516 H=1516:00 +CPU= 26.615
  17:30:02 STEP 1517 H=1517:00 +CPU= 25.947
  17:30:45 STEP 1518 H=1518:00 +CPU= 41.770
  17:31:12 STEP 1519 H=1519:00 +CPU= 25.888
  17:31:40 STEP 1520 H=1520:00 +CPU= 26.873
  17:32:23 STEP 1521 H=1521:00 +CPU= 41.348
  17:32:51 STEP 1522 H=1522:00 +CPU= 26.696
  17:33:18 STEP 1523 H=1523:00 +CPU= 26.110
  17:34:07 STEP 1524 H=1524:00 +CPU= 46.124
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001524
  17:34:34 STEP 1525 H=1525:00 +CPU= 25.814
  17:35:02 STEP 1526 H=1526:00 +CPU= 26.799
  17:35:44 STEP 1527 H=1527:00 +CPU= 40.814
  17:36:12 STEP 1528 H=1528:00 +CPU= 27.080
  17:36:39 STEP 1529 H=1529:00 +CPU= 25.685
  17:37:22 STEP 1530 H=1530:00 +CPU= 41.724
  17:37:49 STEP 1531 H=1531:00 +CPU= 26.001
  17:38:18 STEP 1532 H=1532:00 +CPU= 26.968
  17:39:01 STEP 1533 H=1533:00 +CPU= 41.214
  17:39:29 STEP 1534 H=1534:00 +CPU= 27.345
  17:39:57 STEP 1535 H=1535:00 +CPU= 26.734
  17:40:47 STEP 1536 H=1536:00 +CPU= 47.554
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_63.zip
Uploading the intermediate file: upload_file_63.zip
  17:41:14 STEP 1537 H=1537:00 +CPU= 25.628
  17:41:41 STEP 1538 H=1538:00 +CPU= 26.541
  17:42:28 STEP 1539 H=1539:00 +CPU= 41.155
  17:42:59 STEP 1540 H=1540:00 +CPU= 26.798
  17:43:28 STEP 1541 H=1541:00 +CPU= 26.106
  17:44:24 STEP 1542 H=1542:00 +CPU= 42.458
  17:44:57 STEP 1543 H=1543:00 +CPU= 26.078
  17:46:03 STEP 1544 H=1544:00 +CPU= 27.690
  17:46:50 STEP 1545 H=1545:00 +CPU= 41.043
  17:47:26 STEP 1546 H=1546:00 +CPU= 26.792
  17:48:28 STEP 1547 H=1547:00 +CPU= 27.264
  17:50:00 STEP 1548 H=1548:00 +CPU= 47.500
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001548
  17:50:36 STEP 1549 H=1549:00 +CPU= 26.302
  17:51:04 STEP 1550 H=1550:00 +CPU= 26.572
  17:51:46 STEP 1551 H=1551:00 +CPU= 40.214
  17:52:15 STEP 1552 H=1552:00 +CPU= 26.516
  17:52:43 STEP 1553 H=1553:00 +CPU= 25.616
  17:53:29 STEP 1554 H=1554:00 +CPU= 41.212
  17:53:58 STEP 1555 H=1555:00 +CPU= 26.054
  17:54:27 STEP 1556 H=1556:00 +CPU= 26.124
  17:55:11 STEP 1557 H=1557:00 +CPU= 40.665
  17:55:40 STEP 1558 H=1558:00 +CPU= 26.205
  17:56:43 STEP 1559 H=1559:00 +CPU= 27.155
  17:58:17 STEP 1560 H=1560:00 +CPU= 47.492
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_64.zip
Uploading the intermediate file: upload_file_64.zip
  17:59:07 STEP 1561 H=1561:00 +CPU= 26.385
  18:00:03 STEP 1562 H=1562:00 +CPU= 27.283
  18:00:52 STEP 1563 H=1563:00 +CPU= 39.478
  18:01:17 STEP 1564 H=1564:00 +CPU= 24.798
  18:01:42 STEP 1565 H=1565:00 +CPU= 23.786
  18:02:22 STEP 1566 H=1566:00 +CPU= 38.588
  18:02:46 STEP 1567 H=1567:00 +CPU= 23.542
  18:03:11 STEP 1568 H=1568:00 +CPU= 24.532
  18:03:50 STEP 1569 H=1569:00 +CPU= 37.520
  18:04:15 STEP 1570 H=1570:00 +CPU= 24.670
  18:04:40 STEP 1571 H=1571:00 +CPU= 23.621
  18:05:23 STEP 1572 H=1572:00 +CPU= 42.041
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001572
  18:05:48 STEP 1573 H=1573:00 +CPU= 23.721
  18:06:13 STEP 1574 H=1574:00 +CPU= 24.581
  18:06:52 STEP 1575 H=1575:00 +CPU= 37.715
  18:07:17 STEP 1576 H=1576:00 +CPU= 24.508
  18:07:42 STEP 1577 H=1577:00 +CPU= 23.582
  18:08:22 STEP 1578 H=1578:00 +CPU= 38.524
  18:08:46 STEP 1579 H=1579:00 +CPU= 23.549
  18:09:11 STEP 1580 H=1580:00 +CPU= 24.455
  18:09:50 STEP 1581 H=1581:00 +CPU= 37.473
  18:10:15 STEP 1582 H=1582:00 +CPU= 24.343
  18:10:40 STEP 1583 H=1583:00 +CPU= 24.501
  18:11:25 STEP 1584 H=1584:00 +CPU= 43.516
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_65.zip
Uploading the intermediate file: upload_file_65.zip
  18:11:50 STEP 1585 H=1585:00 +CPU= 23.707
  18:12:15 STEP 1586 H=1586:00 +CPU= 24.465
  18:12:54 STEP 1587 H=1587:00 +CPU= 37.680
  18:13:19 STEP 1588 H=1588:00 +CPU= 24.505
  18:13:43 STEP 1589 H=1589:00 +CPU= 23.793
  18:14:23 STEP 1590 H=1590:00 +CPU= 38.582
  18:14:47 STEP 1591 H=1591:00 +CPU= 23.682
  18:15:12 STEP 1592 H=1592:00 +CPU= 24.328
  18:15:51 STEP 1593 H=1593:00 +CPU= 37.529
  18:16:16 STEP 1594 H=1594:00 +CPU= 24.418
  18:16:40 STEP 1595 H=1595:00 +CPU= 23.550
  18:17:23 STEP 1596 H=1596:00 +CPU= 41.982
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001596
  18:17:47 STEP 1597 H=1597:00 +CPU= 23.463
  18:18:12 STEP 1598 H=1598:00 +CPU= 24.455
  18:18:51 STEP 1599 H=1599:00 +CPU= 37.454
  18:19:16 STEP 1600 H=1600:00 +CPU= 24.352
  18:19:40 STEP 1601 H=1601:00 +CPU= 23.447
  18:20:20 STEP 1602 H=1602:00 +CPU= 38.353
  18:20:44 STEP 1603 H=1603:00 +CPU= 23.596
  18:21:09 STEP 1604 H=1604:00 +CPU= 24.311
  18:21:47 STEP 1605 H=1605:00 +CPU= 37.524
  18:22:12 STEP 1606 H=1606:00 +CPU= 24.346
  18:22:37 STEP 1607 H=1607:00 +CPU= 24.342
  18:23:22 STEP 1608 H=1608:00 +CPU= 43.618
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_66.zip
Uploading the intermediate file: upload_file_66.zip
  18:23:46 STEP 1609 H=1609:00 +CPU= 23.601
  18:24:11 STEP 1610 H=1610:00 +CPU= 24.417
  18:24:49 STEP 1611 H=1611:00 +CPU= 37.226
  18:25:15 STEP 1612 H=1612:00 +CPU= 24.493
  18:25:39 STEP 1613 H=1613:00 +CPU= 23.663
  18:26:19 STEP 1614 H=1614:00 +CPU= 38.695
  18:26:43 STEP 1615 H=1615:00 +CPU= 23.677
  18:27:08 STEP 1616 H=1616:00 +CPU= 24.506
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0383
(argv4) batchid: 987
(argv5) wuid: 12204027
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/12
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5996e67eecd9bb085ef8bcb363c0348f to: /var/lib/boinc/slots/12/oifs_43r3_ps_0383_2018050100_123_987_12204027.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/12/oifs_43r3_ps_0383_2018050100_123_987_12204027.zip
ic_ancil_file: ic_ancil_12204027
ifsdata_file: ifsdata_12204027
climate_data_file: clim_data_12204027
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f9985028ebb54c9c94ecc6b9bdac601b to: /var/lib/boinc/slots/12/ic_ancil_12204027.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/12/ic_ancil_12204027.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f843fae7d1b7a9b729a9cc3a53ba4cd to: /var/lib/boinc/slots/12/ifsdata/ifsdata_12204027.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/12/ifsdata/ifsdata_12204027.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/12/159l_2/clim_data_12204027.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/12/159l_2/clim_data_12204027.zip
Checking for progress XML file: /var/lib/boinc/slots/12/progress_file_12204027.xml
Opened progress file ok : /var/lib/boinc/slots/12/progress_file_12204027.xml
-- Model is restarting --
Adjusting last_iter, 1616, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/12/progress_file_12204027.xml
last_cpu_time: 53966
upload_file_number: 67
last_iter: 1607
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452
The child process has been launched with process id: 12628
Executing the command: /var/lib/boinc/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:12628:12628] [20230108:220419:1673240659:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:05:07 STEP 1608 H=1608:00 +CPU= 44.639
  22:05:34 STEP 1609 H=1609:00 +CPU= 25.625
  22:06:01 STEP 1610 H=1610:00 +CPU= 26.026
  22:06:44 STEP 1611 H=1611:00 +CPU= 40.992
  22:07:11 STEP 1612 H=1612:00 +CPU= 26.678
  22:07:38 STEP 1613 H=1613:00 +CPU= 25.484
  22:08:21 STEP 1614 H=1614:00 +CPU= 41.957
  22:08:47 STEP 1615 H=1615:00 +CPU= 25.293
  22:09:14 STEP 1616 H=1616:00 +CPU= 26.052
  22:09:56 STEP 1617 H=1617:00 +CPU= 41.191
  22:10:23 STEP 1618 H=1618:00 +CPU= 26.335
  22:10:49 STEP 1619 H=1619:00 +CPU= 25.171
  22:11:36 STEP 1620 H=1620:00 +CPU= 45.095
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001620
  22:12:03 STEP 1621 H=1621:00 +CPU= 25.692
  22:12:30 STEP 1622 H=1622:00 +CPU= 26.231
  22:13:12 STEP 1623 H=1623:00 +CPU= 40.926
  22:13:39 STEP 1624 H=1624:00 +CPU= 26.124
  22:14:05 STEP 1625 H=1625:00 +CPU= 25.202
  22:14:48 STEP 1626 H=1626:00 +CPU= 41.657
  22:15:14 STEP 1627 H=1627:00 +CPU= 25.219
  22:15:41 STEP 1628 H=1628:00 +CPU= 26.041
  22:16:23 STEP 1629 H=1629:00 +CPU= 40.809
  22:16:51 STEP 1630 H=1630:00 +CPU= 26.160
  22:17:18 STEP 1631 H=1631:00 +CPU= 26.490
  22:18:07 STEP 1632 H=1632:00 +CPU= 47.400
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12204027/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0383_2018050100_123_987_12204027_0_r2062453452_67.zip
Uploading the intermediate file: upload_file_67.zip
  22:18:34 STEP 1633 H=1633:00 +CPU= 25.315
  22:19:01 STEP 1634 H=1634:00 +CPU= 26.505
  22:19:44 STEP 1635 H=1635:00 +CPU= 41.285
  22:20:11 STEP 1636 H=1636:00 +CPU= 26.436
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:13:00 1493665 22290511 oifs_43r3_ps_0383_2018050100_123_987_12204027_0 4,406,400 40,180 0.0091
09 May 2023 16:12:55 1493665 22290511 oifs_43r3_ps_0383_2018050100_123_987_12204027_0 3,542,400 32,608 0.0092
09 May 2023 16:12:21 1493665 22290511 oifs_43r3_ps_0383_2018050100_123_987_12204027_0 864,000 7,763 0.0090


©2024 climateprediction.net