climateprediction.net home page
Task 22266205

Task 22266205

Name oifs_43r3_ps_0539_1995050100_123_964_12181183_0
Workunit 12181183
Created 22 Dec 2022, 8:37:28 UTC
Sent 29 Dec 2022, 16:54:48 UTC
Report deadline 28 Jan 2023, 16:54:48 UTC
Received 1 Jan 2023, 22:45:52 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1537562
Run time 10 hours 54 min 56 sec
CPU time 10 hours 11 min 21 sec
Validate state Invalid
Credit 6,531.84
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,798.91 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,905.35 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
jects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001404
  14:38:34 STEP 1405 H=1405:00 +CPU= 15.754
  14:38:51 STEP 1406 H=1406:00 +CPU= 16.206
  14:39:15 STEP 1407 H=1407:00 +CPU= 23.987
  14:39:31 STEP 1408 H=1408:00 +CPU= 16.394
  14:39:48 STEP 1409 H=1409:00 +CPU= 15.923
  14:40:12 STEP 1410 H=1410:00 +CPU= 24.628
  14:40:28 STEP 1411 H=1411:00 +CPU= 15.726
  14:40:45 STEP 1412 H=1412:00 +CPU= 16.293
  14:41:09 STEP 1413 H=1413:00 +CPU= 24.251
  14:41:25 STEP 1414 H=1414:00 +CPU= 16.211
  14:41:42 STEP 1415 H=1415:00 +CPU= 16.154
  14:42:10 STEP 1416 H=1416:00 +CPU= 28.147
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_58.zip
Uploading the intermediate file: upload_file_58.zip
  14:42:26 STEP 1417 H=1417:00 +CPU= 15.464
  14:42:42 STEP 1418 H=1418:00 +CPU= 16.053
  14:43:07 STEP 1419 H=1419:00 +CPU= 24.178
  14:43:23 STEP 1420 H=1420:00 +CPU= 16.296
  14:43:39 STEP 1421 H=1421:00 +CPU= 15.906
  14:44:04 STEP 1422 H=1422:00 +CPU= 24.539
  14:44:20 STEP 1423 H=1423:00 +CPU= 15.696
  14:44:36 STEP 1424 H=1424:00 +CPU= 16.264
  14:45:01 STEP 1425 H=1425:00 +CPU= 23.981
  14:45:17 STEP 1426 H=1426:00 +CPU= 16.195
  14:45:33 STEP 1427 H=1427:00 +CPU= 15.474
  14:46:00 STEP 1428 H=1428:00 +CPU= 26.892
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001428
  14:46:15 STEP 1429 H=1429:00 +CPU= 15.464
  14:46:32 STEP 1430 H=1430:00 +CPU= 16.221
  14:46:56 STEP 1431 H=1431:00 +CPU= 23.597
  14:47:12 STEP 1432 H=1432:00 +CPU= 16.009
  14:47:27 STEP 1433 H=1433:00 +CPU= 15.275
  14:47:52 STEP 1434 H=1434:00 +CPU= 24.423
  14:48:07 STEP 1435 H=1435:00 +CPU= 15.439
  14:48:23 STEP 1436 H=1436:00 +CPU= 16.038
  14:48:47 STEP 1437 H=1437:00 +CPU= 23.645
  14:49:03 STEP 1438 H=1438:00 +CPU= 15.704
  14:49:19 STEP 1439 H=1439:00 +CPU= 15.817
  14:49:47 STEP 1440 H=1440:00 +CPU= 27.155
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_59.zip
Uploading the intermediate file: upload_file_59.zip
  14:50:02 STEP 1441 H=1441:00 +CPU= 15.168
  14:50:18 STEP 1442 H=1442:00 +CPU= 16.010
  14:50:43 STEP 1443 H=1443:00 +CPU= 24.058
  14:50:59 STEP 1444 H=1444:00 +CPU= 15.864
  14:51:14 STEP 1445 H=1445:00 +CPU= 15.214
  14:51:39 STEP 1446 H=1446:00 +CPU= 24.759
  14:51:55 STEP 1447 H=1447:00 +CPU= 15.486
  14:52:11 STEP 1448 H=1448:00 +CPU= 15.810
  14:52:35 STEP 1449 H=1449:00 +CPU= 23.858
  14:52:51 STEP 1450 H=1450:00 +CPU= 15.821
  14:53:07 STEP 1451 H=1451:00 +CPU= 15.436
  14:53:33 STEP 1452 H=1452:00 +CPU= 26.613
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001452
  14:53:49 STEP 1453 H=1453:00 +CPU= 15.186
  14:54:05 STEP 1454 H=1454:00 +CPU= 15.863
  14:54:29 STEP 1455 H=1455:00 +CPU= 23.785
  14:54:45 STEP 1456 H=1456:00 +CPU= 15.609
  14:55:00 STEP 1457 H=1457:00 +CPU= 15.406
  14:55:25 STEP 1458 H=1458:00 +CPU= 24.424
  14:55:40 STEP 1459 H=1459:00 +CPU= 15.090
  14:55:56 STEP 1460 H=1460:00 +CPU= 15.781
  14:56:20 STEP 1461 H=1461:00 +CPU= 23.761
  14:56:36 STEP 1462 H=1462:00 +CPU= 15.855
  14:56:52 STEP 1463 H=1463:00 +CPU= 15.930
  14:57:20 STEP 1464 H=1464:00 +CPU= 27.071
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_60.zip
Uploading the intermediate file: upload_file_60.zip
  14:57:35 STEP 1465 H=1465:00 +CPU= 15.168
  14:57:51 STEP 1466 H=1466:00 +CPU= 16.017
  14:58:16 STEP 1467 H=1467:00 +CPU= 23.777
  14:58:32 STEP 1468 H=1468:00 +CPU= 16.207
  14:58:47 STEP 1469 H=1469:00 +CPU= 15.449
  14:59:12 STEP 1470 H=1470:00 +CPU= 24.583
  14:59:27 STEP 1471 H=1471:00 +CPU= 15.107
  14:59:43 STEP 1472 H=1472:00 +CPU= 15.702
  15:00:07 STEP 1473 H=1473:00 +CPU= 23.707
  15:00:23 STEP 1474 H=1474:00 +CPU= 15.899
  15:00:39 STEP 1475 H=1475:00 +CPU= 15.537
  15:01:06 STEP 1476 H=1476:00 +CPU= 26.627
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001476
  15:01:21 STEP 1477 H=1477:00 +CPU= 15.083
  15:01:37 STEP 1478 H=1478:00 +CPU= 16.210
  15:02:02 STEP 1479 H=1479:00 +CPU= 24.093
  15:02:17 STEP 1480 H=1480:00 +CPU= 15.768
  15:02:33 STEP 1481 H=1481:00 +CPU= 15.349
  15:02:58 STEP 1482 H=1482:00 +CPU= 24.619
  15:03:13 STEP 1483 H=1483:00 +CPU= 15.252
  15:03:29 STEP 1484 H=1484:00 +CPU= 15.979
  15:03:54 STEP 1485 H=1485:00 +CPU= 24.182
  15:04:10 STEP 1486 H=1486:00 +CPU= 15.706
  15:04:25 STEP 1487 H=1487:00 +CPU= 15.704
  15:04:54 STEP 1488 H=1488:00 +CPU= 27.997
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_61.zip
Uploading the intermediate file: upload_file_61.zip
  15:05:09 STEP 1489 H=1489:00 +CPU= 15.392
  15:05:26 STEP 1490 H=1490:00 +CPU= 16.036
  15:05:50 STEP 1491 H=1491:00 +CPU= 23.543
  15:06:06 STEP 1492 H=1492:00 +CPU= 15.937
  15:06:21 STEP 1493 H=1493:00 +CPU= 15.014
  15:06:46 STEP 1494 H=1494:00 +CPU= 24.599
  15:07:01 STEP 1495 H=1495:00 +CPU= 15.498
  15:07:17 STEP 1496 H=1496:00 +CPU= 15.605
  15:07:41 STEP 1497 H=1497:00 +CPU= 23.745
  15:07:57 STEP 1498 H=1498:00 +CPU= 15.969
  15:08:12 STEP 1499 H=1499:00 +CPU= 15.274
  15:08:39 STEP 1500 H=1500:00 +CPU= 26.718
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001500
  15:08:55 STEP 1501 H=1501:00 +CPU= 15.191
  15:09:10 STEP 1502 H=1502:00 +CPU= 15.762
  15:09:35 STEP 1503 H=1503:00 +CPU= 23.880
  15:09:50 STEP 1504 H=1504:00 +CPU= 15.799
  15:10:06 STEP 1505 H=1505:00 +CPU= 15.391
  15:10:31 STEP 1506 H=1506:00 +CPU= 24.527
  15:10:46 STEP 1507 H=1507:00 +CPU= 15.185
  15:11:02 STEP 1508 H=1508:00 +CPU= 16.040
  15:11:26 STEP 1509 H=1509:00 +CPU= 23.831
  15:11:42 STEP 1510 H=1510:00 +CPU= 15.642
  15:11:58 STEP 1511 H=1511:00 +CPU= 15.907
  15:12:26 STEP 1512 H=1512:00 +CPU= 27.856
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_62.zip
Uploading the intermediate file: upload_file_62.zip
  15:12:42 STEP 1513 H=1513:00 +CPU= 15.120
  15:12:58 STEP 1514 H=1514:00 +CPU= 16.084
  15:13:22 STEP 1515 H=1515:00 +CPU= 23.835
  15:13:38 STEP 1516 H=1516:00 +CPU= 15.943
  15:13:54 STEP 1517 H=1517:00 +CPU= 15.164
  15:14:18 STEP 1518 H=1518:00 +CPU= 24.285
  15:14:33 STEP 1519 H=1519:00 +CPU= 15.030
  15:14:49 STEP 1520 H=1520:00 +CPU= 15.675
  15:15:13 STEP 1521 H=1521:00 +CPU= 23.773
  15:15:29 STEP 1522 H=1522:00 +CPU= 15.758
  15:15:45 STEP 1523 H=1523:00 +CPU= 15.347
  15:16:12 STEP 1524 H=1524:00 +CPU= 26.817
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001524
  15:16:27 STEP 1525 H=1525:00 +CPU= 15.155
  15:16:43 STEP 1526 H=1526:00 +CPU= 15.739
  15:17:07 STEP 1527 H=1527:00 +CPU= 23.694
  15:17:23 STEP 1528 H=1528:00 +CPU= 15.701
  15:17:38 STEP 1529 H=1529:00 +CPU= 15.017
  15:18:02 STEP 1530 H=1530:00 +CPU= 24.313
  15:18:18 STEP 1531 H=1531:00 +CPU= 15.245
  15:18:34 STEP 1532 H=1532:00 +CPU= 15.860
  15:18:58 STEP 1533 H=1533:00 +CPU= 23.796
  15:19:14 STEP 1534 H=1534:00 +CPU= 15.157
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0539
(argv4) batchid: 964
(argv5) wuid: 12181183
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/18
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d825fcf6092e0da42ddbbad1787c06ac to: /var/lib/boinc/slots/18/oifs_43r3_ps_0539_1995050100_123_964_12181183.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_0539_1995050100_123_964_12181183.zip
ic_ancil_file: ic_ancil_12181183
ifsdata_file: ifsdata_12181183
climate_data_file: clim_data_12181183
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0cf06e8a7b3d411b4e094ac834b9b9ff to: /var/lib/boinc/slots/18/ic_ancil_12181183.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/18/ic_ancil_12181183.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_762f5ead2c02e92fb96d9732f4aaad73 to: /var/lib/boinc/slots/18/ifsdata/ifsdata_12181183.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/18/ifsdata/ifsdata_12181183.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/18/159l_2/clim_data_12181183.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/18/159l_2/clim_data_12181183.zip
Checking for progress XML file: /var/lib/boinc/slots/18/progress_file_12181183.xml
Opened progress file ok : /var/lib/boinc/slots/18/progress_file_12181183.xml
-- Model is restarting --
Adjusting last_iter, 1535, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/18/progress_file_12181183.xml
last_cpu_time: 32176
upload_file_number: 63
last_iter: 1511
last_upload: 5443200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613
The child process has been launched with process id: 8787
Executing the command: /var/lib/boinc/slots/18/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8787:8787] [20230101:170831:1672592911:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:08:59 STEP 1512 H=1512:00 +CPU= 25.731
  17:09:14 STEP 1513 H=1513:00 +CPU= 14.876
  17:09:29 STEP 1514 H=1514:00 +CPU= 15.406
  17:09:53 STEP 1515 H=1515:00 +CPU= 22.974
  17:10:09 STEP 1516 H=1516:00 +CPU= 15.238
  17:10:24 STEP 1517 H=1517:00 +CPU= 14.594
  17:10:48 STEP 1518 H=1518:00 +CPU= 23.424
  17:11:03 STEP 1519 H=1519:00 +CPU= 14.757
  17:11:18 STEP 1520 H=1520:00 +CPU= 15.104
  17:11:41 STEP 1521 H=1521:00 +CPU= 22.665
  17:11:57 STEP 1522 H=1522:00 +CPU= 15.002
  17:12:12 STEP 1523 H=1523:00 +CPU= 14.617
  17:12:38 STEP 1524 H=1524:00 +CPU= 25.703
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001524
  17:12:53 STEP 1525 H=1525:00 +CPU= 14.645
  17:13:09 STEP 1526 H=1526:00 +CPU= 15.157
  17:13:32 STEP 1527 H=1527:00 +CPU= 22.945
  17:13:48 STEP 1528 H=1528:00 +CPU= 15.239
  17:14:03 STEP 1529 H=1529:00 +CPU= 14.502
  17:14:27 STEP 1530 H=1530:00 +CPU= 23.381
  17:14:42 STEP 1531 H=1531:00 +CPU= 14.438
  17:14:57 STEP 1532 H=1532:00 +CPU= 15.065
  17:15:20 STEP 1533 H=1533:00 +CPU= 22.718
  17:15:35 STEP 1534 H=1534:00 +CPU= 14.705
  17:15:51 STEP 1535 H=1535:00 +CPU= 14.811
  17:16:20 STEP 1536 H=1536:00 +CPU= 26.632
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_63.zip
Uploading the intermediate file: upload_file_63.zip
  17:16:35 STEP 1537 H=1537:00 +CPU= 14.578
  17:16:51 STEP 1538 H=1538:00 +CPU= 15.016
  17:17:14 STEP 1539 H=1539:00 +CPU= 22.907
  17:17:30 STEP 1540 H=1540:00 +CPU= 15.094
  17:17:45 STEP 1541 H=1541:00 +CPU= 14.634
  17:18:09 STEP 1542 H=1542:00 +CPU= 23.547
  17:18:24 STEP 1543 H=1543:00 +CPU= 14.507
  17:18:40 STEP 1544 H=1544:00 +CPU= 15.111
  17:19:03 STEP 1545 H=1545:00 +CPU= 22.807
  17:19:18 STEP 1546 H=1546:00 +CPU= 15.139
  17:19:33 STEP 1547 H=1547:00 +CPU= 14.576
  17:20:00 STEP 1548 H=1548:00 +CPU= 25.607
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001548
  17:20:15 STEP 1549 H=1549:00 +CPU= 14.575
  17:20:29 STEP 1550 H=1550:00 +CPU= 14.217
  17:20:51 STEP 1551 H=1551:00 +CPU= 21.691
  17:21:06 STEP 1552 H=1552:00 +CPU= 14.587
  17:21:21 STEP 1553 H=1553:00 +CPU= 14.399
  17:21:45 STEP 1554 H=1554:00 +CPU= 22.987
  17:21:59 STEP 1555 H=1555:00 +CPU= 14.299
  17:22:14 STEP 1556 H=1556:00 +CPU= 14.822
  17:22:37 STEP 1557 H=1557:00 +CPU= 22.496
  17:22:52 STEP 1558 H=1558:00 +CPU= 14.596
  17:23:07 STEP 1559 H=1559:00 +CPU= 14.725
  17:23:35 STEP 1560 H=1560:00 +CPU= 26.516
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_64.zip
Uploading the intermediate file: upload_file_64.zip
  17:23:50 STEP 1561 H=1561:00 +CPU= 14.329
  17:24:05 STEP 1562 H=1562:00 +CPU= 14.844
  17:24:28 STEP 1563 H=1563:00 +CPU= 22.363
  17:24:43 STEP 1564 H=1564:00 +CPU= 14.820
  17:24:58 STEP 1565 H=1565:00 +CPU= 14.275
  17:25:21 STEP 1566 H=1566:00 +CPU= 23.147
  17:25:36 STEP 1567 H=1567:00 +CPU= 14.392
  17:25:51 STEP 1568 H=1568:00 +CPU= 14.710
  17:26:14 STEP 1569 H=1569:00 +CPU= 22.599
  17:26:30 STEP 1570 H=1570:00 +CPU= 14.913
  17:26:44 STEP 1571 H=1571:00 +CPU= 14.333
  17:27:10 STEP 1572 H=1572:00 +CPU= 25.428
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001572
  17:27:25 STEP 1573 H=1573:00 +CPU= 14.243
  17:27:40 STEP 1574 H=1574:00 +CPU= 14.870
  17:28:03 STEP 1575 H=1575:00 +CPU= 22.502
  17:28:18 STEP 1576 H=1576:00 +CPU= 14.631
  17:28:33 STEP 1577 H=1577:00 +CPU= 14.383
  17:28:56 STEP 1578 H=1578:00 +CPU= 23.070
  17:29:10 STEP 1579 H=1579:00 +CPU= 14.192
  17:29:25 STEP 1580 H=1580:00 +CPU= 14.874
  17:29:48 STEP 1581 H=1581:00 +CPU= 22.270
  17:30:03 STEP 1582 H=1582:00 +CPU= 14.793
  17:30:18 STEP 1583 H=1583:00 +CPU= 14.562
  17:30:44 STEP 1584 H=1584:00 +CPU= 26.337
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_65.zip
Uploading the intermediate file: upload_file_65.zip
  17:30:58 STEP 1585 H=1585:00 +CPU= 14.063
  17:31:14 STEP 1586 H=1586:00 +CPU= 14.856
  17:31:36 STEP 1587 H=1587:00 +CPU= 22.290
  17:31:51 STEP 1588 H=1588:00 +CPU= 14.919
  17:32:05 STEP 1589 H=1589:00 +CPU= 14.244
  17:32:29 STEP 1590 H=1590:00 +CPU= 23.098
  17:32:43 STEP 1591 H=1591:00 +CPU= 14.367
  17:32:58 STEP 1592 H=1592:00 +CPU= 14.855
  17:33:21 STEP 1593 H=1593:00 +CPU= 22.345
  17:33:36 STEP 1594 H=1594:00 +CPU= 14.859
  17:33:51 STEP 1595 H=1595:00 +CPU= 14.172
  17:34:17 STEP 1596 H=1596:00 +CPU= 25.296
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001596
  17:34:32 STEP 1597 H=1597:00 +CPU= 14.388
  17:34:47 STEP 1598 H=1598:00 +CPU= 14.900
  17:35:09 STEP 1599 H=1599:00 +CPU= 22.318
  17:35:25 STEP 1600 H=1600:00 +CPU= 15.107
  17:35:39 STEP 1601 H=1601:00 +CPU= 14.374
  17:36:02 STEP 1602 H=1602:00 +CPU= 23.111
  17:36:17 STEP 1603 H=1603:00 +CPU= 14.439
  17:36:32 STEP 1604 H=1604:00 +CPU= 14.800
  17:36:55 STEP 1605 H=1605:00 +CPU= 22.342
  17:37:10 STEP 1606 H=1606:00 +CPU= 14.648
  17:37:25 STEP 1607 H=1607:00 +CPU= 14.747
  17:37:52 STEP 1608 H=1608:00 +CPU= 26.179
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_66.zip
Uploading the intermediate file: upload_file_66.zip
  17:38:06 STEP 1609 H=1609:00 +CPU= 14.345
  17:38:21 STEP 1610 H=1610:00 +CPU= 14.921
  17:38:44 STEP 1611 H=1611:00 +CPU= 22.502
  17:38:59 STEP 1612 H=1612:00 +CPU= 14.741
  17:39:13 STEP 1613 H=1613:00 +CPU= 14.360
  17:39:36 STEP 1614 H=1614:00 +CPU= 23.021
  17:39:51 STEP 1615 H=1615:00 +CPU= 14.297
  17:40:06 STEP 1616 H=1616:00 +CPU= 14.835
  17:40:28 STEP 1617 H=1617:00 +CPU= 22.305
  17:40:43 STEP 1618 H=1618:00 +CPU= 14.893
  17:40:57 STEP 1619 H=1619:00 +CPU= 14.257
  17:41:23 STEP 1620 H=1620:00 +CPU= 24.959
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001620
  17:41:37 STEP 1621 H=1621:00 +CPU= 14.397
  17:41:52 STEP 1622 H=1622:00 +CPU= 14.942
  17:42:15 STEP 1623 H=1623:00 +CPU= 22.512
  17:42:30 STEP 1624 H=1624:00 +CPU= 14.861
  17:42:44 STEP 1625 H=1625:00 +CPU= 14.328
  17:43:07 STEP 1626 H=1626:00 +CPU= 23.110
  17:43:22 STEP 1627 H=1627:00 +CPU= 14.094
  17:43:36 STEP 1628 H=1628:00 +CPU= 14.801
  17:43:59 STEP 1629 H=1629:00 +CPU= 22.295
  17:44:14 STEP 1630 H=1630:00 +CPU= 14.588
  17:44:28 STEP 1631 H=1631:00 +CPU= 14.442
  17:44:59 STEP 1632 H=1632:00 +CPU= 25.743
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0539
(argv4) batchid: 964
(argv5) wuid: 12181183
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/18
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d825fcf6092e0da42ddbbad1787c06ac to: /var/lib/boinc/slots/18/oifs_43r3_ps_0539_1995050100_123_964_12181183.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_0539_1995050100_123_964_12181183.zip
ic_ancil_file: ic_ancil_12181183
ifsdata_file: ifsdata_12181183
climate_data_file: clim_data_12181183
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0cf06e8a7b3d411b4e094ac834b9b9ff to: /var/lib/boinc/slots/18/ic_ancil_12181183.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/18/ic_ancil_12181183.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_762f5ead2c02e92fb96d9732f4aaad73 to: /var/lib/boinc/slots/18/ifsdata/ifsdata_12181183.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/18/ifsdata/ifsdata_12181183.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/18/159l_2/clim_data_12181183.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/18/159l_2/clim_data_12181183.zip
Checking for progress XML file: /var/lib/boinc/slots/18/progress_file_12181183.xml
Opened progress file ok : /var/lib/boinc/slots/18/progress_file_12181183.xml
-- Model is restarting --
Adjusting last_iter, 1632, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/18/progress_file_12181183.xml
last_cpu_time: 34312
upload_file_number: 67
last_iter: 1631
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613
The child process has been launched with process id: 11505
Executing the command: /var/lib/boinc/slots/18/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11505:11505] [20230101:190453:1672599893:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:05:20 STEP 1632 H=1632:00 +CPU= 25.158
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_67.zip
Uploading the intermediate file: upload_file_67.zip
  19:05:35 STEP 1633 H=1633:00 +CPU= 14.394
  19:05:50 STEP 1634 H=1634:00 +CPU= 14.870
  19:06:13 STEP 1635 H=1635:00 +CPU= 22.678
  19:06:28 STEP 1636 H=1636:00 +CPU= 14.811
  19:06:42 STEP 1637 H=1637:00 +CPU= 14.117
  19:07:05 STEP 1638 H=1638:00 +CPU= 22.862
  19:07:20 STEP 1639 H=1639:00 +CPU= 14.449
  19:07:35 STEP 1640 H=1640:00 +CPU= 14.765
  19:07:58 STEP 1641 H=1641:00 +CPU= 22.674
  19:08:14 STEP 1642 H=1642:00 +CPU= 14.840
  19:08:29 STEP 1643 H=1643:00 +CPU= 14.587
  19:08:56 STEP 1644 H=1644:00 +CPU= 25.523
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001644
  19:09:17 STEP 1645 H=1645:00 +CPU= 14.501
  19:09:34 STEP 1646 H=1646:00 +CPU= 15.184
  19:10:00 STEP 1647 H=1647:00 +CPU= 22.952
  19:10:15 STEP 1648 H=1648:00 +CPU= 15.018
  19:10:31 STEP 1649 H=1649:00 +CPU= 14.514
  19:10:55 STEP 1650 H=1650:00 +CPU= 23.382
  19:11:10 STEP 1651 H=1651:00 +CPU= 14.142
  19:11:25 STEP 1652 H=1652:00 +CPU= 14.817
  19:11:48 STEP 1653 H=1653:00 +CPU= 22.398
  19:12:03 STEP 1654 H=1654:00 +CPU= 14.877
  19:12:18 STEP 1655 H=1655:00 +CPU= 14.671
  19:12:47 STEP 1656 H=1656:00 +CPU= 26.409
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_68.zip
Uploading the intermediate file: upload_file_68.zip
  19:13:01 STEP 1657 H=1657:00 +CPU= 14.243
  19:13:16 STEP 1658 H=1658:00 +CPU= 14.989
  19:13:40 STEP 1659 H=1659:00 +CPU= 22.698
  19:13:55 STEP 1660 H=1660:00 +CPU= 14.845
  19:14:10 STEP 1661 H=1661:00 +CPU= 14.341
  19:14:35 STEP 1662 H=1662:00 +CPU= 23.350
  19:14:49 STEP 1663 H=1663:00 +CPU= 14.394
  19:15:06 STEP 1664 H=1664:00 +CPU= 14.965
  19:15:29 STEP 1665 H=1665:00 +CPU= 22.823
  19:15:44 STEP 1666 H=1666:00 +CPU= 14.936
  19:15:59 STEP 1667 H=1667:00 +CPU= 14.409
  19:16:25 STEP 1668 H=1668:00 +CPU= 25.404
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001668
  19:16:40 STEP 1669 H=1669:00 +CPU= 14.398
  19:16:55 STEP 1670 H=1670:00 +CPU= 14.883
  19:17:18 STEP 1671 H=1671:00 +CPU= 22.611
  19:17:33 STEP 1672 H=1672:00 +CPU= 14.987
  19:17:48 STEP 1673 H=1673:00 +CPU= 14.379
  19:18:12 STEP 1674 H=1674:00 +CPU= 23.240
  19:18:29 STEP 1675 H=1675:00 +CPU= 14.042
  19:18:57 STEP 1676 H=1676:00 +CPU= 14.781
  19:19:38 STEP 1677 H=1677:00 +CPU= 22.475
  19:19:53 STEP 1678 H=1678:00 +CPU= 14.831
  19:20:08 STEP 1679 H=1679:00 +CPU= 14.854
  19:20:39 STEP 1680 H=1680:00 +CPU= 26.742
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_69.zip
Uploading the intermediate file: upload_file_69.zip
  19:21:12 STEP 1681 H=1681:00 +CPU= 14.435
  19:21:27 STEP 1682 H=1682:00 +CPU= 14.730
  19:21:50 STEP 1683 H=1683:00 +CPU= 22.545
  19:22:07 STEP 1684 H=1684:00 +CPU= 14.881
  19:22:22 STEP 1685 H=1685:00 +CPU= 14.023
  19:22:48 STEP 1686 H=1686:00 +CPU= 22.829
  19:23:05 STEP 1687 H=1687:00 +CPU= 14.156
  19:23:21 STEP 1688 H=1688:00 +CPU= 13.750
  19:23:48 STEP 1689 H=1689:00 +CPU= 22.072
  19:24:05 STEP 1690 H=1690:00 +CPU= 14.538
  19:24:21 STEP 1691 H=1691:00 +CPU= 14.076
  19:24:50 STEP 1692 H=1692:00 +CPU= 24.639
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001692
  19:25:06 STEP 1693 H=1693:00 +CPU= 13.654
  19:25:24 STEP 1694 H=1694:00 +CPU= 14.014
  19:25:50 STEP 1695 H=1695:00 +CPU= 22.051
  19:26:06 STEP 1696 H=1696:00 +CPU= 14.259
  19:26:23 STEP 1697 H=1697:00 +CPU= 13.975
  19:26:47 STEP 1698 H=1698:00 +CPU= 23.122
  19:27:03 STEP 1699 H=1699:00 +CPU= 13.944
  19:27:19 STEP 1700 H=1700:00 +CPU= 13.738
  19:27:44 STEP 1701 H=1701:00 +CPU= 21.929
  19:27:59 STEP 1702 H=1702:00 +CPU= 14.757
  19:28:15 STEP 1703 H=1703:00 +CPU= 14.681
  19:28:45 STEP 1704 H=1704:00 +CPU= 26.578
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_70.zip
Uploading the intermediate file: upload_file_70.zip
  19:28:59 STEP 1705 H=1705:00 +CPU= 14.343
  19:29:14 STEP 1706 H=1706:00 +CPU= 14.480
  19:29:38 STEP 1707 H=1707:00 +CPU= 22.634
  19:29:57 STEP 1708 H=1708:00 +CPU= 14.829
  19:30:12 STEP 1709 H=1709:00 +CPU= 14.317
  19:30:38 STEP 1710 H=1710:00 +CPU= 22.986
  19:30:53 STEP 1711 H=1711:00 +CPU= 14.158
  19:31:08 STEP 1712 H=1712:00 +CPU= 14.876
  19:31:38 STEP 1713 H=1713:00 +CPU= 22.161
  19:31:54 STEP 1714 H=1714:00 +CPU= 15.085
  19:32:09 STEP 1715 H=1715:00 +CPU= 14.328
  19:32:36 STEP 1716 H=1716:00 +CPU= 25.480
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001716
  19:33:05 STEP 1717 H=1717:00 +CPU= 14.232
  19:33:21 STEP 1718 H=1718:00 +CPU= 14.877
  19:33:46 STEP 1719 H=1719:00 +CPU= 22.689
  19:34:03 STEP 1720 H=1720:00 +CPU= 14.571
  19:34:21 STEP 1721 H=1721:00 +CPU= 14.085
  19:34:47 STEP 1722 H=1722:00 +CPU= 23.192
  19:35:03 STEP 1723 H=1723:00 +CPU= 14.025
  19:35:29 STEP 1724 H=1724:00 +CPU= 14.822
  19:35:55 STEP 1725 H=1725:00 +CPU= 22.322
  19:36:09 STEP 1726 H=1726:00 +CPU= 14.436
  19:36:25 STEP 1727 H=1727:00 +CPU= 14.812
  19:37:17 STEP 1728 H=1728:00 +CPU= 26.458
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_71.zip
Uploading the intermediate file: upload_file_71.zip
  19:37:33 STEP 1729 H=1729:00 +CPU= 13.854
  19:37:48 STEP 1730 H=1730:00 +CPU= 14.768
  19:38:23 STEP 1731 H=1731:00 +CPU= 22.525
  19:38:40 STEP 1732 H=1732:00 +CPU= 14.385
  19:38:55 STEP 1733 H=1733:00 +CPU= 14.252
  19:39:23 STEP 1734 H=1734:00 +CPU= 23.512
  19:40:12 STEP 1735 H=1735:00 +CPU= 14.111
  19:40:37 STEP 1736 H=1736:00 +CPU= 15.025
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0539
(argv4) batchid: 964
(argv5) wuid: 12181183
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/18
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d825fcf6092e0da42ddbbad1787c06ac to: /var/lib/boinc/slots/18/oifs_43r3_ps_0539_1995050100_123_964_12181183.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_0539_1995050100_123_964_12181183.zip
ic_ancil_file: ic_ancil_12181183
ifsdata_file: ifsdata_12181183
climate_data_file: clim_data_12181183
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0cf06e8a7b3d411b4e094ac834b9b9ff to: /var/lib/boinc/slots/18/ic_ancil_12181183.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/18/ic_ancil_12181183.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_762f5ead2c02e92fb96d9732f4aaad73 to: /var/lib/boinc/slots/18/ifsdata/ifsdata_12181183.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/18/ifsdata/ifsdata_12181183.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/18/159l_2/clim_data_12181183.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/18/159l_2/clim_data_12181183.zip
Checking for progress XML file: /var/lib/boinc/slots/18/progress_file_12181183.xml
Opened progress file ok : /var/lib/boinc/slots/18/progress_file_12181183.xml
-- Model is restarting --
Adjusting last_iter, 1737, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/18/progress_file_12181183.xml
last_cpu_time: 36161
upload_file_number: 72
last_iter: 1727
last_upload: 6220800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613
The child process has been launched with process id: 15638
Executing the command: /var/lib/boinc/slots/18/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15638:15638] [20230101:213626:1672608986:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:37:20 STEP 1728 H=1728:00 +CPU= 25.626
  21:37:35 STEP 1729 H=1729:00 +CPU= 14.898
  21:37:50 STEP 1730 H=1730:00 +CPU= 15.095
  21:38:14 STEP 1731 H=1731:00 +CPU= 23.396
  21:38:29 STEP 1732 H=1732:00 +CPU= 15.428
  21:38:44 STEP 1733 H=1733:00 +CPU= 14.927
  21:39:09 STEP 1734 H=1734:00 +CPU= 24.070
  21:39:24 STEP 1735 H=1735:00 +CPU= 14.715
  21:39:39 STEP 1736 H=1736:00 +CPU= 15.024
  21:40:02 STEP 1737 H=1737:00 +CPU= 23.295
  21:40:17 STEP 1738 H=1738:00 +CPU= 15.160
  21:40:32 STEP 1739 H=1739:00 +CPU= 14.512
  21:40:58 STEP 1740 H=1740:00 +CPU= 25.959
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001740
  21:41:13 STEP 1741 H=1741:00 +CPU= 14.569
  21:41:28 STEP 1742 H=1742:00 +CPU= 15.278
  21:41:51 STEP 1743 H=1743:00 +CPU= 22.907
  21:42:07 STEP 1744 H=1744:00 +CPU= 15.215
  21:42:21 STEP 1745 H=1745:00 +CPU= 14.643
  21:42:45 STEP 1746 H=1746:00 +CPU= 23.684
  21:43:00 STEP 1747 H=1747:00 +CPU= 14.724
  21:43:15 STEP 1748 H=1748:00 +CPU= 15.241
  21:43:38 STEP 1749 H=1749:00 +CPU= 22.989
  21:43:54 STEP 1750 H=1750:00 +CPU= 15.270
  21:44:09 STEP 1751 H=1751:00 +CPU= 15.350
  21:44:37 STEP 1752 H=1752:00 +CPU= 27.199
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12181183/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0539_1995050100_123_964_12181183_0_r109453613_72.zip
Uploading the intermediate file: upload_file_72.zip
  21:44:52 STEP 1753 H=1753:00 +CPU= 14.756
  21:45:07 STEP 1754 H=1754:00 +CPU= 15.225
  21:45:30 STEP 1755 H=1755:00 +CPU= 22.724
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:05:07 1537562 22266205 oifs_43r3_ps_0539_1995050100_123_964_12181183_0 4,665,600 27,479 0.0059
09 May 2023 16:03:03 1537562 22266205 oifs_43r3_ps_0539_1995050100_123_964_12181183_0 3,801,600 22,813 0.0060
09 May 2023 16:02:00 1537562 22266205 oifs_43r3_ps_0539_1995050100_123_964_12181183_0 864,000 4,945 0.0057


©2024 cpdn.org