climateprediction.net home page
Task 22297954

Task 22297954

Name oifs_43r3_ps_0405_2010050100_123_979_12196049_1
Workunit 12196049
Created 14 Jan 2023, 20:20:55 UTC
Sent 14 Jan 2023, 20:30:32 UTC
Report deadline 13 Feb 2023, 20:30:32 UTC
Received 17 Jan 2023, 1:43:35 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1467890
Run time 17 hours 46 min 37 sec
CPU time 16 hours 46 min 25 sec
Validate state Invalid
Credit 6,289.92
Device peak FLOPS 4.08 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.60 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,675.79 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:236392:236392] [20230116:225330:1673906010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:236392:236392] [20230116:225330:1673906010:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:54:35 STEP 1464 H=1464:00 +CPU= 53.973
  22:55:00 STEP 1465 H=1465:00 +CPU= 24.851
  22:55:25 STEP 1466 H=1466:00 +CPU= 25.122
  22:56:04 STEP 1467 H=1467:00 +CPU= 38.990
  22:56:32 STEP 1468 H=1468:00 +CPU= 27.459
  22:57:01 STEP 1469 H=1469:00 +CPU= 29.696
  22:57:45 STEP 1470 H=1470:00 +CPU= 43.582
  22:58:15 STEP 1471 H=1471:00 +CPU= 30.190
  22:58:42 STEP 1472 H=1472:00 +CPU= 26.238
  22:59:23 STEP 1473 H=1473:00 +CPU= 41.847
  22:59:54 STEP 1474 H=1474:00 +CPU= 30.855
  23:00:19 STEP 1475 H=1475:00 +CPU= 24.866
  23:01:10 STEP 1476 H=1476:00 +CPU= 50.064
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001476
  23:01:39 STEP 1477 H=1477:00 +CPU= 29.539
  23:02:10 STEP 1478 H=1478:00 +CPU= 30.740
  23:02:56 STEP 1479 H=1479:00 +CPU= 46.313
  23:03:22 STEP 1480 H=1480:00 +CPU= 25.243
  23:03:50 STEP 1481 H=1481:00 +CPU= 28.762
  23:04:30 STEP 1482 H=1482:00 +CPU= 39.744
  23:04:58 STEP 1483 H=1483:00 +CPU= 27.262
  23:05:29 STEP 1484 H=1484:00 +CPU= 30.908
  23:06:16 STEP 1485 H=1485:00 +CPU= 47.432
  23:06:47 STEP 1486 H=1486:00 +CPU= 30.886
  23:07:17 STEP 1487 H=1487:00 +CPU= 30.265
  23:08:03 STEP 1488 H=1488:00 +CPU= 45.072
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_61.zip
Uploading the intermediate file: upload_file_61.zip
  23:08:27 STEP 1489 H=1489:00 +CPU= 23.769
  23:08:52 STEP 1490 H=1490:00 +CPU= 25.208
  23:09:40 STEP 1491 H=1491:00 +CPU= 48.053
  23:10:11 STEP 1492 H=1492:00 +CPU= 30.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:11:01 STEP 1493 H=1493:00 +CPU= 28.016
  23:11:40 STEP 1494 H=1494:00 +CPU= 39.360
  23:12:06 STEP 1495 H=1495:00 +CPU= 26.062
  23:12:37 STEP 1496 H=1496:00 +CPU= 30.775
  23:13:15 STEP 1497 H=1497:00 +CPU= 38.153
  23:13:47 STEP 1498 H=1498:00 +CPU= 31.339
  23:14:15 STEP 1499 H=1499:00 +CPU= 27.878
  23:15:02 STEP 1500 H=1500:00 +CPU= 46.981
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001500
  23:15:31 STEP 1501 H=1501:00 +CPU= 29.641
  23:15:57 STEP 1502 H=1502:00 +CPU= 25.952
  23:16:42 STEP 1503 H=1503:00 +CPU= 44.528
  23:17:07 STEP 1504 H=1504:00 +CPU= 24.913
  23:17:37 STEP 1505 H=1505:00 +CPU= 29.835
  23:18:20 STEP 1506 H=1506:00 +CPU= 43.283
  23:18:48 STEP 1507 H=1507:00 +CPU= 27.959
  23:19:19 STEP 1508 H=1508:00 +CPU= 30.894
  23:20:01 STEP 1509 H=1509:00 +CPU= 42.138
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:21:06 STEP 1510 H=1510:00 +CPU= 32.279
  23:21:31 STEP 1511 H=1511:00 +CPU= 24.885
  23:22:22 STEP 1512 H=1512:00 +CPU= 50.593
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_62.zip
Uploading the intermediate file: upload_file_62.zip
  23:22:51 STEP 1513 H=1513:00 +CPU= 29.538
  23:23:22 STEP 1514 H=1514:00 +CPU= 30.271
  23:24:02 STEP 1515 H=1515:00 +CPU= 40.684
  23:24:27 STEP 1516 H=1516:00 +CPU= 24.719
  23:24:54 STEP 1517 H=1517:00 +CPU= 26.676
  23:25:43 STEP 1518 H=1518:00 +CPU= 48.610
  23:26:08 STEP 1519 H=1519:00 +CPU= 25.630
  23:26:40 STEP 1520 H=1520:00 +CPU= 31.181
  23:27:20 STEP 1521 H=1521:00 +CPU= 39.664
  23:27:50 STEP 1522 H=1522:00 +CPU= 29.887
  23:28:17 STEP 1523 H=1523:00 +CPU= 27.638
  23:29:04 STEP 1524 H=1524:00 +CPU= 46.620
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001524
  23:29:34 STEP 1525 H=1525:00 +CPU= 29.683
  23:30:01 STEP 1526 H=1526:00 +CPU= 26.943
  23:30:40 STEP 1527 H=1527:00 +CPU= 39.508
  23:31:06 STEP 1528 H=1528:00 +CPU= 25.312
  23:31:32 STEP 1529 H=1529:00 +CPU= 26.350
  23:32:21 STEP 1530 H=1530:00 +CPU= 48.700
  23:32:51 STEP 1531 H=1531:00 +CPU= 29.799
  23:33:22 STEP 1532 H=1532:00 +CPU= 30.939
  23:34:01 STEP 1533 H=1533:00 +CPU= 39.027
  23:34:32 STEP 1534 H=1534:00 +CPU= 30.563
  23:34:59 STEP 1535 H=1535:00 +CPU= 27.321
  23:35:48 STEP 1536 H=1536:00 +CPU= 48.871
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_63.zip
Uploading the intermediate file: upload_file_63.zip
  23:36:18 STEP 1537 H=1537:00 +CPU= 29.611
  23:36:46 STEP 1538 H=1538:00 +CPU= 27.565
  23:37:33 STEP 1539 H=1539:00 +CPU= 46.694
  23:38:00 STEP 1540 H=1540:00 +CPU= 27.228
  23:38:25 STEP 1541 H=1541:00 +CPU= 24.965
  23:39:10 STEP 1542 H=1542:00 +CPU= 44.882
  23:39:34 STEP 1543 H=1543:00 +CPU= 24.447
  23:40:05 STEP 1544 H=1544:00 +CPU= 31.128
  23:40:47 STEP 1545 H=1545:00 +CPU= 41.338
  23:41:16 STEP 1546 H=1546:00 +CPU= 29.284
  23:41:46 STEP 1547 H=1547:00 +CPU= 29.642
  23:42:39 STEP 1548 H=1548:00 +CPU= 52.879
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001548
  23:43:09 STEP 1549 H=1549:00 +CPU= 29.744
  23:43:34 STEP 1550 H=1550:00 +CPU= 25.067
  23:44:19 STEP 1551 H=1551:00 +CPU= 45.532
  23:44:46 STEP 1552 H=1552:00 +CPU= 26.896
  23:45:13 STEP 1553 H=1553:00 +CPU= 26.397
  23:45:57 STEP 1554 H=1554:00 +CPU= 43.622
  23:46:22 STEP 1555 H=1555:00 +CPU= 25.283
  23:46:53 STEP 1556 H=1556:00 +CPU= 31.240
  23:47:33 STEP 1557 H=1557:00 +CPU= 40.202
  23:48:02 STEP 1558 H=1558:00 +CPU= 28.627
  23:48:32 STEP 1559 H=1559:00 +CPU= 29.657
  23:49:20 STEP 1560 H=1560:00 +CPU= 47.891
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_64.zip
Uploading the intermediate file: upload_file_64.zip
  23:49:50 STEP 1561 H=1561:00 +CPU= 29.580
  23:50:16 STEP 1562 H=1562:00 +CPU= 25.899
  23:51:02 STEP 1563 H=1563:00 +CPU= 46.587
  23:51:30 STEP 1564 H=1564:00 +CPU= 27.565
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0405
(argv4) batchid: 979
(argv5) wuid: 12196049
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7ed06955a79dd9be2ff16ce1f9896ecf to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0405_2010050100_123_979_12196049.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0405_2010050100_123_979_12196049.zip
ic_ancil_file: ic_ancil_12196049
ifsdata_file: ifsdata_12196049
climate_data_file: clim_data_12196049
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cbe620a22c544a81e5e79239f7fa4ef8 to: /var/lib/boinc-client/slots/0/ic_ancil_12196049.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12196049.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_28ac0ce1bcf0e2594f22717b3a428dec to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12196049.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12196049.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12196049.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12196049.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12196049.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12196049.xml
-- Model is restarting --
Adjusting last_iter, 1565, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12196049.xml
last_cpu_time: 52136
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588
The child process has been launched with process id: 239765
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:239765:239765] [20230116:235243:1673909563:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:53:31 STEP 1560 H=1560:00 +CPU= 45.032
  23:53:51 STEP 1561 H=1561:00 +CPU= 20.320
  23:54:14 STEP 1562 H=1562:00 +CPU= 23.322
  23:54:58 STEP 1563 H=1563:00 +CPU= 43.305
  23:55:28 STEP 1564 H=1564:00 +CPU= 30.498
  23:55:59 STEP 1565 H=1565:00 +CPU= 30.852
  23:56:45 STEP 1566 H=1566:00 +CPU= 45.680
  23:57:15 STEP 1567 H=1567:00 +CPU= 29.614
  23:57:45 STEP 1568 H=1568:00 +CPU= 30.506
  23:58:26 STEP 1569 H=1569:00 +CPU= 41.113
  23:58:58 STEP 1570 H=1570:00 +CPU= 31.079
  23:59:23 STEP 1571 H=1571:00 +CPU= 25.382
  00:00:07 STEP 1572 H=1572:00 +CPU= 44.241
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001572
  00:00:33 STEP 1573 H=1573:00 +CPU= 25.381
  00:01:01 STEP 1574 H=1574:00 +CPU= 27.993
  00:01:50 STEP 1575 H=1575:00 +CPU= 48.553
  00:02:20 STEP 1576 H=1576:00 +CPU= 30.748
  00:02:51 STEP 1577 H=1577:00 +CPU= 30.621
  00:03:29 STEP 1578 H=1578:00 +CPU= 38.361
  00:03:57 STEP 1579 H=1579:00 +CPU= 26.991
  00:04:27 STEP 1580 H=1580:00 +CPU= 30.628
  00:05:15 STEP 1581 H=1581:00 +CPU= 47.885
  00:05:42 STEP 1582 H=1582:00 +CPU= 26.717
  00:06:07 STEP 1583 H=1583:00 +CPU= 24.943
  00:07:01 STEP 1584 H=1584:00 +CPU= 53.604
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_65.zip
Uploading the intermediate file: upload_file_65.zip
  00:07:30 STEP 1585 H=1585:00 +CPU= 28.798
  00:07:57 STEP 1586 H=1586:00 +CPU= 27.814
  00:08:40 STEP 1587 H=1587:00 +CPU= 42.136
  00:09:07 STEP 1588 H=1588:00 +CPU= 27.390
  00:09:37 STEP 1589 H=1589:00 +CPU= 30.180
  00:10:23 STEP 1590 H=1590:00 +CPU= 45.506
  00:10:55 STEP 1591 H=1591:00 +CPU= 30.711
  00:11:25 STEP 1592 H=1592:00 +CPU= 30.605
  00:12:13 STEP 1593 H=1593:00 +CPU= 48.031
  00:12:40 STEP 1594 H=1594:00 +CPU= 26.294
  00:13:05 STEP 1595 H=1595:00 +CPU= 25.670
  00:13:59 STEP 1596 H=1596:00 +CPU= 53.233
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001596
  00:14:29 STEP 1597 H=1597:00 +CPU= 29.837
  00:14:57 STEP 1598 H=1598:00 +CPU= 28.410
  00:15:37 STEP 1599 H=1599:00 +CPU= 39.388
  00:16:03 STEP 1600 H=1600:00 +CPU= 26.433
  00:16:33 STEP 1601 H=1601:00 +CPU= 29.991
  00:17:22 STEP 1602 H=1602:00 +CPU= 48.587
  00:17:52 STEP 1603 H=1603:00 +CPU= 30.345
  00:18:19 STEP 1604 H=1604:00 +CPU= 27.133
  00:19:03 STEP 1605 H=1605:00 +CPU= 43.979
  00:19:34 STEP 1606 H=1606:00 +CPU= 30.450
  00:19:58 STEP 1607 H=1607:00 +CPU= 24.429
  00:20:50 STEP 1608 H=1608:00 +CPU= 51.215
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_66.zip
Uploading the intermediate file: upload_file_66.zip
  00:21:15 STEP 1609 H=1609:00 +CPU= 25.070
  00:21:44 STEP 1610 H=1610:00 +CPU= 29.313
  00:22:32 STEP 1611 H=1611:00 +CPU= 47.426
  00:23:03 STEP 1612 H=1612:00 +CPU= 30.954
  00:23:28 STEP 1613 H=1613:00 +CPU= 24.973
  00:24:11 STEP 1614 H=1614:00 +CPU= 43.298
  00:24:39 STEP 1615 H=1615:00 +CPU= 27.986
  00:25:04 STEP 1616 H=1616:00 +CPU= 24.854
  00:25:52 STEP 1617 H=1617:00 +CPU= 47.683
  00:26:17 STEP 1618 H=1618:00 +CPU= 25.020
  00:26:43 STEP 1619 H=1619:00 +CPU= 26.202
  00:27:33 STEP 1620 H=1620:00 +CPU= 49.501
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001620
  00:27:58 STEP 1621 H=1621:00 +CPU= 25.024
  00:28:29 STEP 1622 H=1622:00 +CPU= 31.290
  00:29:17 STEP 1623 H=1623:00 +CPU= 47.256
  00:29:48 STEP 1624 H=1624:00 +CPU= 31.197
  00:30:14 STEP 1625 H=1625:00 +CPU= 25.867
  00:30:57 STEP 1626 H=1626:00 +CPU= 43.524
  00:31:28 STEP 1627 H=1627:00 +CPU= 30.103
  00:31:54 STEP 1628 H=1628:00 +CPU= 26.389
  00:32:40 STEP 1629 H=1629:00 +CPU= 45.750
  00:33:11 STEP 1630 H=1630:00 +CPU= 30.690
  00:33:38 STEP 1631 H=1631:00 +CPU= 27.074
  00:34:36 STEP 1632 H=1632:00 +CPU= 55.404
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_67.zip
Uploading the intermediate file: upload_file_67.zip
  00:35:01 STEP 1633 H=1633:00 +CPU= 24.918
  00:35:30 STEP 1634 H=1634:00 +CPU= 29.636
  00:36:12 STEP 1635 H=1635:00 +CPU= 41.505
  00:36:41 STEP 1636 H=1636:00 +CPU= 29.257
  00:37:12 STEP 1637 H=1637:00 +CPU= 30.337
  00:37:53 STEP 1638 H=1638:00 +CPU= 41.478
  00:38:23 STEP 1639 H=1639:00 +CPU= 30.057
  00:38:54 STEP 1640 H=1640:00 +CPU= 30.668
  00:39:42 STEP 1641 H=1641:00 +CPU= 47.776
  00:40:08 STEP 1642 H=1642:00 +CPU= 25.690
  00:40:33 STEP 1643 H=1643:00 +CPU= 24.845
  00:41:25 STEP 1644 H=1644:00 +CPU= 52.701
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001644
  00:41:50 STEP 1645 H=1645:00 +CPU= 24.518
  00:42:18 STEP 1646 H=1646:00 +CPU= 28.287
  00:43:06 STEP 1647 H=1647:00 +CPU= 47.613
  00:43:37 STEP 1648 H=1648:00 +CPU= 30.800
  00:44:07 STEP 1649 H=1649:00 +CPU= 30.431
  00:44:49 STEP 1650 H=1650:00 +CPU= 41.931
  00:45:18 STEP 1651 H=1651:00 +CPU= 28.392
  00:45:45 STEP 1652 H=1652:00 +CPU= 27.458
  00:46:26 STEP 1653 H=1653:00 +CPU= 40.683
  00:46:57 STEP 1654 H=1654:00 +CPU= 31.315
  00:47:28 STEP 1655 H=1655:00 +CPU= 30.649
  00:48:20 STEP 1656 H=1656:00 +CPU= 52.022
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_68.zip
Uploading the intermediate file: upload_file_68.zip
  00:48:50 STEP 1657 H=1657:00 +CPU= 29.588
  00:49:21 STEP 1658 H=1658:00 +CPU= 31.056
  00:50:01 STEP 1659 H=1659:00 +CPU= 39.857
  00:50:28 STEP 1660 H=1660:00 +CPU= 26.716
  00:50:59 STEP 1661 H=1661:00 +CPU= 30.560
  00:51:38 STEP 1662 H=1662:00 +CPU= 38.814
  00:52:07 STEP 1663 H=1663:00 +CPU= 29.706
  00:52:38 STEP 1664 H=1664:00 +CPU= 30.718
  00:53:20 STEP 1665 H=1665:00 +CPU= 42.092
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0405
(argv4) batchid: 979
(argv5) wuid: 12196049
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7ed06955a79dd9be2ff16ce1f9896ecf to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0405_2010050100_123_979_12196049.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0405_2010050100_123_979_12196049.zip
ic_ancil_file: ic_ancil_12196049
ifsdata_file: ifsdata_12196049
climate_data_file: clim_data_12196049
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cbe620a22c544a81e5e79239f7fa4ef8 to: /var/lib/boinc-client/slots/0/ic_ancil_12196049.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12196049.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_28ac0ce1bcf0e2594f22717b3a428dec to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12196049.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12196049.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12196049.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12196049.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12196049.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12196049.xml
-- Model is restarting --
Adjusting last_iter, 1666, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12196049.xml
last_cpu_time: 55765
upload_file_number: 69
last_iter: 1655
last_upload: 5961600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588
The child process has been launched with process id: 242886
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:242886:242886] [20230117:005439:1673913279:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:55:40 STEP 1656 H=1656:00 +CPU= 44.796
  00:56:05 STEP 1657 H=1657:00 +CPU= 25.276
  00:56:37 STEP 1658 H=1658:00 +CPU= 31.533
  00:57:24 STEP 1659 H=1659:00 +CPU= 47.291
  00:57:55 STEP 1660 H=1660:00 +CPU= 31.103
  00:58:22 STEP 1661 H=1661:00 +CPU= 26.963
  00:59:06 STEP 1662 H=1662:00 +CPU= 43.814
  00:59:36 STEP 1663 H=1663:00 +CPU= 29.673
  01:00:02 STEP 1664 H=1664:00 +CPU= 26.016
  01:00:49 STEP 1665 H=1665:00 +CPU= 47.549
  01:01:20 STEP 1666 H=1666:00 +CPU= 30.414
  01:01:47 STEP 1667 H=1667:00 +CPU= 26.647
  01:02:36 STEP 1668 H=1668:00 +CPU= 48.024
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001668
  01:03:03 STEP 1669 H=1669:00 +CPU= 27.337
  01:03:34 STEP 1670 H=1670:00 +CPU= 30.576
  01:04:13 STEP 1671 H=1671:00 +CPU= 38.618
  01:04:44 STEP 1672 H=1672:00 +CPU= 31.216
  01:05:11 STEP 1673 H=1673:00 +CPU= 26.487
  01:05:56 STEP 1674 H=1674:00 +CPU= 45.125
  01:06:21 STEP 1675 H=1675:00 +CPU= 25.161
  01:06:47 STEP 1676 H=1676:00 +CPU= 25.312
  01:07:32 STEP 1677 H=1677:00 +CPU= 45.005
  01:08:02 STEP 1678 H=1678:00 +CPU= 30.462
  01:08:33 STEP 1679 H=1679:00 +CPU= 30.998
  01:09:20 STEP 1680 H=1680:00 +CPU= 46.502
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_69.zip
Uploading the intermediate file: upload_file_69.zip
  01:09:49 STEP 1681 H=1681:00 +CPU= 28.588
  01:10:20 STEP 1682 H=1682:00 +CPU= 30.752
  01:11:08 STEP 1683 H=1683:00 +CPU= 47.671
  01:11:35 STEP 1684 H=1684:00 +CPU= 27.279
  01:12:00 STEP 1685 H=1685:00 +CPU= 24.440
  01:12:44 STEP 1686 H=1686:00 +CPU= 44.394
  01:13:14 STEP 1687 H=1687:00 +CPU= 29.834
  01:13:43 STEP 1688 H=1688:00 +CPU= 28.955
  01:14:29 STEP 1689 H=1689:00 +CPU= 45.934
  01:14:55 STEP 1690 H=1690:00 +CPU= 26.026
  01:15:21 STEP 1691 H=1691:00 +CPU= 26.065
  01:16:14 STEP 1692 H=1692:00 +CPU= 53.070
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001692
  01:16:38 STEP 1693 H=1693:00 +CPU= 23.670
  01:17:09 STEP 1694 H=1694:00 +CPU= 31.136
  01:17:48 STEP 1695 H=1695:00 +CPU= 38.324
  01:18:16 STEP 1696 H=1696:00 +CPU= 28.289
  01:18:41 STEP 1697 H=1697:00 +CPU= 25.270
  01:19:24 STEP 1698 H=1698:00 +CPU= 43.072
  01:19:55 STEP 1699 H=1699:00 +CPU= 30.588
  01:20:25 STEP 1700 H=1700:00 +CPU= 29.827
  01:21:10 STEP 1701 H=1701:00 +CPU= 44.646
  01:21:38 STEP 1702 H=1702:00 +CPU= 28.018
  01:22:04 STEP 1703 H=1703:00 +CPU= 26.154
  01:23:01 STEP 1704 H=1704:00 +CPU= 55.658
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_70.zip
Uploading the intermediate file: upload_file_70.zip
  01:23:31 STEP 1705 H=1705:00 +CPU= 30.004
  01:24:03 STEP 1706 H=1706:00 +CPU= 31.008
  01:24:42 STEP 1707 H=1707:00 +CPU= 39.295
  01:25:13 STEP 1708 H=1708:00 +CPU= 30.737
  01:25:42 STEP 1709 H=1709:00 +CPU= 29.478
  01:26:31 STEP 1710 H=1710:00 +CPU= 48.982
  01:26:55 STEP 1711 H=1711:00 +CPU= 23.733
  01:27:21 STEP 1712 H=1712:00 +CPU= 25.683
  01:28:08 STEP 1713 H=1713:00 +CPU= 46.991
  01:28:37 STEP 1714 H=1714:00 +CPU= 28.608
  01:29:01 STEP 1715 H=1715:00 +CPU= 23.848
  01:29:44 STEP 1716 H=1716:00 +CPU= 43.113
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001716
  01:30:11 STEP 1717 H=1717:00 +CPU= 27.539
  01:30:42 STEP 1718 H=1718:00 +CPU= 31.004
  01:31:24 STEP 1719 H=1719:00 +CPU= 41.890
  01:31:56 STEP 1720 H=1720:00 +CPU= 31.359
  01:32:25 STEP 1721 H=1721:00 +CPU= 29.512
  01:33:11 STEP 1722 H=1722:00 +CPU= 45.711
  01:33:41 STEP 1723 H=1723:00 +CPU= 29.524
  01:34:06 STEP 1724 H=1724:00 +CPU= 24.751
  01:34:54 STEP 1725 H=1725:00 +CPU= 47.948
  01:35:19 STEP 1726 H=1726:00 +CPU= 25.861
  01:35:50 STEP 1727 H=1727:00 +CPU= 30.776
  01:36:40 STEP 1728 H=1728:00 +CPU= 48.848
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_71.zip
Uploading the intermediate file: upload_file_71.zip
  01:37:07 STEP 1729 H=1729:00 +CPU= 27.607
  01:37:38 STEP 1730 H=1730:00 +CPU= 31.047
  01:38:22 STEP 1731 H=1731:00 +CPU= 43.171
  01:38:53 STEP 1732 H=1732:00 +CPU= 31.206
  01:39:23 STEP 1733 H=1733:00 +CPU= 29.506
  01:40:07 STEP 1734 H=1734:00 +CPU= 44.055
  01:40:37 STEP 1735 H=1735:00 +CPU= 29.757
  01:41:01 STEP 1736 H=1736:00 +CPU= 24.190
  01:41:48 STEP 1737 H=1737:00 +CPU= 46.667
  01:42:18 STEP 1738 H=1738:00 +CPU= 30.575
  01:42:48 STEP 1739 H=1739:00 +CPU= 30.109
  01:43:34 STEP 1740 H=1740:00 +CPU= 45.428
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001740
  01:44:00 STEP 1741 H=1741:00 +CPU= 26.100
  01:44:31 STEP 1742 H=1742:00 +CPU= 30.784
  01:45:15 STEP 1743 H=1743:00 +CPU= 43.770
  01:45:46 STEP 1744 H=1744:00 +CPU= 31.250
  01:46:15 STEP 1745 H=1745:00 +CPU= 28.423
  01:46:58 STEP 1746 H=1746:00 +CPU= 43.017
  01:47:26 STEP 1747 H=1747:00 +CPU= 28.426
  01:47:51 STEP 1748 H=1748:00 +CPU= 24.530
  01:48:38 STEP 1749 H=1749:00 +CPU= 46.983
  01:49:06 STEP 1750 H=1750:00 +CPU= 28.580
  01:49:32 STEP 1751 H=1751:00 +CPU= 25.122
  01:50:18 STEP 1752 H=1752:00 +CPU= 45.528
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_72.zip
Uploading the intermediate file: upload_file_72.zip
  01:50:46 STEP 1753 H=1753:00 +CPU= 27.582
  01:51:17 STEP 1754 H=1754:00 +CPU= 31.238
  01:52:02 STEP 1755 H=1755:00 +CPU= 44.709
  01:52:32 STEP 1756 H=1756:00 +CPU= 30.523
  01:52:59 STEP 1757 H=1757:00 +CPU= 26.712
  01:53:45 STEP 1758 H=1758:00 +CPU= 45.880
  01:54:15 STEP 1759 H=1759:00 +CPU= 29.659
  01:54:46 STEP 1760 H=1760:00 +CPU= 30.768
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0405
(argv4) batchid: 979
(argv5) wuid: 12196049
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7ed06955a79dd9be2ff16ce1f9896ecf to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0405_2010050100_123_979_12196049.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0405_2010050100_123_979_12196049.zip
ic_ancil_file: ic_ancil_12196049
ifsdata_file: ifsdata_12196049
climate_data_file: clim_data_12196049
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cbe620a22c544a81e5e79239f7fa4ef8 to: /var/lib/boinc-client/slots/0/ic_ancil_12196049.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12196049.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_28ac0ce1bcf0e2594f22717b3a428dec to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12196049.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12196049.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12196049.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12196049.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12196049.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12196049.xml
-- Model is restarting --
Adjusting last_iter, 1761, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12196049.xml
last_cpu_time: 59349
upload_file_number: 73
last_iter: 1751
last_upload: 6307200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588
The child process has been launched with process id: 246243
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:246243:246243] [20230117:015559:1673916959:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:56:58 STEP 1752 H=1752:00 +CPU= 49.893
  01:57:23 STEP 1753 H=1753:00 +CPU= 25.631
  01:57:55 STEP 1754 H=1754:00 +CPU= 31.224
  01:58:37 STEP 1755 H=1755:00 +CPU= 42.457
  01:59:08 STEP 1756 H=1756:00 +CPU= 30.917
  01:59:33 STEP 1757 H=1757:00 +CPU= 25.164
  02:00:16 STEP 1758 H=1758:00 +CPU= 42.820
  02:00:46 STEP 1759 H=1759:00 +CPU= 29.716
  02:01:13 STEP 1760 H=1760:00 +CPU= 27.229
  02:01:55 STEP 1761 H=1761:00 +CPU= 41.980
  02:02:21 STEP 1762 H=1762:00 +CPU= 25.401
  02:02:49 STEP 1763 H=1763:00 +CPU= 27.961
  02:03:42 STEP 1764 H=1764:00 +CPU= 52.620
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001764
  02:04:12 STEP 1765 H=1765:00 +CPU= 30.316
  02:04:40 STEP 1766 H=1766:00 +CPU= 27.772
  02:05:19 STEP 1767 H=1767:00 +CPU= 39.535
  02:05:50 STEP 1768 H=1768:00 +CPU= 30.397
  02:06:19 STEP 1769 H=1769:00 +CPU= 29.447
  02:07:03 STEP 1770 H=1770:00 +CPU= 43.485
  02:07:29 STEP 1771 H=1771:00 +CPU= 26.005
  02:07:54 STEP 1772 H=1772:00 +CPU= 24.515
  02:08:41 STEP 1773 H=1773:00 +CPU= 46.978
  02:09:07 STEP 1774 H=1774:00 +CPU= 26.572
  02:09:36 STEP 1775 H=1775:00 +CPU= 28.403
  02:10:27 STEP 1776 H=1776:00 +CPU= 51.558
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196049/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_2010050100_123_979_12196049_1_r949037588_73.zip
Uploading the intermediate file: upload_file_73.zip
  02:10:55 STEP 1777 H=1777:00 +CPU= 27.230
  02:11:26 STEP 1778 H=1778:00 +CPU= 30.964
  02:12:10 STEP 1779 H=1779:00 +CPU= 44.385
  02:12:41 STEP 1780 H=1780:00 +CPU= 30.371
  02:13:05 STEP 1781 H=1781:00 +CPU= 24.140
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:21:53 1467890 22297954 oifs_43r3_ps_0405_2010050100_123_979_12196049_1 4,492,800 40,384 0.0090
09 May 2023 16:21:45 1467890 22297954 oifs_43r3_ps_0405_2010050100_123_979_12196049_1 3,628,800 32,709 0.0090
09 May 2023 16:21:37 1467890 22297954 oifs_43r3_ps_0405_2010050100_123_979_12196049_1 2,764,800 25,006 0.0090
09 May 2023 16:21:24 1467890 22297954 oifs_43r3_ps_0405_2010050100_123_979_12196049_1 1,900,800 17,305 0.0091
09 May 2023 16:20:18 1467890 22297954 oifs_43r3_ps_0405_2010050100_123_979_12196049_1 950,400 8,560 0.0090


©2024 cpdn.org