climateprediction.net home page
Task 22302609

Task 22302609

Name oifs_43r3_ps_0568_2019050100_123_988_12205212_1
Workunit 12205212
Created 23 Jan 2023, 23:44:06 UTC
Sent 23 Jan 2023, 23:50:19 UTC
Report deadline 24 Mar 2023, 23:50:19 UTC
Received 18 Feb 2023, 10:36:21 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1509518
Run time 17 hours 47 min 1 sec
CPU time 13 hours 41 min 24 sec
Validate state Invalid
Credit 6,284.88
Device peak FLOPS 4.20 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.27 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,860.73 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
8
  16:08:35 STEP 1531 H=1531:00 +CPU= 22.209
  16:08:58 STEP 1532 H=1532:00 +CPU= 22.680
  16:09:34 STEP 1533 H=1533:00 +CPU= 36.368
  16:09:57 STEP 1534 H=1534:00 +CPU= 22.754
  16:10:20 STEP 1535 H=1535:00 +CPU= 22.656
  16:11:05 STEP 1536 H=1536:00 +CPU= 45.462
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_52.zip
Uploading the intermediate file: upload_file_52.zip
  16:11:28 STEP 1537 H=1537:00 +CPU= 22.595
  16:11:55 STEP 1538 H=1538:00 +CPU= 26.745
  16:12:31 STEP 1539 H=1539:00 +CPU= 35.965
  16:12:54 STEP 1540 H=1540:00 +CPU= 22.895
  16:13:19 STEP 1541 H=1541:00 +CPU= 24.491
  16:13:54 STEP 1542 H=1542:00 +CPU= 34.876
  16:14:17 STEP 1543 H=1543:00 +CPU= 23.183
  16:14:44 STEP 1544 H=1544:00 +CPU= 26.836
  16:15:19 STEP 1545 H=1545:00 +CPU= 34.789
  16:15:44 STEP 1546 H=1546:00 +CPU= 25.642
  16:16:09 STEP 1547 H=1547:00 +CPU= 24.521
  16:16:48 STEP 1548 H=1548:00 +CPU= 38.513
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001548
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:40 STEP 1549 H=1549:00 +CPU= 28.311
  16:18:11 STEP 1550 H=1550:00 +CPU= 29.013
  16:19:02 STEP 1551 H=1551:00 +CPU= 45.767
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:19:59 STEP 1552 H=1552:00 +CPU= 31.416
  16:20:34 STEP 1553 H=1553:00 +CPU= 32.599
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:04 STEP 1554 H=1554:00 +CPU= 47.680
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:24:40 STEP 1555 H=1555:00 +CPU= 29.430
  16:25:07 STEP 1556 H=1556:00 +CPU= 25.774
  16:25:47 STEP 1557 H=1557:00 +CPU= 38.550
  16:26:13 STEP 1558 H=1558:00 +CPU= 25.308
  16:26:50 STEP 1559 H=1559:00 +CPU= 33.076
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:28:54 STEP 1560 H=1560:00 +CPU= 50.037
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_53.zip
Uploading the intermediate file: upload_file_53.zip
  16:29:17 STEP 1561 H=1561:00 +CPU= 22.646
  16:29:43 STEP 1562 H=1562:00 +CPU= 25.405
  16:30:19 STEP 1563 H=1563:00 +CPU= 36.624
  16:30:43 STEP 1564 H=1564:00 +CPU= 23.052
  16:31:10 STEP 1565 H=1565:00 +CPU= 27.038
  16:31:50 STEP 1566 H=1566:00 +CPU= 39.045
  16:32:12 STEP 1567 H=1567:00 +CPU= 22.159
  16:32:39 STEP 1568 H=1568:00 +CPU= 27.173
  16:33:15 STEP 1569 H=1569:00 +CPU= 35.095
  16:33:37 STEP 1570 H=1570:00 +CPU= 22.812
  16:34:02 STEP 1571 H=1571:00 +CPU= 24.405
  16:34:40 STEP 1572 H=1572:00 +CPU= 38.010
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001572
  16:35:04 STEP 1573 H=1573:00 +CPU= 24.012
  16:35:30 STEP 1574 H=1574:00 +CPU= 25.701
  16:36:04 STEP 1575 H=1575:00 +CPU= 34.040
  16:36:31 STEP 1576 H=1576:00 +CPU= 26.921
  16:36:54 STEP 1577 H=1577:00 +CPU= 23.206
  16:37:29 STEP 1578 H=1578:00 +CPU= 34.719
  16:37:53 STEP 1579 H=1579:00 +CPU= 24.334
  16:38:16 STEP 1580 H=1580:00 +CPU= 22.709
  16:38:51 STEP 1581 H=1581:00 +CPU= 34.774
  16:39:18 STEP 1582 H=1582:00 +CPU= 26.557
  16:39:40 STEP 1583 H=1583:00 +CPU= 22.699
  16:40:23 STEP 1584 H=1584:00 +CPU= 42.084
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_54.zip
Uploading the intermediate file: upload_file_54.zip
  16:40:46 STEP 1585 H=1585:00 +CPU= 22.888
  16:41:09 STEP 1586 H=1586:00 +CPU= 22.865
  16:41:45 STEP 1587 H=1587:00 +CPU= 36.002
  16:42:08 STEP 1588 H=1588:00 +CPU= 22.849
  16:42:31 STEP 1589 H=1589:00 +CPU= 22.878
  16:43:10 STEP 1590 H=1590:00 +CPU= 39.405
  16:43:32 STEP 1591 H=1591:00 +CPU= 22.410
  16:43:57 STEP 1592 H=1592:00 +CPU= 24.455
  16:44:35 STEP 1593 H=1593:00 +CPU= 37.528
  16:44:58 STEP 1594 H=1594:00 +CPU= 23.489
  16:45:24 STEP 1595 H=1595:00 +CPU= 25.346
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0568
(argv4) batchid: 988
(argv5) wuid: 12205212
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_27f419f4b165abbaf73a14b1a0bb5fa9 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0568_2019050100_123_988_12205212.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0568_2019050100_123_988_12205212.zip
ic_ancil_file: ic_ancil_12205212
ifsdata_file: ifsdata_12205212
climate_data_file: clim_data_12205212
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_13638bc7c4e6293693e3d09fb3af2cda to: /var/lib/boinc-client/slots/0/ic_ancil_12205212.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12205212.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f96f8520aae04e3bb7484c5c59d96365 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205212.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205212.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205212.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205212.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12205212.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12205212.xml
-- Model is restarting --
Adjusting last_iter, 1596, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12205212.xml
last_cpu_time: 41256
upload_file_number: 55
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861
The child process has been launched with process id: 6529
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:6529:6529] [20230217:164645:1676681205:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:47:31 STEP 1584 H=1584:00 +CPU= 41.550
  16:47:53 STEP 1585 H=1585:00 +CPU= 22.032
  16:48:21 STEP 1586 H=1586:00 +CPU= 27.386
  16:48:55 STEP 1587 H=1587:00 +CPU= 33.920
  16:49:18 STEP 1588 H=1588:00 +CPU= 22.788
  16:49:45 STEP 1589 H=1589:00 +CPU= 27.213
  16:50:20 STEP 1590 H=1590:00 +CPU= 34.612
  16:50:42 STEP 1591 H=1591:00 +CPU= 22.091
  16:51:10 STEP 1592 H=1592:00 +CPU= 28.183
  16:51:44 STEP 1593 H=1593:00 +CPU= 34.186
  16:52:08 STEP 1594 H=1594:00 +CPU= 23.979
  16:52:34 STEP 1595 H=1595:00 +CPU= 25.311
  16:53:12 STEP 1596 H=1596:00 +CPU= 37.868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001596
  16:53:37 STEP 1597 H=1597:00 +CPU= 25.000
  16:54:02 STEP 1598 H=1598:00 +CPU= 24.826
  16:54:36 STEP 1599 H=1599:00 +CPU= 34.110
  16:55:02 STEP 1600 H=1600:00 +CPU= 26.442
  16:55:25 STEP 1601 H=1601:00 +CPU= 22.785
  16:56:00 STEP 1602 H=1602:00 +CPU= 34.808
  16:56:25 STEP 1603 H=1603:00 +CPU= 24.687
  16:56:48 STEP 1604 H=1604:00 +CPU= 22.801
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:57:43 STEP 1605 H=1605:00 +CPU= 40.904
  16:58:06 STEP 1606 H=1606:00 +CPU= 23.041
  16:58:30 STEP 1607 H=1607:00 +CPU= 23.793
  16:59:16 STEP 1608 H=1608:00 +CPU= 45.513
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_55.zip
Uploading the intermediate file: upload_file_55.zip
  16:59:38 STEP 1609 H=1609:00 +CPU= 22.214
  17:00:06 STEP 1610 H=1610:00 +CPU= 27.879
  17:00:41 STEP 1611 H=1611:00 +CPU= 34.819
  17:01:04 STEP 1612 H=1612:00 +CPU= 23.059
  17:01:30 STEP 1613 H=1613:00 +CPU= 26.125
  17:02:05 STEP 1614 H=1614:00 +CPU= 34.700
  17:02:27 STEP 1615 H=1615:00 +CPU= 22.164
  17:02:55 STEP 1616 H=1616:00 +CPU= 27.794
  17:03:29 STEP 1617 H=1617:00 +CPU= 34.051
  17:03:52 STEP 1618 H=1618:00 +CPU= 23.160
  17:04:19 STEP 1619 H=1619:00 +CPU= 27.024
  17:04:57 STEP 1620 H=1620:00 +CPU= 37.989
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001620
  17:05:22 STEP 1621 H=1621:00 +CPU= 24.888
  17:05:49 STEP 1622 H=1622:00 +CPU= 26.035
  17:06:23 STEP 1623 H=1623:00 +CPU= 33.982
  17:06:50 STEP 1624 H=1624:00 +CPU= 27.243
  17:07:13 STEP 1625 H=1625:00 +CPU= 23.082
  17:07:48 STEP 1626 H=1626:00 +CPU= 34.890
  17:08:12 STEP 1627 H=1627:00 +CPU= 24.055
  17:08:35 STEP 1628 H=1628:00 +CPU= 22.646
  17:09:11 STEP 1629 H=1629:00 +CPU= 35.681
  17:09:37 STEP 1630 H=1630:00 +CPU= 26.807
  17:10:00 STEP 1631 H=1631:00 +CPU= 22.843
  17:10:43 STEP 1632 H=1632:00 +CPU= 42.660
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_56.zip
Uploading the intermediate file: upload_file_56.zip
  17:11:06 STEP 1633 H=1633:00 +CPU= 22.397
  17:11:29 STEP 1634 H=1634:00 +CPU= 22.937
  17:12:05 STEP 1635 H=1635:00 +CPU= 36.587
  17:12:28 STEP 1636 H=1636:00 +CPU= 22.646
  17:12:51 STEP 1637 H=1637:00 +CPU= 23.175
  17:13:30 STEP 1638 H=1638:00 +CPU= 38.928
  17:13:53 STEP 1639 H=1639:00 +CPU= 22.200
  17:14:19 STEP 1640 H=1640:00 +CPU= 26.489
  17:14:56 STEP 1641 H=1641:00 +CPU= 36.947
  17:15:19 STEP 1642 H=1642:00 +CPU= 22.680
  17:15:46 STEP 1643 H=1643:00 +CPU= 26.635
  17:16:25 STEP 1644 H=1644:00 +CPU= 39.059
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001644
  17:16:47 STEP 1645 H=1645:00 +CPU= 22.072
  17:17:15 STEP 1646 H=1646:00 +CPU= 27.810
  17:17:49 STEP 1647 H=1647:00 +CPU= 34.333
  17:18:13 STEP 1648 H=1648:00 +CPU= 23.966
  17:18:40 STEP 1649 H=1649:00 +CPU= 26.656
  17:19:15 STEP 1650 H=1650:00 +CPU= 34.988
  17:19:39 STEP 1651 H=1651:00 +CPU= 24.567
  17:20:05 STEP 1652 H=1652:00 +CPU= 25.231
  17:20:39 STEP 1653 H=1653:00 +CPU= 34.465
  17:21:07 STEP 1654 H=1654:00 +CPU= 27.668
  17:21:31 STEP 1655 H=1655:00 +CPU= 24.217
  17:22:12 STEP 1656 H=1656:00 +CPU= 40.542
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_57.zip
Uploading the intermediate file: upload_file_57.zip
  17:22:37 STEP 1657 H=1657:00 +CPU= 25.380
  17:23:00 STEP 1658 H=1658:00 +CPU= 22.984
  17:23:39 STEP 1659 H=1659:00 +CPU= 38.168
  17:24:03 STEP 1660 H=1660:00 +CPU= 23.776
  17:24:25 STEP 1661 H=1661:00 +CPU= 22.077
  17:25:06 STEP 1662 H=1662:00 +CPU= 41.040
  17:25:28 STEP 1663 H=1663:00 +CPU= 22.293
  17:25:51 STEP 1664 H=1664:00 +CPU= 22.997
  17:26:28 STEP 1665 H=1665:00 +CPU= 36.732
  17:26:51 STEP 1666 H=1666:00 +CPU= 22.938
  17:27:13 STEP 1667 H=1667:00 +CPU= 22.355
  17:27:57 STEP 1668 H=1668:00 +CPU= 43.605
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001668
  17:28:19 STEP 1669 H=1669:00 +CPU= 22.114
  17:28:46 STEP 1670 H=1670:00 +CPU= 26.425
  17:29:23 STEP 1671 H=1671:00 +CPU= 37.185
  17:29:46 STEP 1672 H=1672:00 +CPU= 22.655
  17:30:12 STEP 1673 H=1673:00 +CPU= 25.860
  17:30:48 STEP 1674 H=1674:00 +CPU= 35.989
  17:31:10 STEP 1675 H=1675:00 +CPU= 22.125
  17:31:39 STEP 1676 H=1676:00 +CPU= 28.529
  17:32:13 STEP 1677 H=1677:00 +CPU= 33.921
  17:32:35 STEP 1678 H=1678:00 +CPU= 22.812
  17:33:03 STEP 1679 H=1679:00 +CPU= 27.954
  17:33:44 STEP 1680 H=1680:00 +CPU= 40.042
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_58.zip
Uploading the intermediate file: upload_file_58.zip
  17:34:08 STEP 1681 H=1681:00 +CPU= 24.332
  17:34:34 STEP 1682 H=1682:00 +CPU= 25.779
  17:35:08 STEP 1683 H=1683:00 +CPU= 34.114
  17:35:35 STEP 1684 H=1684:00 +CPU= 26.710
  17:35:58 STEP 1685 H=1685:00 +CPU= 23.333
  17:36:33 STEP 1686 H=1686:00 +CPU= 34.821
  17:37:01 STEP 1687 H=1687:00 +CPU= 27.387
  17:37:24 STEP 1688 H=1688:00 +CPU= 22.958
  17:37:58 STEP 1689 H=1689:00 +CPU= 34.059
  17:38:27 STEP 1690 H=1690:00 +CPU= 28.566
  17:38:49 STEP 1691 H=1691:00 +CPU= 22.243
  17:39:29 STEP 1692 H=1692:00 +CPU= 39.555
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001692
  17:39:55 STEP 1693 H=1693:00 +CPU= 26.382
  17:40:18 STEP 1694 H=1694:00 +CPU= 22.578
  17:40:55 STEP 1695 H=1695:00 +CPU= 37.426
  17:41:21 STEP 1696 H=1696:00 +CPU= 25.357
  17:41:43 STEP 1697 H=1697:00 +CPU= 22.083
  17:42:23 STEP 1698 H=1698:00 +CPU= 39.442
  17:42:45 STEP 1699 H=1699:00 +CPU= 22.280
  17:43:08 STEP 1700 H=1700:00 +CPU= 22.783
  17:43:48 STEP 1701 H=1701:00 +CPU= 39.733
  17:44:11 STEP 1702 H=1702:00 +CPU= 23.187
  17:44:34 STEP 1703 H=1703:00 +CPU= 22.895
  17:45:16 STEP 1704 H=1704:00 +CPU= 42.373
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_59.zip
Uploading the intermediate file: upload_file_59.zip
  17:45:39 STEP 1705 H=1705:00 +CPU= 22.464
  17:46:03 STEP 1706 H=1706:00 +CPU= 24.030
  17:46:40 STEP 1707 H=1707:00 +CPU= 36.862
  17:47:03 STEP 1708 H=1708:00 +CPU= 22.930
  17:47:28 STEP 1709 H=1709:00 +CPU= 24.685
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0568
(argv4) batchid: 988
(argv5) wuid: 12205212
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_27f419f4b165abbaf73a14b1a0bb5fa9 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0568_2019050100_123_988_12205212.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0568_2019050100_123_988_12205212.zip
ic_ancil_file: ic_ancil_12205212
ifsdata_file: ifsdata_12205212
climate_data_file: clim_data_12205212
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_13638bc7c4e6293693e3d09fb3af2cda to: /var/lib/boinc-client/slots/0/ic_ancil_12205212.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12205212.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f96f8520aae04e3bb7484c5c59d96365 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205212.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205212.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205212.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205212.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12205212.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12205212.xml
-- Model is restarting --
Adjusting last_iter, 1709, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12205212.xml
last_cpu_time: 44873
upload_file_number: 60
last_iter: 1703
last_upload: 6134400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861
The child process has been launched with process id: 7642
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7642:7642] [20230217:174843:1676684923:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:49:26 STEP 1704 H=1704:00 +CPU= 39.739
  17:49:48 STEP 1705 H=1705:00 +CPU= 22.016
  17:50:16 STEP 1706 H=1706:00 +CPU= 27.805
  17:50:51 STEP 1707 H=1707:00 +CPU= 34.083
  17:51:15 STEP 1708 H=1708:00 +CPU= 24.226
  17:51:40 STEP 1709 H=1709:00 +CPU= 25.320
  17:52:15 STEP 1710 H=1710:00 +CPU= 34.667
  17:52:39 STEP 1711 H=1711:00 +CPU= 23.691
  17:53:04 STEP 1712 H=1712:00 +CPU= 25.600
  17:53:38 STEP 1713 H=1713:00 +CPU= 34.193
  17:54:05 STEP 1714 H=1714:00 +CPU= 26.265
  17:54:29 STEP 1715 H=1715:00 +CPU= 23.778
  17:55:07 STEP 1716 H=1716:00 +CPU= 37.803
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001716
  17:55:31 STEP 1717 H=1717:00 +CPU= 24.108
  17:55:54 STEP 1718 H=1718:00 +CPU= 22.883
  17:56:30 STEP 1719 H=1719:00 +CPU= 35.908
  17:56:57 STEP 1720 H=1720:00 +CPU= 27.103
  17:57:19 STEP 1721 H=1721:00 +CPU= 22.122
  17:57:56 STEP 1722 H=1722:00 +CPU= 37.362
  17:58:20 STEP 1723 H=1723:00 +CPU= 23.546
  17:58:43 STEP 1724 H=1724:00 +CPU= 23.008
  17:59:20 STEP 1725 H=1725:00 +CPU= 36.854
  17:59:43 STEP 1726 H=1726:00 +CPU= 22.643
  18:00:06 STEP 1727 H=1727:00 +CPU= 23.014
  18:00:52 STEP 1728 H=1728:00 +CPU= 45.576
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_60.zip
Uploading the intermediate file: upload_file_60.zip
  18:01:14 STEP 1729 H=1729:00 +CPU= 22.374
  18:01:38 STEP 1730 H=1730:00 +CPU= 23.278
  18:02:16 STEP 1731 H=1731:00 +CPU= 38.798
  18:02:39 STEP 1732 H=1732:00 +CPU= 22.630
  18:03:02 STEP 1733 H=1733:00 +CPU= 23.152
  18:03:40 STEP 1734 H=1734:00 +CPU= 37.798
  18:04:02 STEP 1735 H=1735:00 +CPU= 21.934
  18:04:27 STEP 1736 H=1736:00 +CPU= 25.011
  18:05:04 STEP 1737 H=1737:00 +CPU= 36.780
  18:05:27 STEP 1738 H=1738:00 +CPU= 22.516
  18:05:53 STEP 1739 H=1739:00 +CPU= 26.271
  18:06:33 STEP 1740 H=1740:00 +CPU= 39.484
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001740
  18:06:55 STEP 1741 H=1741:00 +CPU= 21.997
  18:07:22 STEP 1742 H=1742:00 +CPU= 27.020
  18:07:56 STEP 1743 H=1743:00 +CPU= 33.922
  18:08:18 STEP 1744 H=1744:00 +CPU= 22.722
  18:08:45 STEP 1745 H=1745:00 +CPU= 27.019
  18:09:20 STEP 1746 H=1746:00 +CPU= 34.794
  18:09:43 STEP 1747 H=1747:00 +CPU= 22.731
  18:10:10 STEP 1748 H=1748:00 +CPU= 27.187
  18:10:44 STEP 1749 H=1749:00 +CPU= 33.992
  18:11:09 STEP 1750 H=1750:00 +CPU= 24.481
  18:11:35 STEP 1751 H=1751:00 +CPU= 25.982
  18:12:15 STEP 1752 H=1752:00 +CPU= 40.237
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_61.zip
Uploading the intermediate file: upload_file_61.zip
  18:12:39 STEP 1753 H=1753:00 +CPU= 23.953
  18:13:02 STEP 1754 H=1754:00 +CPU= 22.390
  18:13:37 STEP 1755 H=1755:00 +CPU= 35.519
  18:14:04 STEP 1756 H=1756:00 +CPU= 26.443
  18:14:26 STEP 1757 H=1757:00 +CPU= 22.298
  18:15:05 STEP 1758 H=1758:00 +CPU= 38.542
  18:15:29 STEP 1759 H=1759:00 +CPU= 24.310
  18:15:52 STEP 1760 H=1760:00 +CPU= 22.697
  18:16:28 STEP 1761 H=1761:00 +CPU= 36.289
  18:16:51 STEP 1762 H=1762:00 +CPU= 22.482
  18:17:13 STEP 1763 H=1763:00 +CPU= 22.003
  18:17:56 STEP 1764 H=1764:00 +CPU= 42.851
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001764
  18:18:18 STEP 1765 H=1765:00 +CPU= 21.975
  18:18:41 STEP 1766 H=1766:00 +CPU= 22.650
  18:19:17 STEP 1767 H=1767:00 +CPU= 36.145
  18:19:39 STEP 1768 H=1768:00 +CPU= 22.399
  18:20:02 STEP 1769 H=1769:00 +CPU= 23.031
  18:20:41 STEP 1770 H=1770:00 +CPU= 38.429
  18:21:03 STEP 1771 H=1771:00 +CPU= 21.976
  18:21:29 STEP 1772 H=1772:00 +CPU= 26.160
  18:22:05 STEP 1773 H=1773:00 +CPU= 36.012
  18:22:27 STEP 1774 H=1774:00 +CPU= 22.313
  18:22:55 STEP 1775 H=1775:00 +CPU= 27.202
  18:23:36 STEP 1776 H=1776:00 +CPU= 41.214
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_62.zip
Uploading the intermediate file: upload_file_62.zip
  18:23:58 STEP 1777 H=1777:00 +CPU= 22.201
  18:24:26 STEP 1778 H=1778:00 +CPU= 27.352
  18:25:00 STEP 1779 H=1779:00 +CPU= 33.886
  18:25:23 STEP 1780 H=1780:00 +CPU= 23.025
  18:25:49 STEP 1781 H=1781:00 +CPU= 26.065
  18:26:24 STEP 1782 H=1782:00 +CPU= 34.644
  18:26:47 STEP 1783 H=1783:00 +CPU= 23.278
  18:27:14 STEP 1784 H=1784:00 +CPU= 26.179
  18:27:48 STEP 1785 H=1785:00 +CPU= 34.040
  18:28:14 STEP 1786 H=1786:00 +CPU= 25.683
  18:28:38 STEP 1787 H=1787:00 +CPU= 24.623
  18:29:16 STEP 1788 H=1788:00 +CPU= 37.889
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001788
  18:29:41 STEP 1789 H=1789:00 +CPU= 24.394
  18:30:03 STEP 1790 H=1790:00 +CPU= 22.722
  18:30:38 STEP 1791 H=1791:00 +CPU= 34.598
  18:31:04 STEP 1792 H=1792:00 +CPU= 25.364
  18:31:26 STEP 1793 H=1793:00 +CPU= 22.031
  18:32:03 STEP 1794 H=1794:00 +CPU= 37.290
  18:32:28 STEP 1795 H=1795:00 +CPU= 25.400
  18:32:51 STEP 1796 H=1796:00 +CPU= 22.587
  18:33:30 STEP 1797 H=1797:00 +CPU= 38.438
  18:33:53 STEP 1798 H=1798:00 +CPU= 23.800
  18:34:16 STEP 1799 H=1799:00 +CPU= 23.007
  18:35:02 STEP 1800 H=1800:00 +CPU= 44.907
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_63.zip
Uploading the intermediate file: upload_file_63.zip
  18:35:24 STEP 1801 H=1801:00 +CPU= 22.070
  18:35:46 STEP 1802 H=1802:00 +CPU= 22.642
  18:36:27 STEP 1803 H=1803:00 +CPU= 40.155
  18:36:49 STEP 1804 H=1804:00 +CPU= 22.416
  18:37:11 STEP 1805 H=1805:00 +CPU= 21.999
  18:37:52 STEP 1806 H=1806:00 +CPU= 40.980
  18:38:14 STEP 1807 H=1807:00 +CPU= 21.764
  18:38:36 STEP 1808 H=1808:00 +CPU= 22.477
  18:39:16 STEP 1809 H=1809:00 +CPU= 39.664
  18:39:39 STEP 1810 H=1810:00 +CPU= 22.327
  18:40:01 STEP 1811 H=1811:00 +CPU= 22.397
  18:40:44 STEP 1812 H=1812:00 +CPU= 42.841
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001812
  18:41:06 STEP 1813 H=1813:00 +CPU= 21.756
  18:41:31 STEP 1814 H=1814:00 +CPU= 25.339
  18:42:08 STEP 1815 H=1815:00 +CPU= 36.335
  18:42:30 STEP 1816 H=1816:00 +CPU= 22.281
  18:42:56 STEP 1817 H=1817:00 +CPU= 25.503
  18:43:32 STEP 1818 H=1818:00 +CPU= 36.069
  18:43:54 STEP 1819 H=1819:00 +CPU= 21.795
  18:44:19 STEP 1820 H=1820:00 +CPU= 25.345
  18:44:53 STEP 1821 H=1821:00 +CPU= 34.023
  18:45:18 STEP 1822 H=1822:00 +CPU= 24.403
  18:45:43 STEP 1823 H=1823:00 +CPU= 25.525
  18:46:24 STEP 1824 H=1824:00 +CPU= 40.573
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_64.zip
Uploading the intermediate file: upload_file_64.zip
  18:46:48 STEP 1825 H=1825:00 +CPU= 23.760
  18:47:10 STEP 1826 H=1826:00 +CPU= 22.595
  18:47:46 STEP 1827 H=1827:00 +CPU= 35.100
  18:48:13 STEP 1828 H=1828:00 +CPU= 27.366
  18:48:35 STEP 1829 H=1829:00 +CPU= 22.215
  18:49:13 STEP 1830 H=1830:00 +CPU= 37.057
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0568
(argv4) batchid: 988
(argv5) wuid: 12205212
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_27f419f4b165abbaf73a14b1a0bb5fa9 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0568_2019050100_123_988_12205212.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0568_2019050100_123_988_12205212.zip
ic_ancil_file: ic_ancil_12205212
ifsdata_file: ifsdata_12205212
climate_data_file: clim_data_12205212
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_13638bc7c4e6293693e3d09fb3af2cda to: /var/lib/boinc-client/slots/0/ic_ancil_12205212.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12205212.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f96f8520aae04e3bb7484c5c59d96365 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205212.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205212.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205212.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205212.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12205212.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12205212.xml
-- Model is restarting --
Adjusting last_iter, 1831, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12205212.xml
last_cpu_time: 48494
upload_file_number: 65
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861
The child process has been launched with process id: 8845
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8845:8845] [20230217:185030:1676688630:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:51:13 STEP 1824 H=1824:00 +CPU= 40.010
  18:51:36 STEP 1825 H=1825:00 +CPU= 22.467
  18:52:04 STEP 1826 H=1826:00 +CPU= 27.897
  18:52:38 STEP 1827 H=1827:00 +CPU= 34.193
  18:53:02 STEP 1828 H=1828:00 +CPU= 23.762
  18:53:27 STEP 1829 H=1829:00 +CPU= 25.273
  18:54:02 STEP 1830 H=1830:00 +CPU= 34.973
  18:54:28 STEP 1831 H=1831:00 +CPU= 25.737
  18:54:54 STEP 1832 H=1832:00 +CPU= 25.808
  18:55:28 STEP 1833 H=1833:00 +CPU= 34.039
  18:55:55 STEP 1834 H=1834:00 +CPU= 27.314
  18:56:18 STEP 1835 H=1835:00 +CPU= 22.975
  18:56:56 STEP 1836 H=1836:00 +CPU= 37.816
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001836
  18:57:22 STEP 1837 H=1837:00 +CPU= 26.207
  18:57:45 STEP 1838 H=1838:00 +CPU= 22.945
  18:58:20 STEP 1839 H=1839:00 +CPU= 34.686
  18:58:47 STEP 1840 H=1840:00 +CPU= 26.807
  18:59:09 STEP 1841 H=1841:00 +CPU= 22.029
  18:59:45 STEP 1842 H=1842:00 +CPU= 36.275
  19:00:12 STEP 1843 H=1843:00 +CPU= 26.401
  19:00:35 STEP 1844 H=1844:00 +CPU= 22.898
  19:01:13 STEP 1845 H=1845:00 +CPU= 38.088
  19:01:38 STEP 1846 H=1846:00 +CPU= 24.824
  19:02:01 STEP 1847 H=1847:00 +CPU= 23.042
  19:02:47 STEP 1848 H=1848:00 +CPU= 45.088
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205212/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0568_2019050100_123_988_12205212_1_r47383861_65.zip
Uploading the intermediate file: upload_file_65.zip
  19:03:09 STEP 1849 H=1849:00 +CPU= 22.418
  19:03:32 STEP 1850 H=1850:00 +CPU= 22.862
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:48:27 1509518 22302609 oifs_43r3_ps_0568_2019050100_123_988_12205212_1 4,489,200 26,681 0.0059
09 May 2023 16:48:25 1509518 22302609 oifs_43r3_ps_0568_2019050100_123_988_12205212_1 3,625,200 20,139 0.0056
09 May 2023 16:48:23 1509518 22302609 oifs_43r3_ps_0568_2019050100_123_988_12205212_1 2,761,200 13,594 0.0049
09 May 2023 16:48:20 1509518 22302609 oifs_43r3_ps_0568_2019050100_123_988_12205212_1 1,897,200 7,189 0.0038
09 May 2023 16:47:10 1509518 22302609 oifs_43r3_ps_0568_2019050100_123_988_12205212_1 864,000 8,693 0.0101


©2024 cpdn.org