climateprediction.net home page
Task 22296787

Task 22296787

Name oifs_43r3_ps_0625_2018050100_123_987_12204269_2
Workunit 12204269
Created 10 Jan 2023, 22:19:11 UTC
Sent 11 Jan 2023, 0:31:31 UTC
Report deadline 10 Feb 2023, 0:31:31 UTC
Received 13 Jan 2023, 6:12:02 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1534673
Run time 21 hours 48 min 57 sec
CPU time 13 hours 8 min 5 sec
Validate state Invalid
Credit 9,434.88
Device peak FLOPS 4.67 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,399.86 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,548.74 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
NC client, suspending the child process
Resuming the child process
  22:54:04 STEP 1547 H=1547:00 +CPU= 33.391
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:55:17 STEP 1548 H=1548:00 +CPU= 58.224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001548
  22:55:52 STEP 1549 H=1549:00 +CPU= 33.034
  22:56:29 STEP 1550 H=1550:00 +CPU= 34.785
  22:57:21 STEP 1551 H=1551:00 +CPU= 48.696
  22:57:56 STEP 1552 H=1552:00 +CPU= 31.618
  22:58:30 STEP 1553 H=1553:00 +CPU= 31.466
  22:59:25 STEP 1554 H=1554:00 +CPU= 51.352
  22:59:59 STEP 1555 H=1555:00 +CPU= 31.358
  23:00:35 STEP 1556 H=1556:00 +CPU= 32.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:01:41 STEP 1557 H=1557:00 +CPU= 50.251
  23:02:16 STEP 1558 H=1558:00 +CPU= 32.496
  23:02:55 STEP 1559 H=1559:00 +CPU= 35.838
  23:03:58 STEP 1560 H=1560:00 +CPU= 58.753
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_33.zip
Uploading the intermediate file: upload_file_33.zip
  23:04:32 STEP 1561 H=1561:00 +CPU= 30.896
  23:05:06 STEP 1562 H=1562:00 +CPU= 31.972
  23:05:59 STEP 1563 H=1563:00 +CPU= 49.005
  23:06:34 STEP 1564 H=1564:00 +CPU= 32.407
  23:07:09 STEP 1565 H=1565:00 +CPU= 32.386
  23:08:03 STEP 1566 H=1566:00 +CPU= 49.689
  23:08:36 STEP 1567 H=1567:00 +CPU= 30.264
  23:09:09 STEP 1568 H=1568:00 +CPU= 30.703
  23:10:03 STEP 1569 H=1569:00 +CPU= 49.716
  23:10:38 STEP 1570 H=1570:00 +CPU= 32.472
  23:11:12 STEP 1571 H=1571:00 +CPU= 31.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:12:22 STEP 1572 H=1572:00 +CPU= 55.087
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001572
  23:13:01 STEP 1573 H=1573:00 +CPU= 36.221
  23:13:36 STEP 1574 H=1574:00 +CPU= 32.703
  23:14:28 STEP 1575 H=1575:00 +CPU= 48.109
  23:15:03 STEP 1576 H=1576:00 +CPU= 31.752
  23:15:36 STEP 1577 H=1577:00 +CPU= 30.518
  23:16:30 STEP 1578 H=1578:00 +CPU= 50.334
  23:17:04 STEP 1579 H=1579:00 +CPU= 31.685
  23:17:39 STEP 1580 H=1580:00 +CPU= 32.275
  23:18:34 STEP 1581 H=1581:00 +CPU= 51.810
  23:19:10 STEP 1582 H=1582:00 +CPU= 32.910
  23:19:45 STEP 1583 H=1583:00 +CPU= 32.387
  23:20:47 STEP 1584 H=1584:00 +CPU= 57.265
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_34.zip
Uploading the intermediate file: upload_file_34.zip
  23:21:21 STEP 1585 H=1585:00 +CPU= 31.330
  23:21:54 STEP 1586 H=1586:00 +CPU= 31.280
  23:22:47 STEP 1587 H=1587:00 +CPU= 48.823
  23:23:22 STEP 1588 H=1588:00 +CPU= 32.269
  23:23:57 STEP 1589 H=1589:00 +CPU= 31.580
  23:24:50 STEP 1590 H=1590:00 +CPU= 49.222
  23:25:24 STEP 1591 H=1591:00 +CPU= 31.769
  23:25:58 STEP 1592 H=1592:00 +CPU= 31.735
  23:26:51 STEP 1593 H=1593:00 +CPU= 49.474
  23:27:26 STEP 1594 H=1594:00 +CPU= 32.062
  23:27:59 STEP 1595 H=1595:00 +CPU= 30.464
  23:28:58 STEP 1596 H=1596:00 +CPU= 54.729
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001596
  23:29:32 STEP 1597 H=1597:00 +CPU= 30.994
  23:30:06 STEP 1598 H=1598:00 +CPU= 32.478
  23:31:00 STEP 1599 H=1599:00 +CPU= 49.482
  23:31:34 STEP 1600 H=1600:00 +CPU= 31.912
  23:32:07 STEP 1601 H=1601:00 +CPU= 30.675
  23:33:05 STEP 1602 H=1602:00 +CPU= 55.285
  23:33:42 STEP 1603 H=1603:00 +CPU= 35.260
  23:34:17 STEP 1604 H=1604:00 +CPU= 33.503
  23:35:13 STEP 1605 H=1605:00 +CPU= 53.563
  23:35:50 STEP 1606 H=1606:00 +CPU= 35.480
  23:36:28 STEP 1607 H=1607:00 +CPU= 35.912
  23:37:33 STEP 1608 H=1608:00 +CPU= 61.881
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_35.zip
Uploading the intermediate file: upload_file_35.zip
  23:38:07 STEP 1609 H=1609:00 +CPU= 32.696
  23:38:44 STEP 1610 H=1610:00 +CPU= 35.119
  23:39:39 STEP 1611 H=1611:00 +CPU= 52.325
  23:40:15 STEP 1612 H=1612:00 +CPU= 34.507
  23:40:51 STEP 1613 H=1613:00 +CPU= 33.769
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:41:56 STEP 1614 H=1614:00 +CPU= 52.426
  23:42:31 STEP 1615 H=1615:00 +CPU= 33.499
  23:43:09 STEP 1616 H=1616:00 +CPU= 35.904
  23:44:05 STEP 1617 H=1617:00 +CPU= 54.143
  23:44:41 STEP 1618 H=1618:00 +CPU= 32.877
  23:45:14 STEP 1619 H=1619:00 +CPU= 31.269
  23:46:12 STEP 1620 H=1620:00 +CPU= 55.146
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001620
  23:46:45 STEP 1621 H=1621:00 +CPU= 30.482
  23:47:18 STEP 1622 H=1622:00 +CPU= 31.648
  23:48:11 STEP 1623 H=1623:00 +CPU= 49.806
  23:48:44 STEP 1624 H=1624:00 +CPU= 31.427
  23:49:19 STEP 1625 H=1625:00 +CPU= 34.086
  23:50:12 STEP 1626 H=1626:00 +CPU= 49.885
  23:50:45 STEP 1627 H=1627:00 +CPU= 30.941
  23:51:19 STEP 1628 H=1628:00 +CPU= 31.949
  23:52:10 STEP 1629 H=1629:00 +CPU= 48.881
  23:52:43 STEP 1630 H=1630:00 +CPU= 31.179
  23:53:17 STEP 1631 H=1631:00 +CPU= 31.398
  23:54:21 STEP 1632 H=1632:00 +CPU= 60.451
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_36.zip
Uploading the intermediate file: upload_file_36.zip
Uploading trickle at timestep: 5875200
  23:54:54 STEP 1633 H=1633:00 +CPU= 31.697
  23:55:29 STEP 1634 H=1634:00 +CPU= 32.584
  23:56:22 STEP 1635 H=1635:00 +CPU= 49.572
  23:56:57 STEP 1636 H=1636:00 +CPU= 32.770
  23:57:32 STEP 1637 H=1637:00 +CPU= 31.916
  23:58:26 STEP 1638 H=1638:00 +CPU= 50.685
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0625
(argv4) batchid: 987
(argv5) wuid: 12204269
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_be72bd4a10f5b751cb0baaa421dc53d9 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0625_2018050100_123_987_12204269.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0625_2018050100_123_987_12204269.zip
ic_ancil_file: ic_ancil_12204269
ifsdata_file: ifsdata_12204269
climate_data_file: clim_data_12204269
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_21a5a86d5205f21a37b70596279cc24b to: /var/lib/boinc-client/slots/5/ic_ancil_12204269.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12204269.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_78825b502bbae3070cf3e0b5749ec435 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12204269.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12204269.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12204269.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12204269.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12204269.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12204269.xml
-- Model is restarting --
Adjusting last_iter, 1639, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12204269.xml
last_cpu_time: 35685
upload_file_number: 37
last_iter: 1631
last_upload: 5875200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059
The child process has been launched with process id: 135121
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:135121:135121] [20230113:060005:1673560805:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:01:15 STEP 1632 H=1632:00 +CPU= 60.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:02:15 STEP 1633 H=1633:00 +CPU= 35.243
  06:02:51 STEP 1634 H=1634:00 +CPU= 33.430
  06:03:45 STEP 1635 H=1635:00 +CPU= 50.916
  06:04:21 STEP 1636 H=1636:00 +CPU= 32.952
  06:04:57 STEP 1637 H=1637:00 +CPU= 33.269
  06:05:54 STEP 1638 H=1638:00 +CPU= 53.348
  06:06:29 STEP 1639 H=1639:00 +CPU= 32.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:07:17 STEP 1640 H=1640:00 +CPU= 33.774
  06:08:16 STEP 1641 H=1641:00 +CPU= 54.517
  06:08:56 STEP 1642 H=1642:00 +CPU= 38.234
  06:09:31 STEP 1643 H=1643:00 +CPU= 34.230
  06:10:38 STEP 1644 H=1644:00 +CPU= 64.373
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001644
  06:11:18 STEP 1645 H=1645:00 +CPU= 38.970
  06:11:58 STEP 1646 H=1646:00 +CPU= 38.774
  06:13:00 STEP 1647 H=1647:00 +CPU= 59.220
  06:13:38 STEP 1648 H=1648:00 +CPU= 37.198
  06:14:17 STEP 1649 H=1649:00 +CPU= 37.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:15:26 STEP 1650 H=1650:00 +CPU= 55.720
  06:16:05 STEP 1651 H=1651:00 +CPU= 37.414
  06:16:43 STEP 1652 H=1652:00 +CPU= 36.688
  06:17:40 STEP 1653 H=1653:00 +CPU= 55.014
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:20:29 STEP 1654 H=1654:00 +CPU= 35.849
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:21:18 STEP 1655 H=1655:00 +CPU= 36.853
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:22:44 STEP 1656 H=1656:00 +CPU= 69.688
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_37.zip
Uploading the intermediate file: upload_file_37.zip
  06:23:23 STEP 1657 H=1657:00 +CPU= 37.999
  06:24:04 STEP 1658 H=1658:00 +CPU= 40.101
  06:25:04 STEP 1659 H=1659:00 +CPU= 56.565
  06:25:42 STEP 1660 H=1660:00 +CPU= 37.603
  06:26:18 STEP 1661 H=1661:00 +CPU= 34.680
  06:27:18 STEP 1662 H=1662:00 +CPU= 57.602
  06:27:57 STEP 1663 H=1663:00 +CPU= 38.543
  06:28:39 STEP 1664 H=1664:00 +CPU= 39.982
  06:29:39 STEP 1665 H=1665:00 +CPU= 59.193
  06:30:18 STEP 1666 H=1666:00 +CPU= 37.222
  06:30:55 STEP 1667 H=1667:00 +CPU= 36.348
  06:32:02 STEP 1668 H=1668:00 +CPU= 64.512
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001668
  06:32:40 STEP 1669 H=1669:00 +CPU= 37.624
  06:33:19 STEP 1670 H=1670:00 +CPU= 37.345
  06:34:16 STEP 1671 H=1671:00 +CPU= 55.749
  06:34:53 STEP 1672 H=1672:00 +CPU= 35.427
  06:35:29 STEP 1673 H=1673:00 +CPU= 34.591
  06:36:26 STEP 1674 H=1674:00 +CPU= 55.955
  06:37:03 STEP 1675 H=1675:00 +CPU= 35.640
  06:37:42 STEP 1676 H=1676:00 +CPU= 37.038
  06:38:39 STEP 1677 H=1677:00 +CPU= 55.108
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:39:27 STEP 1678 H=1678:00 +CPU= 35.186
  06:40:05 STEP 1679 H=1679:00 +CPU= 37.359
  06:41:13 STEP 1680 H=1680:00 +CPU= 65.385
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_38.zip
Uploading the intermediate file: upload_file_38.zip
  06:41:50 STEP 1681 H=1681:00 +CPU= 36.141
  06:42:30 STEP 1682 H=1682:00 +CPU= 38.536
  06:43:26 STEP 1683 H=1683:00 +CPU= 54.986
  06:44:04 STEP 1684 H=1684:00 +CPU= 36.757
  06:44:40 STEP 1685 H=1685:00 +CPU= 34.710
  06:45:36 STEP 1686 H=1686:00 +CPU= 54.624
  06:46:11 STEP 1687 H=1687:00 +CPU= 34.216
  06:46:48 STEP 1688 H=1688:00 +CPU= 36.129
  06:47:44 STEP 1689 H=1689:00 +CPU= 53.539
  06:48:21 STEP 1690 H=1690:00 +CPU= 35.843
  06:48:55 STEP 1691 H=1691:00 +CPU= 33.330
  06:49:57 STEP 1692 H=1692:00 +CPU= 60.475
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001692
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:50:44 STEP 1693 H=1693:00 +CPU= 34.496
  06:51:21 STEP 1694 H=1694:00 +CPU= 35.611
  06:52:18 STEP 1695 H=1695:00 +CPU= 54.794
  06:52:55 STEP 1696 H=1696:00 +CPU= 36.085
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:53:51 STEP 1697 H=1697:00 +CPU= 34.189
  06:54:48 STEP 1698 H=1698:00 +CPU= 55.403
  06:55:24 STEP 1699 H=1699:00 +CPU= 34.499
  06:56:02 STEP 1700 H=1700:00 +CPU= 36.266
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:02:01 STEP 1701 H=1701:00 +CPU= 52.827
  07:02:39 STEP 1702 H=1702:00 +CPU= 36.463
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:03:30 STEP 1703 H=1703:00 +CPU= 37.413
  07:04:35 STEP 1704 H=1704:00 +CPU= 62.174
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_39.zip
Uploading the intermediate file: upload_file_39.zip
  07:05:11 STEP 1705 H=1705:00 +CPU= 33.616
  07:05:47 STEP 1706 H=1706:00 +CPU= 34.845
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:08:56 STEP 1707 H=1707:00 +CPU= 52.512
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:10:23 STEP 1708 H=1708:00 +CPU= 35.077
  07:10:57 STEP 1709 H=1709:00 +CPU= 32.184
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:16:58 STEP 1710 H=1710:00 +CPU= 53.644
  07:17:36 STEP 1711 H=1711:00 +CPU= 36.760
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:18:34 STEP 1712 H=1712:00 +CPU= 35.804
  07:19:30 STEP 1713 H=1713:00 +CPU= 54.010
  07:20:08 STEP 1714 H=1714:00 +CPU= 36.561
  07:20:44 STEP 1715 H=1715:00 +CPU= 34.915
  07:21:44 STEP 1716 H=1716:00 +CPU= 56.916
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001716
  07:22:18 STEP 1717 H=1717:00 +CPU= 31.522
  07:22:52 STEP 1718 H=1718:00 +CPU= 32.204
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:32:10 STEP 1719 H=1719:00 +CPU= 50.859
  07:32:48 STEP 1720 H=1720:00 +CPU= 36.435
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:35:26 STEP 1721 H=1721:00 +CPU= 33.559
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:37:14 STEP 1722 H=1722:00 +CPU= 55.024
  07:37:50 STEP 1723 H=1723:00 +CPU= 34.910
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:40:30 STEP 1724 H=1724:00 +CPU= 35.412
  07:41:28 STEP 1725 H=1725:00 +CPU= 55.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:44:29 STEP 1726 H=1726:00 +CPU= 36.548
  07:45:06 STEP 1727 H=1727:00 +CPU= 36.122
  07:46:15 STEP 1728 H=1728:00 +CPU= 65.708
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_40.zip
Uploading the intermediate file: upload_file_40.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:47:01 STEP 1729 H=1729:00 +CPU= 33.006
  07:47:34 STEP 1730 H=1730:00 +CPU= 30.547
  07:48:24 STEP 1731 H=1731:00 +CPU= 46.382
  07:48:57 STEP 1732 H=1732:00 +CPU= 31.069
  07:49:30 STEP 1733 H=1733:00 +CPU= 31.515
  07:50:22 STEP 1734 H=1734:00 +CPU= 48.168
  07:50:55 STEP 1735 H=1735:00 +CPU= 30.777
  07:51:30 STEP 1736 H=1736:00 +CPU= 33.338
  07:52:21 STEP 1737 H=1737:00 +CPU= 47.528
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:56:46 STEP 1738 H=1738:00 +CPU= 30.764
  07:57:18 STEP 1739 H=1739:00 +CPU= 29.903
  07:58:18 STEP 1740 H=1740:00 +CPU= 56.553
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001740
  07:58:52 STEP 1741 H=1741:00 +CPU= 31.909
  07:59:27 STEP 1742 H=1742:00 +CPU= 33.108
  08:00:20 STEP 1743 H=1743:00 +CPU= 49.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:08:51 STEP 1744 H=1744:00 +CPU= 32.664
  08:09:26 STEP 1745 H=1745:00 +CPU= 33.169
  08:10:20 STEP 1746 H=1746:00 +CPU= 50.347
  08:10:55 STEP 1747 H=1747:00 +CPU= 33.516
  08:11:32 STEP 1748 H=1748:00 +CPU= 35.288
  08:12:25 STEP 1749 H=1749:00 +CPU= 49.980
  08:13:00 STEP 1750 H=1750:00 +CPU= 32.689
  08:13:35 STEP 1751 H=1751:00 +CPU= 32.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:20:05 STEP 1752 H=1752:00 +CPU= 59.080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_41.zip
Uploading the intermediate file: upload_file_41.zip
  08:20:39 STEP 1753 H=1753:00 +CPU= 32.516
  08:21:14 STEP 1754 H=1754:00 +CPU= 32.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:26:42 STEP 1755 H=1755:00 +CPU= 49.788
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:27:50 STEP 1756 H=1756:00 +CPU= 33.810
  08:28:23 STEP 1757 H=1757:00 +CPU= 31.198
  08:29:15 STEP 1758 H=1758:00 +CPU= 49.795
  08:29:48 STEP 1759 H=1759:00 +CPU= 31.041
  08:30:22 STEP 1760 H=1760:00 +CPU= 33.352
  08:31:16 STEP 1761 H=1761:00 +CPU= 51.541
  08:31:50 STEP 1762 H=1762:00 +CPU= 33.839
  08:32:26 STEP 1763 H=1763:00 +CPU= 34.478
  08:33:27 STEP 1764 H=1764:00 +CPU= 58.792
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001764
  08:34:02 STEP 1765 H=1765:00 +CPU= 33.186
  08:34:37 STEP 1766 H=1766:00 +CPU= 33.938
  08:35:36 STEP 1767 H=1767:00 +CPU= 57.876
  08:36:14 STEP 1768 H=1768:00 +CPU= 36.324
  08:36:52 STEP 1769 H=1769:00 +CPU= 37.110
  08:37:50 STEP 1770 H=1770:00 +CPU= 56.591
  08:38:26 STEP 1771 H=1771:00 +CPU= 34.048
  08:39:02 STEP 1772 H=1772:00 +CPU= 35.030
  08:39:57 STEP 1773 H=1773:00 +CPU= 53.066
  08:40:33 STEP 1774 H=1774:00 +CPU= 34.673
  08:41:10 STEP 1775 H=1775:00 +CPU= 36.064
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:42:27 STEP 1776 H=1776:00 +CPU= 63.293
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_42.zip
Uploading the intermediate file: upload_file_42.zip
  08:43:02 STEP 1777 H=1777:00 +CPU= 33.767
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:43:50 STEP 1778 H=1778:00 +CPU= 35.990
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:45:38 STEP 1779 H=1779:00 +CPU= 53.664
  08:46:14 STEP 1780 H=1780:00 +CPU= 34.765
  08:46:50 STEP 1781 H=1781:00 +CPU= 34.375
  08:47:47 STEP 1782 H=1782:00 +CPU= 54.430
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:49:04 STEP 1783 H=1783:00 +CPU= 33.383
  08:49:40 STEP 1784 H=1784:00 +CPU= 33.925
  08:50:37 STEP 1785 H=1785:00 +CPU= 54.243
  08:51:13 STEP 1786 H=1786:00 +CPU= 33.408
  08:51:48 STEP 1787 H=1787:00 +CPU= 34.121
  08:52:49 STEP 1788 H=1788:00 +CPU= 58.016
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001788
  08:53:24 STEP 1789 H=1789:00 +CPU= 33.151
  08:54:01 STEP 1790 H=1790:00 +CPU= 35.235
  08:54:55 STEP 1791 H=1791:00 +CPU= 52.247
  08:55:33 STEP 1792 H=1792:00 +CPU= 35.646
  08:56:08 STEP 1793 H=1793:00 +CPU= 33.686
  08:57:07 STEP 1794 H=1794:00 +CPU= 56.040
  08:57:42 STEP 1795 H=1795:00 +CPU= 33.923
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:58:31 STEP 1796 H=1796:00 +CPU= 36.407
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:59:40 STEP 1797 H=1797:00 +CPU= 55.396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:00:48 STEP 1798 H=1798:00 +CPU= 34.821
  09:01:26 STEP 1799 H=1799:00 +CPU= 36.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:02:41 STEP 1800 H=1800:00 +CPU= 61.512
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_43.zip
Uploading the intermediate file: upload_file_43.zip
  09:03:17 STEP 1801 H=1801:00 +CPU= 34.152
  09:03:52 STEP 1802 H=1802:00 +CPU= 33.567
  09:04:47 STEP 1803 H=1803:00 +CPU= 52.268
  09:05:24 STEP 1804 H=1804:00 +CPU= 34.699
  09:06:00 STEP 1805 H=1805:00 +CPU= 34.995
  09:06:57 STEP 1806 H=1806:00 +CPU= 54.231
  09:07:30 STEP 1807 H=1807:00 +CPU= 30.775
  09:08:03 STEP 1808 H=1808:00 +CPU= 30.920
  09:08:55 STEP 1809 H=1809:00 +CPU= 48.352
  09:09:28 STEP 1810 H=1810:00 +CPU= 30.836
  09:10:01 STEP 1811 H=1811:00 +CPU= 30.402
  09:10:59 STEP 1812 H=1812:00 +CPU= 54.248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001812
  09:11:31 STEP 1813 H=1813:00 +CPU= 30.316
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            2  MAX ETADOT VERTICAL VEL.=   3.341659593234334E-007
  LON  =    177.500000000000       degrees
  LAT  =   -35.3269057618723       degrees
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:12:15 STEP 1814 H=1814:00 +CPU= 31.224
  09:13:06 STEP 1815 H=1815:00 +CPU= 47.660
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            3  MAX ETADOT VERTICAL VEL.=   4.335516291625997E-007
  LON  =    178.750000000000       degrees
  LAT  =   -36.4483927977945       degrees
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:14:01 STEP 1816 H=1816:00 +CPU= 30.987
  09:14:36 STEP 1817 H=1817:00 +CPU= 32.520
  09:15:28 STEP 1818 H=1818:00 +CPU= 49.185
  09:16:00 STEP 1819 H=1819:00 +CPU= 30.346
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            4  MAX ETADOT VERTICAL VEL.=   1.898447257395042E-007
  LON  =    180.000000000000       degrees
  LAT  =   -38.6913661222016       degrees
  09:16:39 STEP 1820 H=1820:00 +CPU= 36.655
  09:17:35 STEP 1821 H=1821:00 +CPU= 52.961
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            5  MAX ETADOT VERTICAL VEL.=   3.125846168893587E-008
  LON  =    178.750000000000       degrees
  LAT  =   -39.8128523737712       degrees
  09:18:10 STEP 1822 H=1822:00 +CPU= 32.082
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:19:04 STEP 1823 H=1823:00 +CPU= 30.866
  09:20:05 STEP 1824 H=1824:00 +CPU= 57.231
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_44.zip
Uploading the intermediate file: upload_file_44.zip
  09:20:36 STEP 1825 H=1825:00 +CPU= 28.974
  09:21:08 STEP 1826 H=1826:00 +CPU= 29.896
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           22  MAX ETADOT VERTICAL VEL.=   7.357212677974339E-008
  LON  =    177.187500000000       degrees
  LAT  =   -43.1773092288349       degrees
  09:21:57 STEP 1827 H=1827:00 +CPU= 45.394
  09:22:28 STEP 1828 H=1828:00 +CPU= 29.586
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:23:24 STEP 1829 H=1829:00 +CPU= 31.988
  09:24:15 STEP 1830 H=1830:00 +CPU= 47.684
  09:24:46 STEP 1831 H=1831:00 +CPU= 28.439
  09:25:18 STEP 1832 H=1832:00 +CPU= 30.058
  09:26:08 STEP 1833 H=1833:00 +CPU= 46.145
  09:26:40 STEP 1834 H=1834:00 +CPU= 29.972
  09:27:12 STEP 1835 H=1835:00 +CPU= 29.555
  09:28:07 STEP 1836 H=1836:00 +CPU= 51.069
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001836
  09:28:38 STEP 1837 H=1837:00 +CPU= 28.854
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:29:51 STEP 1838 H=1838:00 +CPU= 29.533
  09:30:41 STEP 1839 H=1839:00 +CPU= 46.097
  09:31:13 STEP 1840 H=1840:00 +CPU= 29.378
  09:31:44 STEP 1841 H=1841:00 +CPU= 28.790
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:33:17 STEP 1842 H=1842:00 +CPU= 46.848
  09:33:48 STEP 1843 H=1843:00 +CPU= 28.568
  09:34:19 STEP 1844 H=1844:00 +CPU= 28.826
  09:35:07 STEP 1845 H=1845:00 +CPU= 45.273
  09:35:39 STEP 1846 H=1846:00 +CPU= 29.245
  09:36:11 STEP 1847 H=1847:00 +CPU= 29.159
  09:37:08 STEP 1848 H=1848:00 +CPU= 52.680
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_45.zip
Uploading the intermediate file: upload_file_45.zip
  09:37:38 STEP 1849 H=1849:00 +CPU= 27.833
  09:38:09 STEP 1850 H=1850:00 +CPU= 28.520
  09:38:57 STEP 1851 H=1851:00 +CPU= 44.935
  09:39:28 STEP 1852 H=1852:00 +CPU= 29.081
  09:39:59 STEP 1853 H=1853:00 +CPU= 28.667
  09:40:48 STEP 1854 H=1854:00 +CPU= 45.704
  09:41:18 STEP 1855 H=1855:00 +CPU= 28.043
  09:41:50 STEP 1856 H=1856:00 +CPU= 28.967
  09:42:38 STEP 1857 H=1857:00 +CPU= 45.485
  09:43:09 STEP 1858 H=1858:00 +CPU= 28.633
  09:43:38 STEP 1859 H=1859:00 +CPU= 27.531
  09:44:32 STEP 1860 H=1860:00 +CPU= 50.095
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001860
  09:45:03 STEP 1861 H=1861:00 +CPU= 28.229
  09:45:38 STEP 1862 H=1862:00 +CPU= 33.375
  09:46:43 STEP 1863 H=1863:00 +CPU= 62.870
  09:47:27 STEP 1864 H=1864:00 +CPU= 42.366
  09:48:06 STEP 1865 H=1865:00 +CPU= 38.022
  09:49:11 STEP 1866 H=1866:00 +CPU= 63.631
  09:49:53 STEP 1867 H=1867:00 +CPU= 40.560
  09:50:34 STEP 1868 H=1868:00 +CPU= 39.513
  09:51:36 STEP 1869 H=1869:00 +CPU= 60.587
  09:52:19 STEP 1870 H=1870:00 +CPU= 41.385
  09:53:01 STEP 1871 H=1871:00 +CPU= 41.306
  09:54:13 STEP 1872 H=1872:00 +CPU= 69.242
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_46.zip
Uploading the intermediate file: upload_file_46.zip
Uploading trickle at timestep: 6739200
  09:54:54 STEP 1873 H=1873:00 +CPU= 40.498
  09:55:38 STEP 1874 H=1874:00 +CPU= 42.201
  09:56:40 STEP 1875 H=1875:00 +CPU= 60.289
  09:57:22 STEP 1876 H=1876:00 +CPU= 40.896
  09:58:04 STEP 1877 H=1877:00 +CPU= 40.445
  09:59:05 STEP 1878 H=1878:00 +CPU= 57.760
  09:59:47 STEP 1879 H=1879:00 +CPU= 41.604
  10:00:30 STEP 1880 H=1880:00 +CPU= 41.737
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:02:36 STEP 1881 H=1881:00 +CPU= 61.778
  10:03:20 STEP 1882 H=1882:00 +CPU= 42.196
  10:04:00 STEP 1883 H=1883:00 +CPU= 39.483
  10:05:11 STEP 1884 H=1884:00 +CPU= 68.475
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:06:02 STEP 1885 H=1885:00 +CPU= 39.181
  10:06:43 STEP 1886 H=1886:00 +CPU= 40.190
  10:07:44 STEP 1887 H=1887:00 +CPU= 58.644
  10:08:25 STEP 1888 H=1888:00 +CPU= 39.116
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0625
(argv4) batchid: 987
(argv5) wuid: 12204269
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_be72bd4a10f5b751cb0baaa421dc53d9 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0625_2018050100_123_987_12204269.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0625_2018050100_123_987_12204269.zip
ic_ancil_file: ic_ancil_12204269
ifsdata_file: ifsdata_12204269
climate_data_file: clim_data_12204269
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_21a5a86d5205f21a37b70596279cc24b to: /var/lib/boinc-client/slots/5/ic_ancil_12204269.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12204269.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_78825b502bbae3070cf3e0b5749ec435 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12204269.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12204269.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12204269.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12204269.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12204269.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12204269.xml
-- Model is restarting --
Adjusting last_iter, 1888, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12204269.xml
last_cpu_time: 46282
upload_file_number: 47
last_iter: 1871
last_upload: 6739200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059
The child process has been launched with process id: 172516
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:172516:172516] [20230113:124337:1673585017:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:44:42 STEP 1872 H=1872:00 +CPU= 59.028
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:52:19 STEP 1873 H=1873:00 +CPU= 31.638
  12:52:55 STEP 1874 H=1874:00 +CPU= 34.956
  12:53:50 STEP 1875 H=1875:00 +CPU= 52.797
  12:54:26 STEP 1876 H=1876:00 +CPU= 35.240
  12:54:58 STEP 1877 H=1877:00 +CPU= 31.654
  12:55:51 STEP 1878 H=1878:00 +CPU= 50.802
  12:56:24 STEP 1879 H=1879:00 +CPU= 31.484
  12:56:59 STEP 1880 H=1880:00 +CPU= 32.938
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:59:22 STEP 1881 H=1881:00 +CPU= 47.077
  12:59:53 STEP 1882 H=1882:00 +CPU= 28.885
  13:00:24 STEP 1883 H=1883:00 +CPU= 29.309
  13:01:19 STEP 1884 H=1884:00 +CPU= 51.925
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:02:22 STEP 1885 H=1885:00 +CPU= 29.528
  13:02:56 STEP 1886 H=1886:00 +CPU= 31.462
  13:03:47 STEP 1887 H=1887:00 +CPU= 48.266
  13:04:20 STEP 1888 H=1888:00 +CPU= 30.613
  13:04:53 STEP 1889 H=1889:00 +CPU= 30.934
  13:05:44 STEP 1890 H=1890:00 +CPU= 47.624
  13:06:16 STEP 1891 H=1891:00 +CPU= 29.436
  13:06:49 STEP 1892 H=1892:00 +CPU= 30.445
  13:07:40 STEP 1893 H=1893:00 +CPU= 46.852
  13:08:12 STEP 1894 H=1894:00 +CPU= 30.546
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:09:57 STEP 1895 H=1895:00 +CPU= 30.685
  13:10:58 STEP 1896 H=1896:00 +CPU= 58.543
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204269/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0625_2018050100_123_987_12204269_2_r1542460059_47.zip
Uploading the intermediate file: upload_file_47.zip
  13:11:32 STEP 1897 H=1897:00 +CPU= 31.639
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:17:30 1534673 22296787 oifs_43r3_ps_0625_2018050100_123_987_12204269_2 6,739,200 45,535 0.0068
09 May 2023 16:17:22 1534673 22296787 oifs_43r3_ps_0625_2018050100_123_987_12204269_2 5,875,200 35,431 0.0060
09 May 2023 16:17:22 1534673 22296787 oifs_43r3_ps_0625_2018050100_123_987_12204269_2 5,875,200 35,431 0.0060
09 May 2023 16:17:22 1534673 22296787 oifs_43r3_ps_0625_2018050100_123_987_12204269_2 5,011,200 25,913 0.0052
09 May 2023 16:16:05 1534673 22296787 oifs_43r3_ps_0625_2018050100_123_987_12204269_2 1,728,000 18,064 0.0105
09 May 2023 16:16:05 1534673 22296787 oifs_43r3_ps_0625_2018050100_123_987_12204269_2 864,000 9,095 0.0105
09 May 2023 16:16:05 1534673 22296787 oifs_43r3_ps_0625_2018050100_123_987_12204269_2 864,000 9,095 0.0105


©2024 cpdn.org