climateprediction.net home page
Task 22270851

Task 22270851

Name oifs_43r3_ps_0879_1999050100_123_968_12185523_0
Workunit 12185523
Created 22 Dec 2022, 19:57:59 UTC
Sent 22 Dec 2022, 20:04:45 UTC
Report deadline 21 Jan 2023, 20:04:45 UTC
Received 23 Dec 2022, 18:31:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1536225
Run time 19 hours 38 min 14 sec
CPU time 19 hours 31 min 28 sec
Validate state Invalid
Credit 7,741.44
Device peak FLOPS 3.85 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,416.07 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.45 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
teprediction.net/jf_1bcbd074e567f4ec25f4f1eb156e0221 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0879_1999050100_123_968_12185523.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0879_1999050100_123_968_12185523.zip
ic_ancil_file: ic_ancil_12185523
ifsdata_file: ifsdata_12185523
climate_data_file: clim_data_12185523
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2a05ea718d29723cc70740742bf8fed5 to: /var/lib/boinc-client/slots/3/ic_ancil_12185523.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12185523.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eccbb12003008c0fc38cf2bc2c03af45 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12185523.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12185523.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12185523.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12185523.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12185523.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12185523.xml
-- Model is restarting --
Adjusting last_iter, 1689, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12185523.xml
last_cpu_time: 56855
upload_file_number: 70
last_iter: 1679
last_upload: 6048000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996
The child process has been launched with process id: 84628
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:84628:84628] [20221223:135046:1671799846:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:51:43 STEP 1680 H=1680:00 +CPU= 53.762
  13:52:13 STEP 1681 H=1681:00 +CPU= 29.748
  13:52:43 STEP 1682 H=1682:00 +CPU= 30.499
  13:53:28 STEP 1683 H=1683:00 +CPU= 44.943
  13:53:53 STEP 1684 H=1684:00 +CPU= 24.432
  13:54:16 STEP 1685 H=1685:00 +CPU= 23.181
  13:55:03 STEP 1686 H=1686:00 +CPU= 46.687
  13:55:32 STEP 1687 H=1687:00 +CPU= 29.226
  13:56:04 STEP 1688 H=1688:00 +CPU= 31.080
  13:56:50 STEP 1689 H=1689:00 +CPU= 46.482
  13:57:21 STEP 1690 H=1690:00 +CPU= 30.672
  13:57:51 STEP 1691 H=1691:00 +CPU= 29.691
  13:58:41 STEP 1692 H=1692:00 +CPU= 50.058
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001692
  13:59:05 STEP 1693 H=1693:00 +CPU= 23.486
  13:59:29 STEP 1694 H=1694:00 +CPU= 24.574
  14:00:15 STEP 1695 H=1695:00 +CPU= 45.197
  14:00:45 STEP 1696 H=1696:00 +CPU= 30.350
  14:01:14 STEP 1697 H=1697:00 +CPU= 29.328
  14:02:03 STEP 1698 H=1698:00 +CPU= 48.156
  14:02:32 STEP 1699 H=1699:00 +CPU= 29.568
  14:03:03 STEP 1700 H=1700:00 +CPU= 30.501
  14:03:50 STEP 1701 H=1701:00 +CPU= 46.776
  14:04:16 STEP 1702 H=1702:00 +CPU= 26.327
  14:04:42 STEP 1703 H=1703:00 +CPU= 25.796
  14:05:34 STEP 1704 H=1704:00 +CPU= 51.760
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_70.zip
Uploading the intermediate file: upload_file_70.zip
  14:06:04 STEP 1705 H=1705:00 +CPU= 29.916
  14:06:36 STEP 1706 H=1706:00 +CPU= 31.152
  14:07:24 STEP 1707 H=1707:00 +CPU= 48.111
  14:07:54 STEP 1708 H=1708:00 +CPU= 30.440
  14:08:24 STEP 1709 H=1709:00 +CPU= 29.923
  14:09:08 STEP 1710 H=1710:00 +CPU= 43.623
  14:09:32 STEP 1711 H=1711:00 +CPU= 23.558
  14:09:57 STEP 1712 H=1712:00 +CPU= 25.235
  14:10:44 STEP 1713 H=1713:00 +CPU= 46.359
  14:11:15 STEP 1714 H=1714:00 +CPU= 31.128
  14:11:42 STEP 1715 H=1715:00 +CPU= 27.348
  14:12:24 STEP 1716 H=1716:00 +CPU= 41.261
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001716
  14:12:52 STEP 1717 H=1717:00 +CPU= 27.675
  14:13:22 STEP 1718 H=1718:00 +CPU= 30.703
  14:14:10 STEP 1719 H=1719:00 +CPU= 47.147
  14:14:41 STEP 1720 H=1720:00 +CPU= 30.698
  14:15:10 STEP 1721 H=1721:00 +CPU= 29.484
  14:15:59 STEP 1722 H=1722:00 +CPU= 48.815
  14:16:30 STEP 1723 H=1723:00 +CPU= 30.413
  14:16:59 STEP 1724 H=1724:00 +CPU= 29.294
  14:17:37 STEP 1725 H=1725:00 +CPU= 37.528
  14:18:02 STEP 1726 H=1726:00 +CPU= 24.782
  14:18:32 STEP 1727 H=1727:00 +CPU= 30.417
  14:19:25 STEP 1728 H=1728:00 +CPU= 52.827
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_71.zip
Uploading the intermediate file: upload_file_71.zip
  14:19:49 STEP 1729 H=1729:00 +CPU= 23.622
  14:20:13 STEP 1730 H=1730:00 +CPU= 24.066
  14:21:00 STEP 1731 H=1731:00 +CPU= 46.566
  14:21:31 STEP 1732 H=1732:00 +CPU= 30.776
  14:22:01 STEP 1733 H=1733:00 +CPU= 29.600
  14:22:49 STEP 1734 H=1734:00 +CPU= 48.272
  14:23:15 STEP 1735 H=1735:00 +CPU= 25.709
  14:23:39 STEP 1736 H=1736:00 +CPU= 23.934
  14:24:21 STEP 1737 H=1737:00 +CPU= 41.584
  14:24:51 STEP 1738 H=1738:00 +CPU= 30.736
  14:25:22 STEP 1739 H=1739:00 +CPU= 30.505
  14:26:09 STEP 1740 H=1740:00 +CPU= 46.499
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001740
  14:26:32 STEP 1741 H=1741:00 +CPU= 23.436
  14:26:58 STEP 1742 H=1742:00 +CPU= 25.816
  14:27:45 STEP 1743 H=1743:00 +CPU= 47.165
  14:28:16 STEP 1744 H=1744:00 +CPU= 30.691
  14:28:46 STEP 1745 H=1745:00 +CPU= 29.787
  14:29:30 STEP 1746 H=1746:00 +CPU= 43.418
  14:29:53 STEP 1747 H=1747:00 +CPU= 23.187
  14:30:18 STEP 1748 H=1748:00 +CPU= 25.171
  14:31:04 STEP 1749 H=1749:00 +CPU= 45.938
  14:31:36 STEP 1750 H=1750:00 +CPU= 31.064
  14:32:07 STEP 1751 H=1751:00 +CPU= 31.031
  14:33:00 STEP 1752 H=1752:00 +CPU= 53.535
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_72.zip
Uploading the intermediate file: upload_file_72.zip
  14:33:24 STEP 1753 H=1753:00 +CPU= 23.487
  14:33:48 STEP 1754 H=1754:00 +CPU= 24.004
  14:34:33 STEP 1755 H=1755:00 +CPU= 44.827
  14:35:04 STEP 1756 H=1756:00 +CPU= 30.795
  14:35:34 STEP 1757 H=1757:00 +CPU= 30.143
  14:36:23 STEP 1758 H=1758:00 +CPU= 48.544
  14:36:54 STEP 1759 H=1759:00 +CPU= 30.431
  14:37:25 STEP 1760 H=1760:00 +CPU= 30.875
  14:38:10 STEP 1761 H=1761:00 +CPU= 44.933
  14:38:34 STEP 1762 H=1762:00 +CPU= 24.419
  14:38:58 STEP 1763 H=1763:00 +CPU= 23.572
  14:39:49 STEP 1764 H=1764:00 +CPU= 51.189
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001764
  14:40:19 STEP 1765 H=1765:00 +CPU= 29.928
  14:40:50 STEP 1766 H=1766:00 +CPU= 30.672
  14:41:36 STEP 1767 H=1767:00 +CPU= 45.761
  14:42:00 STEP 1768 H=1768:00 +CPU= 24.109
  14:42:24 STEP 1769 H=1769:00 +CPU= 23.521
  14:43:10 STEP 1770 H=1770:00 +CPU= 46.250
  14:43:40 STEP 1771 H=1771:00 +CPU= 29.341
  14:44:10 STEP 1772 H=1772:00 +CPU= 30.281
  14:44:58 STEP 1773 H=1773:00 +CPU= 47.673
  14:45:25 STEP 1774 H=1774:00 +CPU= 26.927
  14:45:49 STEP 1775 H=1775:00 +CPU= 24.062
  14:46:38 STEP 1776 H=1776:00 +CPU= 48.474
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_73.zip
Uploading the intermediate file: upload_file_73.zip
  14:47:08 STEP 1777 H=1777:00 +CPU= 30.182
  14:47:39 STEP 1778 H=1778:00 +CPU= 30.993
  14:48:27 STEP 1779 H=1779:00 +CPU= 47.410
  14:48:57 STEP 1780 H=1780:00 +CPU= 30.441
  14:49:28 STEP 1781 H=1781:00 +CPU= 30.471
  14:50:12 STEP 1782 H=1782:00 +CPU= 44.087
  14:50:35 STEP 1783 H=1783:00 +CPU= 23.313
  14:51:00 STEP 1784 H=1784:00 +CPU= 24.549
  14:51:48 STEP 1785 H=1785:00 +CPU= 47.361
  14:52:19 STEP 1786 H=1786:00 +CPU= 30.984
  14:52:48 STEP 1787 H=1787:00 +CPU= 29.678
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0879
(argv4) batchid: 968
(argv5) wuid: 12185523
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1bcbd074e567f4ec25f4f1eb156e0221 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0879_1999050100_123_968_12185523.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0879_1999050100_123_968_12185523.zip
ic_ancil_file: ic_ancil_12185523
ifsdata_file: ifsdata_12185523
climate_data_file: clim_data_12185523
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2a05ea718d29723cc70740742bf8fed5 to: /var/lib/boinc-client/slots/3/ic_ancil_12185523.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12185523.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eccbb12003008c0fc38cf2bc2c03af45 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12185523.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12185523.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12185523.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12185523.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12185523.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12185523.xml
-- Model is restarting --
Adjusting last_iter, 1788, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12185523.xml
last_cpu_time: 60584
upload_file_number: 74
last_iter: 1775
last_upload: 6393600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996
The child process has been launched with process id: 85504
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:85504:85504] [20221223:145424:1671803664:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:55:21 STEP 1776 H=1776:00 +CPU= 53.480
  14:55:51 STEP 1777 H=1777:00 +CPU= 29.718
  14:56:19 STEP 1778 H=1778:00 +CPU= 28.194
  14:56:57 STEP 1779 H=1779:00 +CPU= 37.578
  14:57:23 STEP 1780 H=1780:00 +CPU= 25.966
  14:57:53 STEP 1781 H=1781:00 +CPU= 30.094
  14:58:42 STEP 1782 H=1782:00 +CPU= 48.436
  14:59:12 STEP 1783 H=1783:00 +CPU= 29.966
  14:59:43 STEP 1784 H=1784:00 +CPU= 31.180
  15:00:30 STEP 1785 H=1785:00 +CPU= 46.667
  15:01:00 STEP 1786 H=1786:00 +CPU= 30.308
  15:01:31 STEP 1787 H=1787:00 +CPU= 30.370
  15:02:21 STEP 1788 H=1788:00 +CPU= 49.929
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001788
  15:02:45 STEP 1789 H=1789:00 +CPU= 23.391
  15:03:09 STEP 1790 H=1790:00 +CPU= 24.205
  15:03:55 STEP 1791 H=1791:00 +CPU= 45.894
  15:04:26 STEP 1792 H=1792:00 +CPU= 30.841
  15:04:56 STEP 1793 H=1793:00 +CPU= 29.677
  15:05:44 STEP 1794 H=1794:00 +CPU= 48.116
  15:06:14 STEP 1795 H=1795:00 +CPU= 29.545
  15:06:44 STEP 1796 H=1796:00 +CPU= 30.426
  15:07:32 STEP 1797 H=1797:00 +CPU= 47.359
  15:08:02 STEP 1798 H=1798:00 +CPU= 29.844
  15:08:31 STEP 1799 H=1799:00 +CPU= 29.159
  15:09:24 STEP 1800 H=1800:00 +CPU= 53.389
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_74.zip
Uploading the intermediate file: upload_file_74.zip
  15:09:54 STEP 1801 H=1801:00 +CPU= 29.103
  15:10:23 STEP 1802 H=1802:00 +CPU= 29.263
  15:11:09 STEP 1803 H=1803:00 +CPU= 45.950
  15:11:39 STEP 1804 H=1804:00 +CPU= 29.905
  15:12:08 STEP 1805 H=1805:00 +CPU= 29.211
  15:12:55 STEP 1806 H=1806:00 +CPU= 46.878
  15:13:25 STEP 1807 H=1807:00 +CPU= 29.027
  15:13:55 STEP 1808 H=1808:00 +CPU= 29.894
  15:14:40 STEP 1809 H=1809:00 +CPU= 45.668
  15:15:10 STEP 1810 H=1810:00 +CPU= 29.645
  15:15:39 STEP 1811 H=1811:00 +CPU= 28.835
  15:16:30 STEP 1812 H=1812:00 +CPU= 50.711
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001812
  15:16:59 STEP 1813 H=1813:00 +CPU= 28.746
  15:17:29 STEP 1814 H=1814:00 +CPU= 29.759
  15:18:14 STEP 1815 H=1815:00 +CPU= 45.729
  15:18:44 STEP 1816 H=1816:00 +CPU= 29.974
  15:19:13 STEP 1817 H=1817:00 +CPU= 28.945
  15:20:01 STEP 1818 H=1818:00 +CPU= 46.902
  15:20:29 STEP 1819 H=1819:00 +CPU= 28.874
  15:20:59 STEP 1820 H=1820:00 +CPU= 29.859
  15:21:46 STEP 1821 H=1821:00 +CPU= 46.067
  15:22:15 STEP 1822 H=1822:00 +CPU= 29.738
  15:22:45 STEP 1823 H=1823:00 +CPU= 29.805
  15:23:39 STEP 1824 H=1824:00 +CPU= 53.121
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_75.zip
Uploading the intermediate file: upload_file_75.zip
  15:24:08 STEP 1825 H=1825:00 +CPU= 28.748
  15:24:38 STEP 1826 H=1826:00 +CPU= 30.068
  15:25:24 STEP 1827 H=1827:00 +CPU= 46.105
  15:25:54 STEP 1828 H=1828:00 +CPU= 29.812
  15:26:23 STEP 1829 H=1829:00 +CPU= 28.844
  15:27:09 STEP 1830 H=1830:00 +CPU= 46.451
  15:27:39 STEP 1831 H=1831:00 +CPU= 29.204
  15:28:08 STEP 1832 H=1832:00 +CPU= 29.665
  15:28:54 STEP 1833 H=1833:00 +CPU= 45.786
  15:29:24 STEP 1834 H=1834:00 +CPU= 29.844
  15:29:53 STEP 1835 H=1835:00 +CPU= 29.025
  15:30:44 STEP 1836 H=1836:00 +CPU= 50.431
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001836
  15:31:13 STEP 1837 H=1837:00 +CPU= 28.852
  15:31:43 STEP 1838 H=1838:00 +CPU= 29.750
  15:32:29 STEP 1839 H=1839:00 +CPU= 45.659
  15:32:58 STEP 1840 H=1840:00 +CPU= 29.701
  15:33:27 STEP 1841 H=1841:00 +CPU= 28.873
  15:34:14 STEP 1842 H=1842:00 +CPU= 46.829
  15:34:43 STEP 1843 H=1843:00 +CPU= 28.636
  15:35:13 STEP 1844 H=1844:00 +CPU= 29.844
  15:35:59 STEP 1845 H=1845:00 +CPU= 46.329
  15:36:29 STEP 1846 H=1846:00 +CPU= 29.662
  15:36:59 STEP 1847 H=1847:00 +CPU= 29.376
  15:37:52 STEP 1848 H=1848:00 +CPU= 53.544
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_76.zip
Uploading the intermediate file: upload_file_76.zip
  15:38:21 STEP 1849 H=1849:00 +CPU= 28.826
  15:38:51 STEP 1850 H=1850:00 +CPU= 29.717
  15:39:37 STEP 1851 H=1851:00 +CPU= 46.095
  15:40:07 STEP 1852 H=1852:00 +CPU= 29.893
  15:40:37 STEP 1853 H=1853:00 +CPU= 29.429
  15:41:24 STEP 1854 H=1854:00 +CPU= 46.967
  15:41:53 STEP 1855 H=1855:00 +CPU= 28.884
  15:42:23 STEP 1856 H=1856:00 +CPU= 29.547
  15:43:09 STEP 1857 H=1857:00 +CPU= 45.981
  15:43:39 STEP 1858 H=1858:00 +CPU= 29.849
  15:44:08 STEP 1859 H=1859:00 +CPU= 28.837
  15:44:59 STEP 1860 H=1860:00 +CPU= 50.763
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001860
  15:45:27 STEP 1861 H=1861:00 +CPU= 28.893
  15:45:57 STEP 1862 H=1862:00 +CPU= 29.550
  15:46:43 STEP 1863 H=1863:00 +CPU= 45.990
  15:47:13 STEP 1864 H=1864:00 +CPU= 29.868
  15:47:42 STEP 1865 H=1865:00 +CPU= 29.132
  15:48:29 STEP 1866 H=1866:00 +CPU= 46.566
  15:48:58 STEP 1867 H=1867:00 +CPU= 28.856
  15:49:28 STEP 1868 H=1868:00 +CPU= 29.642
  15:50:14 STEP 1869 H=1869:00 +CPU= 45.790
  15:50:44 STEP 1870 H=1870:00 +CPU= 29.838
  15:51:14 STEP 1871 H=1871:00 +CPU= 29.936
  15:52:07 STEP 1872 H=1872:00 +CPU= 53.516
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_77.zip
Uploading the intermediate file: upload_file_77.zip
  15:52:37 STEP 1873 H=1873:00 +CPU= 29.223
  15:53:06 STEP 1874 H=1874:00 +CPU= 29.681
  15:53:52 STEP 1875 H=1875:00 +CPU= 45.365
  15:54:22 STEP 1876 H=1876:00 +CPU= 29.688
  15:54:51 STEP 1877 H=1877:00 +CPU= 28.919
  15:55:38 STEP 1878 H=1878:00 +CPU= 46.863
  15:56:07 STEP 1879 H=1879:00 +CPU= 29.031
  15:56:37 STEP 1880 H=1880:00 +CPU= 29.940
  15:57:23 STEP 1881 H=1881:00 +CPU= 45.919
  15:57:53 STEP 1882 H=1882:00 +CPU= 29.674
  15:58:21 STEP 1883 H=1883:00 +CPU= 28.570
  15:59:13 STEP 1884 H=1884:00 +CPU= 51.025
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001884
  15:59:42 STEP 1885 H=1885:00 +CPU= 29.135
  16:00:12 STEP 1886 H=1886:00 +CPU= 29.767
  16:00:58 STEP 1887 H=1887:00 +CPU= 45.941
  16:01:28 STEP 1888 H=1888:00 +CPU= 29.726
  16:01:57 STEP 1889 H=1889:00 +CPU= 28.933
  16:02:44 STEP 1890 H=1890:00 +CPU= 46.867
  16:03:13 STEP 1891 H=1891:00 +CPU= 28.943
  16:03:43 STEP 1892 H=1892:00 +CPU= 29.960
  16:04:29 STEP 1893 H=1893:00 +CPU= 46.171
  16:04:59 STEP 1894 H=1894:00 +CPU= 29.959
  16:05:29 STEP 1895 H=1895:00 +CPU= 29.664
  16:06:23 STEP 1896 H=1896:00 +CPU= 53.940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_78.zip
Uploading the intermediate file: upload_file_78.zip
  16:06:52 STEP 1897 H=1897:00 +CPU= 29.093
  16:07:22 STEP 1898 H=1898:00 +CPU= 29.917
  16:08:08 STEP 1899 H=1899:00 +CPU= 45.911
  16:08:38 STEP 1900 H=1900:00 +CPU= 29.931
  16:09:07 STEP 1901 H=1901:00 +CPU= 29.004
  16:09:54 STEP 1902 H=1902:00 +CPU= 46.891
  16:10:23 STEP 1903 H=1903:00 +CPU= 28.839
  16:10:53 STEP 1904 H=1904:00 +CPU= 29.774
  16:11:39 STEP 1905 H=1905:00 +CPU= 45.798
  16:12:09 STEP 1906 H=1906:00 +CPU= 30.204
  16:12:39 STEP 1907 H=1907:00 +CPU= 29.227
  16:13:29 STEP 1908 H=1908:00 +CPU= 50.318
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001908
  16:13:58 STEP 1909 H=1909:00 +CPU= 29.216
  16:14:28 STEP 1910 H=1910:00 +CPU= 29.686
  16:15:14 STEP 1911 H=1911:00 +CPU= 45.385
  16:15:44 STEP 1912 H=1912:00 +CPU= 29.792
  16:16:12 STEP 1913 H=1913:00 +CPU= 28.838
  16:17:00 STEP 1914 H=1914:00 +CPU= 47.106
  16:17:29 STEP 1915 H=1915:00 +CPU= 29.052
  16:17:59 STEP 1916 H=1916:00 +CPU= 29.987
  16:18:45 STEP 1917 H=1917:00 +CPU= 45.931
  16:19:15 STEP 1918 H=1918:00 +CPU= 29.603
  16:19:45 STEP 1919 H=1919:00 +CPU= 29.795
  16:20:38 STEP 1920 H=1920:00 +CPU= 53.674
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_79.zip
Uploading the intermediate file: upload_file_79.zip
  16:21:08 STEP 1921 H=1921:00 +CPU= 29.091
  16:21:38 STEP 1922 H=1922:00 +CPU= 29.970
  16:22:23 STEP 1923 H=1923:00 +CPU= 45.585
  16:22:54 STEP 1924 H=1924:00 +CPU= 30.042
  16:23:22 STEP 1925 H=1925:00 +CPU= 28.835
  16:24:09 STEP 1926 H=1926:00 +CPU= 46.656
  16:24:38 STEP 1927 H=1927:00 +CPU= 29.134
  16:25:08 STEP 1928 H=1928:00 +CPU= 29.609
  16:25:54 STEP 1929 H=1929:00 +CPU= 45.352
  16:26:24 STEP 1930 H=1930:00 +CPU= 30.147
  16:26:53 STEP 1931 H=1931:00 +CPU= 28.977
  16:27:44 STEP 1932 H=1932:00 +CPU= 50.409
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001932
  16:28:12 STEP 1933 H=1933:00 +CPU= 28.819
  16:28:42 STEP 1934 H=1934:00 +CPU= 29.698
  16:29:28 STEP 1935 H=1935:00 +CPU= 45.894
  16:29:58 STEP 1936 H=1936:00 +CPU= 29.888
  16:30:27 STEP 1937 H=1937:00 +CPU= 28.860
  16:31:14 STEP 1938 H=1938:00 +CPU= 46.689
  16:31:43 STEP 1939 H=1939:00 +CPU= 28.953
  16:32:13 STEP 1940 H=1940:00 +CPU= 29.866
  16:32:58 STEP 1941 H=1941:00 +CPU= 45.311
  16:33:28 STEP 1942 H=1942:00 +CPU= 29.668
  16:33:58 STEP 1943 H=1943:00 +CPU= 29.795
  16:34:52 STEP 1944 H=1944:00 +CPU= 53.752
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_80.zip
Uploading the intermediate file: upload_file_80.zip
  16:35:21 STEP 1945 H=1945:00 +CPU= 28.955
  16:35:51 STEP 1946 H=1946:00 +CPU= 29.485
  16:36:36 STEP 1947 H=1947:00 +CPU= 45.564
  16:37:06 STEP 1948 H=1948:00 +CPU= 29.823
  16:37:35 STEP 1949 H=1949:00 +CPU= 28.692
  16:38:22 STEP 1950 H=1950:00 +CPU= 47.203
  16:38:51 STEP 1951 H=1951:00 +CPU= 29.042
  16:39:21 STEP 1952 H=1952:00 +CPU= 29.798
  16:40:07 STEP 1953 H=1953:00 +CPU= 45.780
  16:40:37 STEP 1954 H=1954:00 +CPU= 29.241
  16:41:06 STEP 1955 H=1955:00 +CPU= 29.127
  16:41:56 STEP 1956 H=1956:00 +CPU= 50.459
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001956
  16:42:25 STEP 1957 H=1957:00 +CPU= 28.892
  16:42:55 STEP 1958 H=1958:00 +CPU= 29.868
  16:43:42 STEP 1959 H=1959:00 +CPU= 46.260
  16:44:12 STEP 1960 H=1960:00 +CPU= 29.953
  16:44:41 STEP 1961 H=1961:00 +CPU= 29.069
  16:45:28 STEP 1962 H=1962:00 +CPU= 47.108
  16:45:57 STEP 1963 H=1963:00 +CPU= 29.008
  16:46:27 STEP 1964 H=1964:00 +CPU= 29.657
  16:47:14 STEP 1965 H=1965:00 +CPU= 46.700
  16:47:44 STEP 1966 H=1966:00 +CPU= 29.986
  16:48:14 STEP 1967 H=1967:00 +CPU= 29.979
  16:49:07 STEP 1968 H=1968:00 +CPU= 53.046
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_81.zip
Uploading the intermediate file: upload_file_81.zip
  16:49:36 STEP 1969 H=1969:00 +CPU= 28.768
  16:50:08 STEP 1970 H=1970:00 +CPU= 31.593
  16:50:55 STEP 1971 H=1971:00 +CPU= 46.879
  16:51:26 STEP 1972 H=1972:00 +CPU= 30.557
  16:51:56 STEP 1973 H=1973:00 +CPU= 30.142
  16:52:44 STEP 1974 H=1974:00 +CPU= 47.926
  16:53:15 STEP 1975 H=1975:00 +CPU= 30.525
  16:53:46 STEP 1976 H=1976:00 +CPU= 31.002
  16:54:32 STEP 1977 H=1977:00 +CPU= 46.401
  16:55:04 STEP 1978 H=1978:00 +CPU= 31.280
  16:55:34 STEP 1979 H=1979:00 +CPU= 29.950
  16:56:26 STEP 1980 H=1980:00 +CPU= 51.845
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001980
  16:56:56 STEP 1981 H=1981:00 +CPU= 29.814
  16:57:27 STEP 1982 H=1982:00 +CPU= 30.908
  16:58:13 STEP 1983 H=1983:00 +CPU= 45.768
  16:58:44 STEP 1984 H=1984:00 +CPU= 30.982
  16:59:14 STEP 1985 H=1985:00 +CPU= 29.709
  17:00:01 STEP 1986 H=1986:00 +CPU= 47.457
  17:00:31 STEP 1987 H=1987:00 +CPU= 30.146
  17:01:03 STEP 1988 H=1988:00 +CPU= 31.545
  17:01:50 STEP 1989 H=1989:00 +CPU= 46.869
  17:02:21 STEP 1990 H=1990:00 +CPU= 30.300
  17:02:52 STEP 1991 H=1991:00 +CPU= 31.100
  17:03:48 STEP 1992 H=1992:00 +CPU= 55.839
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_82.zip
Uploading the intermediate file: upload_file_82.zip
  17:04:18 STEP 1993 H=1993:00 +CPU= 29.857
  17:04:49 STEP 1994 H=1994:00 +CPU= 31.074
  17:05:36 STEP 1995 H=1995:00 +CPU= 46.773
  17:06:07 STEP 1996 H=1996:00 +CPU= 30.940
  17:06:37 STEP 1997 H=1997:00 +CPU= 29.537
  17:07:25 STEP 1998 H=1998:00 +CPU= 47.734
  17:07:55 STEP 1999 H=1999:00 +CPU= 29.770
  17:08:26 STEP 2000 H=2000:00 +CPU= 31.488
  17:09:13 STEP 2001 H=2001:00 +CPU= 47.138
  17:09:45 STEP 2002 H=2002:00 +CPU= 31.203
  17:10:15 STEP 2003 H=2003:00 +CPU= 30.050
  17:11:07 STEP 2004 H=2004:00 +CPU= 52.013
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002004
  17:11:37 STEP 2005 H=2005:00 +CPU= 29.697
  17:12:08 STEP 2006 H=2006:00 +CPU= 30.600
  17:12:54 STEP 2007 H=2007:00 +CPU= 46.347
  17:13:25 STEP 2008 H=2008:00 +CPU= 30.769
  17:13:54 STEP 2009 H=2009:00 +CPU= 29.294
  17:14:42 STEP 2010 H=2010:00 +CPU= 47.837
  17:15:12 STEP 2011 H=2011:00 +CPU= 29.542
  17:15:43 STEP 2012 H=2012:00 +CPU= 30.877
  17:16:31 STEP 2013 H=2013:00 +CPU= 47.641
  17:17:01 STEP 2014 H=2014:00 +CPU= 30.276
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0879
(argv4) batchid: 968
(argv5) wuid: 12185523
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1bcbd074e567f4ec25f4f1eb156e0221 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0879_1999050100_123_968_12185523.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0879_1999050100_123_968_12185523.zip
ic_ancil_file: ic_ancil_12185523
ifsdata_file: ifsdata_12185523
climate_data_file: clim_data_12185523
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2a05ea718d29723cc70740742bf8fed5 to: /var/lib/boinc-client/slots/3/ic_ancil_12185523.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12185523.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eccbb12003008c0fc38cf2bc2c03af45 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12185523.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12185523.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12185523.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12185523.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12185523.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12185523.xml
-- Model is restarting --
Adjusting last_iter, 2015, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12185523.xml
last_cpu_time: 69137
upload_file_number: 83
last_iter: 1991
last_upload: 7171200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996
The child process has been launched with process id: 88879
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hackerbob-Macmini:1:1:88879:88879] [20221223:181139:1671815499:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:12:47 STEP 1992 H=1992:00 +CPU= 53.398
  18:13:16 STEP 1993 H=1993:00 +CPU= 29.501
  18:13:47 STEP 1994 H=1994:00 +CPU= 30.608
  18:14:33 STEP 1995 H=1995:00 +CPU= 45.796
  18:15:04 STEP 1996 H=1996:00 +CPU= 30.945
  18:15:34 STEP 1997 H=1997:00 +CPU= 29.990
  18:16:21 STEP 1998 H=1998:00 +CPU= 47.150
  18:16:52 STEP 1999 H=1999:00 +CPU= 30.221
  18:17:23 STEP 2000 H=2000:00 +CPU= 30.907
  18:18:04 STEP 2001 H=2001:00 +CPU= 41.551
  18:18:29 STEP 2002 H=2002:00 +CPU= 24.481
  18:18:53 STEP 2003 H=2003:00 +CPU= 24.352
  18:19:44 STEP 2004 H=2004:00 +CPU= 50.402
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002004
  18:20:15 STEP 2005 H=2005:00 +CPU= 30.506
  18:20:46 STEP 2006 H=2006:00 +CPU= 31.219
  18:21:34 STEP 2007 H=2007:00 +CPU= 47.506
  18:22:05 STEP 2008 H=2008:00 +CPU= 30.928
  18:22:35 STEP 2009 H=2009:00 +CPU= 29.824
  18:23:23 STEP 2010 H=2010:00 +CPU= 48.431
  18:23:53 STEP 2011 H=2011:00 +CPU= 29.569
  18:24:23 STEP 2012 H=2012:00 +CPU= 30.328
  18:25:12 STEP 2013 H=2013:00 +CPU= 47.671
  18:25:43 STEP 2014 H=2014:00 +CPU= 30.754
  18:26:13 STEP 2015 H=2015:00 +CPU= 30.188
  18:27:08 STEP 2016 H=2016:00 +CPU= 54.627
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185523/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0879_1999050100_123_968_12185523_0_r1404444996_83.zip
Uploading the intermediate file: upload_file_83.zip
  18:27:38 STEP 2017 H=2017:00 +CPU= 29.686
  18:28:09 STEP 2018 H=2018:00 +CPU= 30.165
  18:28:56 STEP 2019 H=2019:00 +CPU= 47.027
  18:29:26 STEP 2020 H=2020:00 +CPU= 30.486
  18:29:56 STEP 2021 H=2021:00 +CPU= 29.663
  18:30:43 STEP 2022 H=2022:00 +CPU= 46.668
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:50:00 1536225 22270851 oifs_43r3_ps_0879_1999050100_123_968_12185523_0 5,529,600 51,048 0.0092
09 May 2023 15:49:30 1536225 22270851 oifs_43r3_ps_0879_1999050100_123_968_12185523_0 3,456,000 29,646 0.0086
09 May 2023 15:49:19 1536225 22270851 oifs_43r3_ps_0879_1999050100_123_968_12185523_0 2,592,000 22,254 0.0086
09 May 2023 15:49:08 1536225 22270851 oifs_43r3_ps_0879_1999050100_123_968_12185523_0 1,728,000 14,894 0.0086
09 May 2023 15:48:57 1536225 22270851 oifs_43r3_ps_0879_1999050100_123_968_12185523_0 864,000 7,532 0.0087


©2024 climateprediction.net