climateprediction.net home page
Task 22303872

Task 22303872

Name oifs_43r3_ps_0441_1981050100_123_950_12167085_2
Workunit 12167085
Created 26 Jan 2023, 10:48:48 UTC
Sent 26 Jan 2023, 10:49:10 UTC
Report deadline 27 Mar 2023, 10:49:10 UTC
Received 27 Jan 2023, 19:45:27 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1493665
Run time 20 hours 49 min 40 sec
CPU time 18 hours 50 min 49 sec
Validate state Invalid
Credit 6,531.84
Device peak FLOPS 4.47 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,545.17 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,674.05 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
oinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_69.zip
Uploading the intermediate file: upload_file_69.zip
  04:34:54 STEP 1681 H=1681:00 +CPU= 26.429
  04:35:22 STEP 1682 H=1682:00 +CPU= 27.037
  04:36:08 STEP 1683 H=1683:00 +CPU= 41.594
  04:36:37 STEP 1684 H=1684:00 +CPU= 26.992
  04:37:05 STEP 1685 H=1685:00 +CPU= 26.163
  04:37:51 STEP 1686 H=1686:00 +CPU= 42.340
  04:38:20 STEP 1687 H=1687:00 +CPU= 26.332
  04:38:49 STEP 1688 H=1688:00 +CPU= 26.888
  04:39:34 STEP 1689 H=1689:00 +CPU= 41.501
  04:40:04 STEP 1690 H=1690:00 +CPU= 27.431
  04:40:33 STEP 1691 H=1691:00 +CPU= 26.472
  04:41:22 STEP 1692 H=1692:00 +CPU= 46.025
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001692
  04:41:51 STEP 1693 H=1693:00 +CPU= 26.001
  04:42:20 STEP 1694 H=1694:00 +CPU= 26.960
  04:43:06 STEP 1695 H=1695:00 +CPU= 41.959
  04:43:40 STEP 1696 H=1696:00 +CPU= 27.421
  04:44:09 STEP 1697 H=1697:00 +CPU= 26.345
  04:44:55 STEP 1698 H=1698:00 +CPU= 42.752
  04:45:24 STEP 1699 H=1699:00 +CPU= 26.460
  04:45:54 STEP 1700 H=1700:00 +CPU= 26.969
  04:46:38 STEP 1701 H=1701:00 +CPU= 41.471
  04:47:08 STEP 1702 H=1702:00 +CPU= 27.084
  04:47:37 STEP 1703 H=1703:00 +CPU= 27.234
  04:48:31 STEP 1704 H=1704:00 +CPU= 48.671
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_70.zip
Uploading the intermediate file: upload_file_70.zip
  04:49:00 STEP 1705 H=1705:00 +CPU= 26.402
  04:49:30 STEP 1706 H=1706:00 +CPU= 27.212
  04:50:17 STEP 1707 H=1707:00 +CPU= 41.960
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0441
(argv4) batchid: 950
(argv5) wuid: 12167085
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4d99730c22a7c993fdb6ade587f1c9ce to: /var/lib/boinc/slots/1/oifs_43r3_ps_0441_1981050100_123_950_12167085.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0441_1981050100_123_950_12167085.zip
ic_ancil_file: ic_ancil_12167085
ifsdata_file: ifsdata_12167085
climate_data_file: clim_data_12167085
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3491416c55535c88ee2f1f0531242c15 to: /var/lib/boinc/slots/1/ic_ancil_12167085.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12167085.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9603068b1261249a178e76b73a262bbc to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12167085.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12167085.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12167085.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12167085.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12167085.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12167085.xml
-- Model is restarting --
Adjusting last_iter, 1708, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12167085.xml
last_cpu_time: 56566
upload_file_number: 71
last_iter: 1703
last_upload: 6134400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270
The child process has been launched with process id: 10597
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:10597:10597] [20230127:055630:1674824190:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:57:31 STEP 1704 H=1704:00 +CPU= 46.950
  05:58:00 STEP 1705 H=1705:00 +CPU= 26.062
  05:58:29 STEP 1706 H=1706:00 +CPU= 26.826
  05:59:14 STEP 1707 H=1707:00 +CPU= 41.389
  05:59:44 STEP 1708 H=1708:00 +CPU= 27.141
  06:00:16 STEP 1709 H=1709:00 +CPU= 26.774
  06:01:03 STEP 1710 H=1710:00 +CPU= 42.651
  06:01:31 STEP 1711 H=1711:00 +CPU= 25.950
  06:02:01 STEP 1712 H=1712:00 +CPU= 27.177
  06:02:46 STEP 1713 H=1713:00 +CPU= 41.666
  06:03:15 STEP 1714 H=1714:00 +CPU= 27.338
  06:03:44 STEP 1715 H=1715:00 +CPU= 26.117
  06:04:34 STEP 1716 H=1716:00 +CPU= 45.905
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001716
  06:05:02 STEP 1717 H=1717:00 +CPU= 26.474
  06:05:32 STEP 1718 H=1718:00 +CPU= 27.040
  06:06:16 STEP 1719 H=1719:00 +CPU= 41.558
  06:06:46 STEP 1720 H=1720:00 +CPU= 27.278
  06:07:14 STEP 1721 H=1721:00 +CPU= 26.024
  06:08:00 STEP 1722 H=1722:00 +CPU= 42.261
  06:08:28 STEP 1723 H=1723:00 +CPU= 26.117
  06:08:58 STEP 1724 H=1724:00 +CPU= 26.934
  06:09:43 STEP 1725 H=1725:00 +CPU= 41.598
  06:10:12 STEP 1726 H=1726:00 +CPU= 26.951
  06:10:41 STEP 1727 H=1727:00 +CPU= 26.968
  06:11:34 STEP 1728 H=1728:00 +CPU= 48.292
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_71.zip
Uploading the intermediate file: upload_file_71.zip
  06:12:02 STEP 1729 H=1729:00 +CPU= 26.136
  06:12:32 STEP 1730 H=1730:00 +CPU= 26.926
  06:13:17 STEP 1731 H=1731:00 +CPU= 41.594
  06:13:46 STEP 1732 H=1732:00 +CPU= 27.034
  06:14:20 STEP 1733 H=1733:00 +CPU= 26.426
  06:15:07 STEP 1734 H=1734:00 +CPU= 42.659
  06:15:36 STEP 1735 H=1735:00 +CPU= 26.466
  06:16:05 STEP 1736 H=1736:00 +CPU= 27.178
  06:16:52 STEP 1737 H=1737:00 +CPU= 41.555
  06:17:22 STEP 1738 H=1738:00 +CPU= 26.939
  06:17:51 STEP 1739 H=1739:00 +CPU= 26.334
  06:18:43 STEP 1740 H=1740:00 +CPU= 46.328
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001740
  06:19:12 STEP 1741 H=1741:00 +CPU= 26.323
  06:19:43 STEP 1742 H=1742:00 +CPU= 27.664
  06:20:29 STEP 1743 H=1743:00 +CPU= 41.510
  06:20:59 STEP 1744 H=1744:00 +CPU= 26.982
  06:21:28 STEP 1745 H=1745:00 +CPU= 26.375
  06:22:17 STEP 1746 H=1746:00 +CPU= 42.500
  06:22:45 STEP 1747 H=1747:00 +CPU= 25.942
  06:23:15 STEP 1748 H=1748:00 +CPU= 27.155
  06:24:01 STEP 1749 H=1749:00 +CPU= 41.635
  06:24:30 STEP 1750 H=1750:00 +CPU= 26.894
  06:25:02 STEP 1751 H=1751:00 +CPU= 27.471
  06:25:54 STEP 1752 H=1752:00 +CPU= 48.381
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_72.zip
Uploading the intermediate file: upload_file_72.zip
  06:26:23 STEP 1753 H=1753:00 +CPU= 26.164
  06:26:52 STEP 1754 H=1754:00 +CPU= 26.776
  06:27:38 STEP 1755 H=1755:00 +CPU= 41.612
  06:28:08 STEP 1756 H=1756:00 +CPU= 27.379
  06:28:36 STEP 1757 H=1757:00 +CPU= 26.225
  06:29:24 STEP 1758 H=1758:00 +CPU= 42.670
  06:29:53 STEP 1759 H=1759:00 +CPU= 26.462
  06:30:22 STEP 1760 H=1760:00 +CPU= 27.016
  06:31:08 STEP 1761 H=1761:00 +CPU= 41.759
  06:31:38 STEP 1762 H=1762:00 +CPU= 27.076
  06:32:06 STEP 1763 H=1763:00 +CPU= 26.120
  06:32:57 STEP 1764 H=1764:00 +CPU= 46.232
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001764
  06:33:25 STEP 1765 H=1765:00 +CPU= 26.330
  06:33:55 STEP 1766 H=1766:00 +CPU= 27.134
  06:34:40 STEP 1767 H=1767:00 +CPU= 41.611
  06:35:10 STEP 1768 H=1768:00 +CPU= 26.986
  06:35:38 STEP 1769 H=1769:00 +CPU= 26.011
  06:36:24 STEP 1770 H=1770:00 +CPU= 42.893
  06:36:53 STEP 1771 H=1771:00 +CPU= 26.050
  06:37:23 STEP 1772 H=1772:00 +CPU= 27.122
  06:38:13 STEP 1773 H=1773:00 +CPU= 41.514
  06:38:42 STEP 1774 H=1774:00 +CPU= 27.034
  06:39:12 STEP 1775 H=1775:00 +CPU= 27.104
  06:40:06 STEP 1776 H=1776:00 +CPU= 48.829
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_73.zip
Uploading the intermediate file: upload_file_73.zip
  06:40:34 STEP 1777 H=1777:00 +CPU= 26.372
  06:41:04 STEP 1778 H=1778:00 +CPU= 27.009
  06:41:50 STEP 1779 H=1779:00 +CPU= 41.890
  06:42:19 STEP 1780 H=1780:00 +CPU= 27.149
  06:42:48 STEP 1781 H=1781:00 +CPU= 26.289
  06:43:34 STEP 1782 H=1782:00 +CPU= 42.197
  06:44:03 STEP 1783 H=1783:00 +CPU= 26.354
  06:44:34 STEP 1784 H=1784:00 +CPU= 27.319
  06:45:19 STEP 1785 H=1785:00 +CPU= 41.640
  06:45:49 STEP 1786 H=1786:00 +CPU= 27.244
  06:46:17 STEP 1787 H=1787:00 +CPU= 26.182
  06:47:09 STEP 1788 H=1788:00 +CPU= 46.283
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001788
  06:47:37 STEP 1789 H=1789:00 +CPU= 26.070
  06:48:07 STEP 1790 H=1790:00 +CPU= 27.230
  06:48:53 STEP 1791 H=1791:00 +CPU= 41.653
  06:49:22 STEP 1792 H=1792:00 +CPU= 27.062
  06:49:51 STEP 1793 H=1793:00 +CPU= 26.176
  06:50:38 STEP 1794 H=1794:00 +CPU= 42.493
  06:51:07 STEP 1795 H=1795:00 +CPU= 26.399
  06:51:36 STEP 1796 H=1796:00 +CPU= 27.067
  06:52:26 STEP 1797 H=1797:00 +CPU= 41.700
  06:52:56 STEP 1798 H=1798:00 +CPU= 27.146
  06:53:25 STEP 1799 H=1799:00 +CPU= 27.098
  06:54:19 STEP 1800 H=1800:00 +CPU= 48.565
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_74.zip
Uploading the intermediate file: upload_file_74.zip
  06:54:48 STEP 1801 H=1801:00 +CPU= 26.153
  06:55:18 STEP 1802 H=1802:00 +CPU= 27.725
  06:56:04 STEP 1803 H=1803:00 +CPU= 41.697
  06:56:33 STEP 1804 H=1804:00 +CPU= 27.068
  06:57:02 STEP 1805 H=1805:00 +CPU= 26.165
  06:57:49 STEP 1806 H=1806:00 +CPU= 42.644
  06:58:23 STEP 1807 H=1807:00 +CPU= 26.554
  06:58:54 STEP 1808 H=1808:00 +CPU= 27.301
  06:59:43 STEP 1809 H=1809:00 +CPU= 42.103
  07:00:16 STEP 1810 H=1810:00 +CPU= 27.372
  07:00:47 STEP 1811 H=1811:00 +CPU= 26.627
  07:01:44 STEP 1812 H=1812:00 +CPU= 47.006
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001812
  07:02:16 STEP 1813 H=1813:00 +CPU= 26.727
  07:02:48 STEP 1814 H=1814:00 +CPU= 27.418
  07:03:39 STEP 1815 H=1815:00 +CPU= 42.095
  07:04:12 STEP 1816 H=1816:00 +CPU= 27.574
  07:04:42 STEP 1817 H=1817:00 +CPU= 26.407
  07:05:32 STEP 1818 H=1818:00 +CPU= 42.912
  07:06:05 STEP 1819 H=1819:00 +CPU= 26.660
  07:06:37 STEP 1820 H=1820:00 +CPU= 27.638
  07:07:27 STEP 1821 H=1821:00 +CPU= 41.967
  07:07:59 STEP 1822 H=1822:00 +CPU= 27.202
  07:08:31 STEP 1823 H=1823:00 +CPU= 27.435
  07:09:28 STEP 1824 H=1824:00 +CPU= 48.882
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_75.zip
Uploading the intermediate file: upload_file_75.zip
  07:09:59 STEP 1825 H=1825:00 +CPU= 26.361
  07:10:28 STEP 1826 H=1826:00 +CPU= 26.907
  07:11:13 STEP 1827 H=1827:00 +CPU= 41.446
  07:11:42 STEP 1828 H=1828:00 +CPU= 27.059
  07:12:11 STEP 1829 H=1829:00 +CPU= 26.230
  07:13:02 STEP 1830 H=1830:00 +CPU= 42.672
  07:13:31 STEP 1831 H=1831:00 +CPU= 26.431
  07:14:00 STEP 1832 H=1832:00 +CPU= 27.033
  07:14:45 STEP 1833 H=1833:00 +CPU= 41.657
  07:15:14 STEP 1834 H=1834:00 +CPU= 26.986
  07:15:45 STEP 1835 H=1835:00 +CPU= 26.477
  07:16:39 STEP 1836 H=1836:00 +CPU= 46.312
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001836
  07:17:07 STEP 1837 H=1837:00 +CPU= 26.089
  07:17:37 STEP 1838 H=1838:00 +CPU= 27.140
  07:18:22 STEP 1839 H=1839:00 +CPU= 41.420
  07:18:51 STEP 1840 H=1840:00 +CPU= 27.054
  07:19:20 STEP 1841 H=1841:00 +CPU= 26.252
  07:20:06 STEP 1842 H=1842:00 +CPU= 42.335
  07:20:34 STEP 1843 H=1843:00 +CPU= 26.204
  07:21:03 STEP 1844 H=1844:00 +CPU= 26.886
  07:21:48 STEP 1845 H=1845:00 +CPU= 41.479
  07:22:17 STEP 1846 H=1846:00 +CPU= 26.885
  07:22:46 STEP 1847 H=1847:00 +CPU= 26.924
  07:23:45 STEP 1848 H=1848:00 +CPU= 48.826
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_76.zip
Uploading the intermediate file: upload_file_76.zip
  07:24:14 STEP 1849 H=1849:00 +CPU= 26.117
  07:24:43 STEP 1850 H=1850:00 +CPU= 26.940
  07:25:28 STEP 1851 H=1851:00 +CPU= 41.434
  07:25:58 STEP 1852 H=1852:00 +CPU= 27.326
  07:26:27 STEP 1853 H=1853:00 +CPU= 26.371
  07:27:14 STEP 1854 H=1854:00 +CPU= 42.690
  07:27:43 STEP 1855 H=1855:00 +CPU= 26.349
  07:28:12 STEP 1856 H=1856:00 +CPU= 27.052
  07:28:58 STEP 1857 H=1857:00 +CPU= 41.504
  07:29:27 STEP 1858 H=1858:00 +CPU= 27.013
  07:29:56 STEP 1859 H=1859:00 +CPU= 26.231
  07:30:47 STEP 1860 H=1860:00 +CPU= 46.205
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001860
  07:31:16 STEP 1861 H=1861:00 +CPU= 26.302
  07:31:45 STEP 1862 H=1862:00 +CPU= 27.099
  07:32:31 STEP 1863 H=1863:00 +CPU= 41.505
  07:33:04 STEP 1864 H=1864:00 +CPU= 27.176
  07:33:34 STEP 1865 H=1865:00 +CPU= 26.496
  07:34:22 STEP 1866 H=1866:00 +CPU= 42.822
  07:34:52 STEP 1867 H=1867:00 +CPU= 26.449
  07:35:22 STEP 1868 H=1868:00 +CPU= 27.295
  07:36:09 STEP 1869 H=1869:00 +CPU= 42.067
  07:36:41 STEP 1870 H=1870:00 +CPU= 27.596
  07:37:10 STEP 1871 H=1871:00 +CPU= 27.061
  07:38:08 STEP 1872 H=1872:00 +CPU= 48.939
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_77.zip
Uploading the intermediate file: upload_file_77.zip
  07:38:37 STEP 1873 H=1873:00 +CPU= 26.379
  07:39:07 STEP 1874 H=1874:00 +CPU= 27.009
  07:39:52 STEP 1875 H=1875:00 +CPU= 41.300
  07:40:21 STEP 1876 H=1876:00 +CPU= 27.155
  07:40:50 STEP 1877 H=1877:00 +CPU= 26.129
  07:41:37 STEP 1878 H=1878:00 +CPU= 42.602
  07:42:07 STEP 1879 H=1879:00 +CPU= 26.287
  07:42:36 STEP 1880 H=1880:00 +CPU= 27.092
  07:43:22 STEP 1881 H=1881:00 +CPU= 41.910
  07:43:52 STEP 1882 H=1882:00 +CPU= 26.896
  07:44:20 STEP 1883 H=1883:00 +CPU= 26.168
  07:45:12 STEP 1884 H=1884:00 +CPU= 46.947
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001884
  07:45:41 STEP 1885 H=1885:00 +CPU= 26.451
  07:46:11 STEP 1886 H=1886:00 +CPU= 27.195
  07:46:57 STEP 1887 H=1887:00 +CPU= 41.764
  07:47:26 STEP 1888 H=1888:00 +CPU= 27.015
  07:47:56 STEP 1889 H=1889:00 +CPU= 26.832
  07:48:44 STEP 1890 H=1890:00 +CPU= 42.648
  07:49:14 STEP 1891 H=1891:00 +CPU= 26.300
  07:49:44 STEP 1892 H=1892:00 +CPU= 27.123
  07:50:31 STEP 1893 H=1893:00 +CPU= 41.638
  07:51:01 STEP 1894 H=1894:00 +CPU= 27.021
  07:51:31 STEP 1895 H=1895:00 +CPU= 27.197
  07:52:25 STEP 1896 H=1896:00 +CPU= 48.704
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_78.zip
Uploading the intermediate file: upload_file_78.zip
  07:53:00 STEP 1897 H=1897:00 +CPU= 26.591
  07:53:29 STEP 1898 H=1898:00 +CPU= 27.269
  07:54:16 STEP 1899 H=1899:00 +CPU= 41.895
  07:54:46 STEP 1900 H=1900:00 +CPU= 27.286
  07:55:16 STEP 1901 H=1901:00 +CPU= 26.510
  07:56:04 STEP 1902 H=1902:00 +CPU= 42.958
  07:56:33 STEP 1903 H=1903:00 +CPU= 26.506
  07:57:03 STEP 1904 H=1904:00 +CPU= 27.296
  07:57:49 STEP 1905 H=1905:00 +CPU= 41.539
  07:58:19 STEP 1906 H=1906:00 +CPU= 27.293
  07:58:48 STEP 1907 H=1907:00 +CPU= 26.619
  07:59:39 STEP 1908 H=1908:00 +CPU= 46.607
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001908
  08:00:09 STEP 1909 H=1909:00 +CPU= 26.654
  08:00:38 STEP 1910 H=1910:00 +CPU= 27.276
  08:01:24 STEP 1911 H=1911:00 +CPU= 41.650
  08:01:55 STEP 1912 H=1912:00 +CPU= 27.293
  08:02:23 STEP 1913 H=1913:00 +CPU= 26.295
  08:03:13 STEP 1914 H=1914:00 +CPU= 42.794
  08:03:46 STEP 1915 H=1915:00 +CPU= 26.680
  08:04:18 STEP 1916 H=1916:00 +CPU= 27.506
  08:05:04 STEP 1917 H=1917:00 +CPU= 41.599
  08:05:34 STEP 1918 H=1918:00 +CPU= 27.096
  08:06:06 STEP 1919 H=1919:00 +CPU= 27.333
  08:06:59 STEP 1920 H=1920:00 +CPU= 48.525
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_79.zip
Uploading the intermediate file: upload_file_79.zip
  08:07:28 STEP 1921 H=1921:00 +CPU= 26.064
  08:08:02 STEP 1922 H=1922:00 +CPU= 27.453
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0441
(argv4) batchid: 950
(argv5) wuid: 12167085
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4d99730c22a7c993fdb6ade587f1c9ce to: /var/lib/boinc/slots/1/oifs_43r3_ps_0441_1981050100_123_950_12167085.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0441_1981050100_123_950_12167085.zip
ic_ancil_file: ic_ancil_12167085
ifsdata_file: ifsdata_12167085
climate_data_file: clim_data_12167085
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3491416c55535c88ee2f1f0531242c15 to: /var/lib/boinc/slots/1/ic_ancil_12167085.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12167085.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9603068b1261249a178e76b73a262bbc to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12167085.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12167085.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12167085.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12167085.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12167085.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12167085.xml
-- Model is restarting --
Adjusting last_iter, 1923, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12167085.xml
last_cpu_time: 63663
upload_file_number: 80
last_iter: 1919
last_upload: 6912000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270
The child process has been launched with process id: 11775
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:11775:11775] [20230127:100201:1674838921:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:03:01 STEP 1920 H=1920:00 +CPU= 47.181
  10:03:28 STEP 1921 H=1921:00 +CPU= 26.207
  10:03:57 STEP 1922 H=1922:00 +CPU= 27.217
  10:04:41 STEP 1923 H=1923:00 +CPU= 41.181
  10:05:13 STEP 1924 H=1924:00 +CPU= 27.750
  10:05:41 STEP 1925 H=1925:00 +CPU= 26.177
  10:06:26 STEP 1926 H=1926:00 +CPU= 42.765
  10:06:54 STEP 1927 H=1927:00 +CPU= 26.165
  10:07:23 STEP 1928 H=1928:00 +CPU= 27.177
  10:08:08 STEP 1929 H=1929:00 +CPU= 41.773
  10:08:37 STEP 1930 H=1930:00 +CPU= 27.072
  10:09:06 STEP 1931 H=1931:00 +CPU= 26.444
  10:09:55 STEP 1932 H=1932:00 +CPU= 46.263
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001932
  10:10:24 STEP 1933 H=1933:00 +CPU= 26.683
  10:10:54 STEP 1934 H=1934:00 +CPU= 27.798
  10:11:38 STEP 1935 H=1935:00 +CPU= 41.693
  10:12:07 STEP 1936 H=1936:00 +CPU= 26.953
  10:12:35 STEP 1937 H=1937:00 +CPU= 26.336
  10:13:21 STEP 1938 H=1938:00 +CPU= 42.900
  10:13:49 STEP 1939 H=1939:00 +CPU= 26.432
  10:14:18 STEP 1940 H=1940:00 +CPU= 27.400
  10:15:03 STEP 1941 H=1941:00 +CPU= 41.418
  10:15:39 STEP 1942 H=1942:00 +CPU= 27.675
  10:16:10 STEP 1943 H=1943:00 +CPU= 27.517
  10:17:04 STEP 1944 H=1944:00 +CPU= 48.706
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_80.zip
Uploading the intermediate file: upload_file_80.zip
  10:17:32 STEP 1945 H=1945:00 +CPU= 26.116
  10:18:02 STEP 1946 H=1946:00 +CPU= 27.061
  10:18:52 STEP 1947 H=1947:00 +CPU= 42.017
  10:19:22 STEP 1948 H=1948:00 +CPU= 27.035
  10:19:51 STEP 1949 H=1949:00 +CPU= 26.581
  10:20:38 STEP 1950 H=1950:00 +CPU= 43.345
  10:21:07 STEP 1951 H=1951:00 +CPU= 26.407
  10:21:37 STEP 1952 H=1952:00 +CPU= 27.381
  10:22:23 STEP 1953 H=1953:00 +CPU= 41.984
  10:22:53 STEP 1954 H=1954:00 +CPU= 27.678
  10:23:23 STEP 1955 H=1955:00 +CPU= 26.593
  10:24:13 STEP 1956 H=1956:00 +CPU= 46.682
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001956
  10:24:46 STEP 1957 H=1957:00 +CPU= 26.554
  10:25:15 STEP 1958 H=1958:00 +CPU= 27.198
  10:26:01 STEP 1959 H=1959:00 +CPU= 41.904
  10:26:31 STEP 1960 H=1960:00 +CPU= 27.655
  10:26:59 STEP 1961 H=1961:00 +CPU= 26.172
  10:27:45 STEP 1962 H=1962:00 +CPU= 42.424
  10:28:14 STEP 1963 H=1963:00 +CPU= 26.390
  10:28:43 STEP 1964 H=1964:00 +CPU= 27.039
  10:29:29 STEP 1965 H=1965:00 +CPU= 41.604
  10:29:58 STEP 1966 H=1966:00 +CPU= 27.211
  10:30:28 STEP 1967 H=1967:00 +CPU= 27.206
  10:31:28 STEP 1968 H=1968:00 +CPU= 49.070
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_81.zip
Uploading the intermediate file: upload_file_81.zip
  10:31:57 STEP 1969 H=1969:00 +CPU= 26.202
  10:32:26 STEP 1970 H=1970:00 +CPU= 26.947
  10:33:11 STEP 1971 H=1971:00 +CPU= 41.725
  10:33:39 STEP 1972 H=1972:00 +CPU= 26.988
  10:34:07 STEP 1973 H=1973:00 +CPU= 26.136
  10:34:54 STEP 1974 H=1974:00 +CPU= 42.765
  10:35:23 STEP 1975 H=1975:00 +CPU= 26.762
  10:35:52 STEP 1976 H=1976:00 +CPU= 27.390
  10:36:37 STEP 1977 H=1977:00 +CPU= 41.656
  10:37:06 STEP 1978 H=1978:00 +CPU= 28.030
  10:37:35 STEP 1979 H=1979:00 +CPU= 26.506
  10:38:24 STEP 1980 H=1980:00 +CPU= 46.097
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001980
  10:38:52 STEP 1981 H=1981:00 +CPU= 26.197
  10:39:21 STEP 1982 H=1982:00 +CPU= 27.221
  10:40:05 STEP 1983 H=1983:00 +CPU= 41.435
  10:40:34 STEP 1984 H=1984:00 +CPU= 27.208
  10:41:02 STEP 1985 H=1985:00 +CPU= 26.247
  10:41:48 STEP 1986 H=1986:00 +CPU= 42.860
  10:42:17 STEP 1987 H=1987:00 +CPU= 26.435
  10:42:46 STEP 1988 H=1988:00 +CPU= 27.187
  10:43:35 STEP 1989 H=1989:00 +CPU= 41.885
  10:44:05 STEP 1990 H=1990:00 +CPU= 27.154
  10:44:34 STEP 1991 H=1991:00 +CPU= 27.259
  10:45:30 STEP 1992 H=1992:00 +CPU= 48.873
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_82.zip
Uploading the intermediate file: upload_file_82.zip
  10:45:58 STEP 1993 H=1993:00 +CPU= 26.423
  10:46:28 STEP 1994 H=1994:00 +CPU= 27.468
  10:47:13 STEP 1995 H=1995:00 +CPU= 42.010
  10:47:42 STEP 1996 H=1996:00 +CPU= 27.329
  10:48:11 STEP 1997 H=1997:00 +CPU= 26.549
  10:48:56 STEP 1998 H=1998:00 +CPU= 42.347
  10:49:24 STEP 1999 H=1999:00 +CPU= 26.460
  10:49:53 STEP 2000 H=2000:00 +CPU= 27.123
  10:50:41 STEP 2001 H=2001:00 +CPU= 42.043
  10:51:11 STEP 2002 H=2002:00 +CPU= 27.112
  10:51:41 STEP 2003 H=2003:00 +CPU= 26.713
  10:52:37 STEP 2004 H=2004:00 +CPU= 46.893
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002004
  10:53:06 STEP 2005 H=2005:00 +CPU= 26.716
  10:53:36 STEP 2006 H=2006:00 +CPU= 27.510
  10:54:23 STEP 2007 H=2007:00 +CPU= 41.900
  10:54:52 STEP 2008 H=2008:00 +CPU= 27.052
  10:55:21 STEP 2009 H=2009:00 +CPU= 26.349
  10:56:08 STEP 2010 H=2010:00 +CPU= 42.691
  10:56:39 STEP 2011 H=2011:00 +CPU= 26.754
  10:57:08 STEP 2012 H=2012:00 +CPU= 27.307
  10:57:54 STEP 2013 H=2013:00 +CPU= 41.750
  10:58:23 STEP 2014 H=2014:00 +CPU= 27.198
  10:58:53 STEP 2015 H=2015:00 +CPU= 27.469
  10:59:53 STEP 2016 H=2016:00 +CPU= 49.848
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_83.zip
Uploading the intermediate file: upload_file_83.zip
  11:00:26 STEP 2017 H=2017:00 +CPU= 26.709
  11:00:57 STEP 2018 H=2018:00 +CPU= 27.152
  11:01:42 STEP 2019 H=2019:00 +CPU= 41.417
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0441
(argv4) batchid: 950
(argv5) wuid: 12167085
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4d99730c22a7c993fdb6ade587f1c9ce to: /var/lib/boinc/slots/1/oifs_43r3_ps_0441_1981050100_123_950_12167085.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0441_1981050100_123_950_12167085.zip
ic_ancil_file: ic_ancil_12167085
ifsdata_file: ifsdata_12167085
climate_data_file: clim_data_12167085
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3491416c55535c88ee2f1f0531242c15 to: /var/lib/boinc/slots/1/ic_ancil_12167085.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12167085.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9603068b1261249a178e76b73a262bbc to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12167085.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12167085.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12167085.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12167085.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12167085.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12167085.xml
-- Model is restarting --
Adjusting last_iter, 2020, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12167085.xml
last_cpu_time: 66940
upload_file_number: 84
last_iter: 2015
last_upload: 7257600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270
The child process has been launched with process id: 24024
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:24024:24024] [20230127:112733:1674844053:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:28:33 STEP 2016 H=2016:00 +CPU= 47.098
  11:29:02 STEP 2017 H=2017:00 +CPU= 26.559
  11:29:32 STEP 2018 H=2018:00 +CPU= 27.516
  11:30:16 STEP 2019 H=2019:00 +CPU= 41.738
  11:30:45 STEP 2020 H=2020:00 +CPU= 27.222
  11:31:12 STEP 2021 H=2021:00 +CPU= 26.359
  11:31:58 STEP 2022 H=2022:00 +CPU= 42.797
  11:32:25 STEP 2023 H=2023:00 +CPU= 26.174
  11:32:54 STEP 2024 H=2024:00 +CPU= 27.020
  11:33:38 STEP 2025 H=2025:00 +CPU= 41.586
  11:34:07 STEP 2026 H=2026:00 +CPU= 27.062
  11:34:35 STEP 2027 H=2027:00 +CPU= 26.488
  11:35:28 STEP 2028 H=2028:00 +CPU= 46.879
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002028
  11:36:01 STEP 2029 H=2029:00 +CPU= 26.907
  11:36:34 STEP 2030 H=2030:00 +CPU= 27.963
  11:37:27 STEP 2031 H=2031:00 +CPU= 42.761
  11:37:59 STEP 2032 H=2032:00 +CPU= 27.901
  11:38:29 STEP 2033 H=2033:00 +CPU= 26.634
  11:39:17 STEP 2034 H=2034:00 +CPU= 43.161
  11:39:47 STEP 2035 H=2035:00 +CPU= 26.501
  11:40:18 STEP 2036 H=2036:00 +CPU= 27.666
  11:41:05 STEP 2037 H=2037:00 +CPU= 42.137
  11:41:36 STEP 2038 H=2038:00 +CPU= 27.515
  11:42:08 STEP 2039 H=2039:00 +CPU= 28.268
  11:43:03 STEP 2040 H=2040:00 +CPU= 49.279
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167085/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0441_1981050100_123_950_12167085_2_r727100270_84.zip
Uploading the intermediate file: upload_file_84.zip
  11:43:33 STEP 2041 H=2041:00 +CPU= 26.814
  11:44:03 STEP 2042 H=2042:00 +CPU= 27.502
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:42:57 1493665 22303872 oifs_43r3_ps_0441_1981050100_123_950_12167085_2 4,665,600 42,948 0.0092
09 May 2023 16:42:28 1493665 22303872 oifs_43r3_ps_0441_1981050100_123_950_12167085_2 3,110,400 28,355 0.0091
09 May 2023 16:42:21 1493665 22303872 oifs_43r3_ps_0441_1981050100_123_950_12167085_2 2,246,400 20,353 0.0091
09 May 2023 16:42:09 1493665 22303872 oifs_43r3_ps_0441_1981050100_123_950_12167085_2 864,000 7,812 0.0090


©2024 cpdn.org