climateprediction.net home page
Task 22254927

Task 22254927

Name oifs_43r3_ps_0961_1984050100_123_953_12170605_0
Workunit 12170605
Created 21 Dec 2022, 4:45:32 UTC
Sent 5 Jan 2023, 4:08:27 UTC
Report deadline 4 Feb 2023, 4:08:27 UTC
Received 9 Jan 2023, 15:22:26 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1515226
Run time 17 hours 56 min 16 sec
CPU time 17 hours 34 min 52 sec
Validate state Invalid
Credit 8,467.20
Device peak FLOPS 4.58 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,403.68 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,351.06 MB
Stderr 
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
05/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_84.zip
Uploading the intermediate file: upload_file_84.zip
  23:42:27 STEP 2041 H=2041:00 +CPU= 23.136
  23:42:51 STEP 2042 H=2042:00 +CPU= 23.812
  23:43:27 STEP 2043 H=2043:00 +CPU= 35.917
  23:43:51 STEP 2044 H=2044:00 +CPU= 23.836
  23:44:14 STEP 2045 H=2045:00 +CPU= 23.106
  23:44:51 STEP 2046 H=2046:00 +CPU= 36.640
  23:45:14 STEP 2047 H=2047:00 +CPU= 22.993
  23:47:01 STEP 2048 H=2048:00 +CPU= 24.310
  23:47:38 STEP 2049 H=2049:00 +CPU= 36.240
  23:48:02 STEP 2050 H=2050:00 +CPU= 24.180
  23:48:26 STEP 2051 H=2051:00 +CPU= 23.065
  23:49:06 STEP 2052 H=2052:00 +CPU= 40.480
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002052
  23:49:30 STEP 2053 H=2053:00 +CPU= 23.382
  23:49:54 STEP 2054 H=2054:00 +CPU= 24.017
  23:50:31 STEP 2055 H=2055:00 +CPU= 36.366
  23:50:55 STEP 2056 H=2056:00 +CPU= 24.126
  23:51:20 STEP 2057 H=2057:00 +CPU= 23.234
  23:51:57 STEP 2058 H=2058:00 +CPU= 37.239
  23:52:20 STEP 2059 H=2059:00 +CPU= 23.291
  23:52:44 STEP 2060 H=2060:00 +CPU= 23.901
  23:53:21 STEP 2061 H=2061:00 +CPU= 36.315
  23:53:45 STEP 2062 H=2062:00 +CPU= 23.837
  23:54:09 STEP 2063 H=2063:00 +CPU= 23.770
  23:54:58 STEP 2064 H=2064:00 +CPU= 42.752
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_85.zip
Uploading the intermediate file: upload_file_85.zip
  23:55:22 STEP 2065 H=2065:00 +CPU= 23.412
  23:55:46 STEP 2066 H=2066:00 +CPU= 23.898
  23:56:22 STEP 2067 H=2067:00 +CPU= 36.448
  23:56:47 STEP 2068 H=2068:00 +CPU= 24.147
  23:57:10 STEP 2069 H=2069:00 +CPU= 23.272
  23:57:47 STEP 2070 H=2070:00 +CPU= 37.159
  23:58:11 STEP 2071 H=2071:00 +CPU= 23.454
  23:58:35 STEP 2072 H=2072:00 +CPU= 24.140
  23:59:11 STEP 2073 H=2073:00 +CPU= 36.329
  23:59:35 STEP 2074 H=2074:00 +CPU= 24.284
  23:59:59 STEP 2075 H=2075:00 +CPU= 23.188
  00:00:40 STEP 2076 H=2076:00 +CPU= 40.685
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002076
  00:01:03 STEP 2077 H=2077:00 +CPU= 23.363
  00:01:27 STEP 2078 H=2078:00 +CPU= 24.328
  00:02:04 STEP 2079 H=2079:00 +CPU= 36.760
  00:02:28 STEP 2080 H=2080:00 +CPU= 24.193
  00:02:52 STEP 2081 H=2081:00 +CPU= 23.476
  00:03:30 STEP 2082 H=2082:00 +CPU= 37.596
  00:03:53 STEP 2083 H=2083:00 +CPU= 23.366
  00:04:17 STEP 2084 H=2084:00 +CPU= 24.243
  00:04:54 STEP 2085 H=2085:00 +CPU= 36.693
  00:05:18 STEP 2086 H=2086:00 +CPU= 23.814
  00:05:42 STEP 2087 H=2087:00 +CPU= 23.806
  00:06:28 STEP 2088 H=2088:00 +CPU= 42.270
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_86.zip
Uploading the intermediate file: upload_file_86.zip
  00:06:51 STEP 2089 H=2089:00 +CPU= 23.297
  00:07:15 STEP 2090 H=2090:00 +CPU= 24.224
  00:07:52 STEP 2091 H=2091:00 +CPU= 36.437
  00:08:16 STEP 2092 H=2092:00 +CPU= 24.037
  00:08:39 STEP 2093 H=2093:00 +CPU= 23.338
  00:09:17 STEP 2094 H=2094:00 +CPU= 37.447
  00:09:41 STEP 2095 H=2095:00 +CPU= 23.408
  00:10:05 STEP 2096 H=2096:00 +CPU= 24.367
  00:10:42 STEP 2097 H=2097:00 +CPU= 36.825
  00:11:07 STEP 2098 H=2098:00 +CPU= 24.263
  00:11:30 STEP 2099 H=2099:00 +CPU= 23.306
  00:12:11 STEP 2100 H=2100:00 +CPU= 40.804
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002100
  00:12:34 STEP 2101 H=2101:00 +CPU= 23.567
  00:12:59 STEP 2102 H=2102:00 +CPU= 24.327
  00:13:35 STEP 2103 H=2103:00 +CPU= 36.754
  00:13:59 STEP 2104 H=2104:00 +CPU= 23.972
  00:14:23 STEP 2105 H=2105:00 +CPU= 23.384
  00:15:00 STEP 2106 H=2106:00 +CPU= 37.400
  00:15:24 STEP 2107 H=2107:00 +CPU= 23.147
  00:15:48 STEP 2108 H=2108:00 +CPU= 24.259
  00:16:24 STEP 2109 H=2109:00 +CPU= 36.460
  00:16:48 STEP 2110 H=2110:00 +CPU= 23.807
  00:17:12 STEP 2111 H=2111:00 +CPU= 24.007
  00:18:00 STEP 2112 H=2112:00 +CPU= 42.416
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_87.zip
Uploading the intermediate file: upload_file_87.zip
  00:18:23 STEP 2113 H=2113:00 +CPU= 23.130
  00:18:47 STEP 2114 H=2114:00 +CPU= 23.928
  00:19:24 STEP 2115 H=2115:00 +CPU= 36.152
  00:19:48 STEP 2116 H=2116:00 +CPU= 23.848
  00:20:11 STEP 2117 H=2117:00 +CPU= 23.212
  00:20:48 STEP 2118 H=2118:00 +CPU= 36.920
  00:21:11 STEP 2119 H=2119:00 +CPU= 23.224
  00:21:35 STEP 2120 H=2120:00 +CPU= 23.943
  00:22:11 STEP 2121 H=2121:00 +CPU= 36.007
  00:22:35 STEP 2122 H=2122:00 +CPU= 23.989
  00:22:58 STEP 2123 H=2123:00 +CPU= 23.260
  00:23:39 STEP 2124 H=2124:00 +CPU= 40.223
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002124
  00:24:02 STEP 2125 H=2125:00 +CPU= 23.149
  00:24:26 STEP 2126 H=2126:00 +CPU= 23.975
  00:25:02 STEP 2127 H=2127:00 +CPU= 36.167
  00:25:27 STEP 2128 H=2128:00 +CPU= 23.958
  00:25:50 STEP 2129 H=2129:00 +CPU= 23.041
  00:26:27 STEP 2130 H=2130:00 +CPU= 37.022
  00:26:50 STEP 2131 H=2131:00 +CPU= 23.581
  00:27:15 STEP 2132 H=2132:00 +CPU= 24.518
  00:27:52 STEP 2133 H=2133:00 +CPU= 37.055
  00:28:15 STEP 2134 H=2134:00 +CPU= 23.130
  00:28:40 STEP 2135 H=2135:00 +CPU= 24.453
  00:29:26 STEP 2136 H=2136:00 +CPU= 42.964
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_88.zip
Uploading the intermediate file: upload_file_88.zip
  00:29:49 STEP 2137 H=2137:00 +CPU= 23.323
  00:30:13 STEP 2138 H=2138:00 +CPU= 24.056
  00:30:50 STEP 2139 H=2139:00 +CPU= 36.546
  00:31:14 STEP 2140 H=2140:00 +CPU= 24.197
  00:31:38 STEP 2141 H=2141:00 +CPU= 23.411
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0961
(argv4) batchid: 953
(argv5) wuid: 12170605
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_77a7ea4be94e3fc7739e322b9042cae4 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1984050100_123_953_12170605.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1984050100_123_953_12170605.zip
ic_ancil_file: ic_ancil_12170605
ifsdata_file: ifsdata_12170605
climate_data_file: clim_data_12170605
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_47c7ae48391dc870d656de8603bf0780 to: /var/lib/boinc-client/slots/0/ic_ancil_12170605.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12170605.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9b1340e8345bc6b3adf712578a5eb318 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12170605.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12170605.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12170605.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12170605.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12170605.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12170605.xml
-- Model is restarting --
Adjusting last_iter, 2142, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12170605.xml
last_cpu_time: 57814
upload_file_number: 89
last_iter: 2135
last_upload: 7689600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600
The child process has been launched with process id: 2597
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:2597:2597] [20230107:013245:1673083965:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:33:52 STEP 2136 H=2136:00 +CPU= 40.346
  01:35:27 STEP 2137 H=2137:00 +CPU= 23.639
  01:36:45 STEP 2138 H=2138:00 +CPU= 24.794
  01:37:24 STEP 2139 H=2139:00 +CPU= 36.113
  01:37:48 STEP 2140 H=2140:00 +CPU= 23.989
  01:38:11 STEP 2141 H=2141:00 +CPU= 23.162
  01:38:54 STEP 2142 H=2142:00 +CPU= 37.363
  01:39:17 STEP 2143 H=2143:00 +CPU= 23.463
  01:39:42 STEP 2144 H=2144:00 +CPU= 24.256
  01:40:25 STEP 2145 H=2145:00 +CPU= 36.714
  01:40:50 STEP 2146 H=2146:00 +CPU= 24.327
  01:41:14 STEP 2147 H=2147:00 +CPU= 23.289
  01:42:02 STEP 2148 H=2148:00 +CPU= 40.293
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002148
  01:42:26 STEP 2149 H=2149:00 +CPU= 23.557
  01:42:50 STEP 2150 H=2150:00 +CPU= 24.097
  01:43:26 STEP 2151 H=2151:00 +CPU= 35.968
  01:43:50 STEP 2152 H=2152:00 +CPU= 23.966
  01:44:13 STEP 2153 H=2153:00 +CPU= 23.016
  01:44:51 STEP 2154 H=2154:00 +CPU= 37.044
  01:45:14 STEP 2155 H=2155:00 +CPU= 23.234
  01:45:38 STEP 2156 H=2156:00 +CPU= 23.843
  01:46:14 STEP 2157 H=2157:00 +CPU= 36.158
  01:46:38 STEP 2158 H=2158:00 +CPU= 23.965
  01:47:03 STEP 2159 H=2159:00 +CPU= 23.591
  01:47:58 STEP 2160 H=2160:00 +CPU= 42.878
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_89.zip
Uploading the intermediate file: upload_file_89.zip
  01:48:22 STEP 2161 H=2161:00 +CPU= 23.296
  01:48:46 STEP 2162 H=2162:00 +CPU= 23.878
  01:49:22 STEP 2163 H=2163:00 +CPU= 36.197
  01:49:46 STEP 2164 H=2164:00 +CPU= 23.997
  01:50:16 STEP 2165 H=2165:00 +CPU= 23.711
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0961
(argv4) batchid: 953
(argv5) wuid: 12170605
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_77a7ea4be94e3fc7739e322b9042cae4 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1984050100_123_953_12170605.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1984050100_123_953_12170605.zip
ic_ancil_file: ic_ancil_12170605
ifsdata_file: ifsdata_12170605
climate_data_file: clim_data_12170605
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_47c7ae48391dc870d656de8603bf0780 to: /var/lib/boinc-client/slots/0/ic_ancil_12170605.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12170605.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9b1340e8345bc6b3adf712578a5eb318 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12170605.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12170605.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12170605.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12170605.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12170605.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12170605.xml
-- Model is restarting --
Adjusting last_iter, 2166, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12170605.xml
last_cpu_time: 58698
upload_file_number: 90
last_iter: 2159
last_upload: 7776000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600
The child process has been launched with process id: 3195
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:3195:3195] [20230107:104343:1673117023:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:44:55 STEP 2160 H=2160:00 +CPU= 40.926
  10:45:47 STEP 2161 H=2161:00 +CPU= 23.403
  10:46:16 STEP 2162 H=2162:00 +CPU= 24.048
  10:46:56 STEP 2163 H=2163:00 +CPU= 35.809
  10:47:20 STEP 2164 H=2164:00 +CPU= 24.167
  10:47:43 STEP 2165 H=2165:00 +CPU= 23.413
  10:48:22 STEP 2166 H=2166:00 +CPU= 37.087
  10:48:46 STEP 2167 H=2167:00 +CPU= 23.390
  10:49:10 STEP 2168 H=2168:00 +CPU= 24.046
  10:49:46 STEP 2169 H=2169:00 +CPU= 36.124
  10:50:10 STEP 2170 H=2170:00 +CPU= 24.037
  10:50:33 STEP 2171 H=2171:00 +CPU= 23.171
  10:51:18 STEP 2172 H=2172:00 +CPU= 40.888
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002172
  10:51:42 STEP 2173 H=2173:00 +CPU= 24.192
  10:52:07 STEP 2174 H=2174:00 +CPU= 24.891
  10:52:45 STEP 2175 H=2175:00 +CPU= 37.061
  10:53:10 STEP 2176 H=2176:00 +CPU= 24.873
  10:53:34 STEP 2177 H=2177:00 +CPU= 24.008
  10:54:12 STEP 2178 H=2178:00 +CPU= 37.378
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0961
(argv4) batchid: 953
(argv5) wuid: 12170605
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_77a7ea4be94e3fc7739e322b9042cae4 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1984050100_123_953_12170605.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1984050100_123_953_12170605.zip
ic_ancil_file: ic_ancil_12170605
ifsdata_file: ifsdata_12170605
climate_data_file: clim_data_12170605
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_47c7ae48391dc870d656de8603bf0780 to: /var/lib/boinc-client/slots/0/ic_ancil_12170605.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12170605.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9b1340e8345bc6b3adf712578a5eb318 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12170605.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12170605.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12170605.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12170605.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12170605.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12170605.xml
-- Model is restarting --
Adjusting last_iter, 2179, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12170605.xml
last_cpu_time: 59269
upload_file_number: 90
last_iter: 2159
last_upload: 7776000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600
The child process has been launched with process id: 5760
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5760:5760] [20230109:023622:1673260582:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:37:32 STEP 2160 H=2160:00 +CPU= 41.394
  02:38:35 STEP 2161 H=2161:00 +CPU= 24.469
  02:39:06 STEP 2162 H=2162:00 +CPU= 25.395
  02:39:46 STEP 2163 H=2163:00 +CPU= 36.800
  02:40:11 STEP 2164 H=2164:00 +CPU= 24.595
  02:40:35 STEP 2165 H=2165:00 +CPU= 23.942
  02:41:14 STEP 2166 H=2166:00 +CPU= 37.436
  02:41:39 STEP 2167 H=2167:00 +CPU= 24.711
  02:42:04 STEP 2168 H=2168:00 +CPU= 25.597
  02:42:43 STEP 2169 H=2169:00 +CPU= 37.802
  02:43:09 STEP 2170 H=2170:00 +CPU= 25.339
  02:43:33 STEP 2171 H=2171:00 +CPU= 24.230
  02:44:18 STEP 2172 H=2172:00 +CPU= 40.849
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002172
  02:44:43 STEP 2173 H=2173:00 +CPU= 24.261
  02:45:08 STEP 2174 H=2174:00 +CPU= 24.976
  02:45:40 STEP 2175 H=2175:00 +CPU= 31.355
  02:45:55 STEP 2176 H=2176:00 +CPU= 15.546
  02:46:10 STEP 2177 H=2177:00 +CPU= 15.120
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0961
(argv4) batchid: 953
(argv5) wuid: 12170605
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_77a7ea4be94e3fc7739e322b9042cae4 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1984050100_123_953_12170605.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1984050100_123_953_12170605.zip
ic_ancil_file: ic_ancil_12170605
ifsdata_file: ifsdata_12170605
climate_data_file: clim_data_12170605
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_47c7ae48391dc870d656de8603bf0780 to: /var/lib/boinc-client/slots/0/ic_ancil_12170605.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12170605.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9b1340e8345bc6b3adf712578a5eb318 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12170605.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12170605.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12170605.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12170605.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12170605.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12170605.xml
-- Model is restarting --
Adjusting last_iter, 2178, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12170605.xml
last_cpu_time: 59788
upload_file_number: 90
last_iter: 2159
last_upload: 7776000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600
The child process has been launched with process id: 5917
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Ken-Boinc-4:1:1:5917:5917] [20230109:051939:1673270379:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:20:52 STEP 2160 H=2160:00 +CPU= 41.629
  05:22:25 STEP 2161 H=2161:00 +CPU= 23.761
  05:22:59 STEP 2162 H=2162:00 +CPU= 24.493
  05:23:39 STEP 2163 H=2163:00 +CPU= 37.372
  05:24:04 STEP 2164 H=2164:00 +CPU= 24.977
  05:24:28 STEP 2165 H=2165:00 +CPU= 24.163
  05:25:06 STEP 2166 H=2166:00 +CPU= 38.353
  05:25:31 STEP 2167 H=2167:00 +CPU= 24.153
  05:25:56 STEP 2168 H=2168:00 +CPU= 24.968
  05:26:33 STEP 2169 H=2169:00 +CPU= 37.415
  05:26:58 STEP 2170 H=2170:00 +CPU= 24.928
  05:27:22 STEP 2171 H=2171:00 +CPU= 23.382
  05:28:09 STEP 2172 H=2172:00 +CPU= 40.644
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002172
  05:28:54 STEP 2173 H=2173:00 +CPU= 23.936
  05:29:29 STEP 2174 H=2174:00 +CPU= 24.785
  05:30:07 STEP 2175 H=2175:00 +CPU= 37.702
  05:30:32 STEP 2176 H=2176:00 +CPU= 25.300
  05:30:57 STEP 2177 H=2177:00 +CPU= 24.443
  05:31:35 STEP 2178 H=2178:00 +CPU= 38.574
  05:32:00 STEP 2179 H=2179:00 +CPU= 24.326
  05:32:25 STEP 2180 H=2180:00 +CPU= 25.105
  05:33:03 STEP 2181 H=2181:00 +CPU= 37.564
  05:33:28 STEP 2182 H=2182:00 +CPU= 25.136
  05:33:54 STEP 2183 H=2183:00 +CPU= 25.146
  05:34:51 STEP 2184 H=2184:00 +CPU= 43.999
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_90.zip
Uploading the intermediate file: upload_file_90.zip
  05:35:16 STEP 2185 H=2185:00 +CPU= 24.088
  05:35:41 STEP 2186 H=2186:00 +CPU= 24.520
  05:36:17 STEP 2187 H=2187:00 +CPU= 36.653
  05:36:42 STEP 2188 H=2188:00 +CPU= 24.455
  05:37:06 STEP 2189 H=2189:00 +CPU= 23.730
  05:37:43 STEP 2190 H=2190:00 +CPU= 37.512
  05:38:07 STEP 2191 H=2191:00 +CPU= 23.797
  05:38:32 STEP 2192 H=2192:00 +CPU= 24.516
  05:39:09 STEP 2193 H=2193:00 +CPU= 36.619
  05:39:33 STEP 2194 H=2194:00 +CPU= 24.524
  05:39:57 STEP 2195 H=2195:00 +CPU= 23.681
  05:40:38 STEP 2196 H=2196:00 +CPU= 40.658
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002196
  05:41:02 STEP 2197 H=2197:00 +CPU= 23.704
  05:41:26 STEP 2198 H=2198:00 +CPU= 24.514
  05:42:03 STEP 2199 H=2199:00 +CPU= 36.602
  05:42:28 STEP 2200 H=2200:00 +CPU= 24.420
  05:42:51 STEP 2201 H=2201:00 +CPU= 23.725
  05:43:29 STEP 2202 H=2202:00 +CPU= 37.383
  05:43:53 STEP 2203 H=2203:00 +CPU= 23.772
  05:44:17 STEP 2204 H=2204:00 +CPU= 24.454
  05:44:54 STEP 2205 H=2205:00 +CPU= 36.534
  05:45:18 STEP 2206 H=2206:00 +CPU= 24.444
  05:45:43 STEP 2207 H=2207:00 +CPU= 24.384
  05:46:29 STEP 2208 H=2208:00 +CPU= 42.128
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_91.zip
Uploading the intermediate file: upload_file_91.zip
  05:46:53 STEP 2209 H=2209:00 +CPU= 23.686
  05:47:17 STEP 2210 H=2210:00 +CPU= 24.439
  05:47:59 STEP 2211 H=2211:00 +CPU= 36.705
  05:48:24 STEP 2212 H=2212:00 +CPU= 24.498
  05:48:48 STEP 2213 H=2213:00 +CPU= 23.799
  05:49:26 STEP 2214 H=2214:00 +CPU= 37.773
  05:49:50 STEP 2215 H=2215:00 +CPU= 23.744
  05:50:14 STEP 2216 H=2216:00 +CPU= 24.506
  05:50:51 STEP 2217 H=2217:00 +CPU= 36.698
  05:51:21 STEP 2218 H=2218:00 +CPU= 24.818
  05:51:45 STEP 2219 H=2219:00 +CPU= 23.638
  05:52:26 STEP 2220 H=2220:00 +CPU= 40.745
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002220
  05:52:54 STEP 2221 H=2221:00 +CPU= 24.131
  05:53:20 STEP 2222 H=2222:00 +CPU= 25.578
  05:53:58 STEP 2223 H=2223:00 +CPU= 38.077
  05:54:23 STEP 2224 H=2224:00 +CPU= 25.320
  05:54:48 STEP 2225 H=2225:00 +CPU= 24.484
  05:55:27 STEP 2226 H=2226:00 +CPU= 38.836
  05:55:51 STEP 2227 H=2227:00 +CPU= 24.550
  05:56:17 STEP 2228 H=2228:00 +CPU= 25.411
  05:56:55 STEP 2229 H=2229:00 +CPU= 38.020
  05:57:20 STEP 2230 H=2230:00 +CPU= 25.351
  05:57:45 STEP 2231 H=2231:00 +CPU= 25.298
  05:58:34 STEP 2232 H=2232:00 +CPU= 44.312
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_92.zip
Uploading the intermediate file: upload_file_92.zip
  05:58:59 STEP 2233 H=2233:00 +CPU= 24.476
  05:59:24 STEP 2234 H=2234:00 +CPU= 24.438
  06:00:01 STEP 2235 H=2235:00 +CPU= 36.783
  06:00:25 STEP 2236 H=2236:00 +CPU= 24.490
  06:00:49 STEP 2237 H=2237:00 +CPU= 23.757
  06:01:27 STEP 2238 H=2238:00 +CPU= 37.982
  06:01:51 STEP 2239 H=2239:00 +CPU= 23.792
  06:02:15 STEP 2240 H=2240:00 +CPU= 24.570
  06:02:52 STEP 2241 H=2241:00 +CPU= 37.050
  06:03:17 STEP 2242 H=2242:00 +CPU= 24.453
  06:03:41 STEP 2243 H=2243:00 +CPU= 23.626
  06:04:22 STEP 2244 H=2244:00 +CPU= 40.944
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002244
  06:04:45 STEP 2245 H=2245:00 +CPU= 23.687
  06:05:10 STEP 2246 H=2246:00 +CPU= 24.691
  06:05:47 STEP 2247 H=2247:00 +CPU= 37.384
  06:06:12 STEP 2248 H=2248:00 +CPU= 24.320
  06:06:35 STEP 2249 H=2249:00 +CPU= 23.490
  06:07:13 STEP 2250 H=2250:00 +CPU= 37.711
  06:07:37 STEP 2251 H=2251:00 +CPU= 23.364
  06:08:01 STEP 2252 H=2252:00 +CPU= 24.560
  06:08:38 STEP 2253 H=2253:00 +CPU= 37.001
  06:09:03 STEP 2254 H=2254:00 +CPU= 24.102
  06:09:27 STEP 2255 H=2255:00 +CPU= 24.434
  06:10:13 STEP 2256 H=2256:00 +CPU= 43.430
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170605/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1984050100_123_953_12170605_0_r1808264600_93.zip
Uploading the intermediate file: upload_file_93.zip
  06:10:39 STEP 2257 H=2257:00 +CPU= 23.447
  06:11:04 STEP 2258 H=2258:00 +CPU= 25.000
  06:11:42 STEP 2259 H=2259:00 +CPU= 37.719
  06:12:06 STEP 2260 H=2260:00 +CPU= 24.716
  06:12:31 STEP 2261 H=2261:00 +CPU= 24.242
  06:13:09 STEP 2262 H=2262:00 +CPU= 38.682
  06:13:33 STEP 2263 H=2263:00 +CPU= 23.921
  06:13:59 STEP 2264 H=2264:00 +CPU= 25.131
  06:14:36 STEP 2265 H=2265:00 +CPU= 37.716
  06:15:01 STEP 2266 H=2266:00 +CPU= 24.783
  06:15:26 STEP 2267 H=2267:00 +CPU= 24.318
  06:16:08 STEP 2268 H=2268:00 +CPU= 42.221
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002268
  06:16:32 STEP 2269 H=2269:00 +CPU= 23.909
  06:16:57 STEP 2270 H=2270:00 +CPU= 25.086
  06:17:35 STEP 2271 H=2271:00 +CPU= 37.932
  06:18:00 STEP 2272 H=2272:00 +CPU= 24.729
  06:18:24 STEP 2273 H=2273:00 +CPU= 24.377
  06:19:03 STEP 2274 H=2274:00 +CPU= 38.768
  06:19:27 STEP 2275 H=2275:00 +CPU= 23.948
  06:19:52 STEP 2276 H=2276:00 +CPU= 25.109
  06:20:30 STEP 2277 H=2277:00 +CPU= 37.727
  06:20:55 STEP 2278 H=2278:00 +CPU= 24.898
  06:21:20 STEP 2279 H=2279:00 +CPU= 25.172
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:10:32 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 6,048,000 44,442 0.0073
09 May 2023 16:10:25 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 5,184,000 38,011 0.0073
09 May 2023 16:10:19 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 4,320,000 31,782 0.0074
09 May 2023 16:10:19 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 4,320,000 31,782 0.0074
09 May 2023 16:10:13 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 3,456,000 25,464 0.0074
09 May 2023 16:10:07 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 2,592,000 19,055 0.0074
09 May 2023 16:10:00 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 1,728,000 12,710 0.0074
09 May 2023 16:10:00 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 1,728,000 12,710 0.0074
09 May 2023 16:09:54 1515226 22254927 oifs_43r3_ps_0961_1984050100_123_953_12170605_0 864,000 6,344 0.0073


©2024 cpdn.org