climateprediction.net home page
Task 22295768

Task 22295768

Name oifs_43r3_ps_0517_1996050100_123_965_12182161_1
Workunit 12182161
Created 5 Jan 2023, 10:27:02 UTC
Sent 5 Jan 2023, 11:02:20 UTC
Report deadline 4 Feb 2023, 11:02:20 UTC
Received 6 Jan 2023, 5:36:12 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532338
Run time 15 hours 14 min 24 sec
CPU time 15 hours 12 min 59 sec
Validate state Invalid
Credit 11,007.36
Device peak FLOPS 5.62 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,786.15 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,514.62 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
STEP 2016 H=2016:00 +CPU= 29.439
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_83.zip
Uploading the intermediate file: upload_file_83.zip
  00:34:36 STEP 2017 H=2017:00 +CPU= 14.725
  00:34:52 STEP 2018 H=2018:00 +CPU= 15.976
  00:35:18 STEP 2019 H=2019:00 +CPU= 26.098
  00:35:34 STEP 2020 H=2020:00 +CPU= 15.749
  00:35:49 STEP 2021 H=2021:00 +CPU= 15.174
  00:36:15 STEP 2022 H=2022:00 +CPU= 25.420
  00:36:29 STEP 2023 H=2023:00 +CPU= 14.122
  00:36:43 STEP 2024 H=2024:00 +CPU= 14.547
  00:37:07 STEP 2025 H=2025:00 +CPU= 23.822
  00:37:22 STEP 2026 H=2026:00 +CPU= 15.285
  00:37:38 STEP 2027 H=2027:00 +CPU= 15.685
  00:38:05 STEP 2028 H=2028:00 +CPU= 27.304
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002028
  00:38:22 STEP 2029 H=2029:00 +CPU= 16.441
  00:38:38 STEP 2030 H=2030:00 +CPU= 16.418
  00:39:04 STEP 2031 H=2031:00 +CPU= 25.479
  00:39:20 STEP 2032 H=2032:00 +CPU= 15.919
  00:39:35 STEP 2033 H=2033:00 +CPU= 15.462
  00:40:01 STEP 2034 H=2034:00 +CPU= 26.074
  00:40:17 STEP 2035 H=2035:00 +CPU= 15.669
  00:40:33 STEP 2036 H=2036:00 +CPU= 16.400
  00:41:00 STEP 2037 H=2037:00 +CPU= 26.754
  00:41:18 STEP 2038 H=2038:00 +CPU= 17.777
  00:41:35 STEP 2039 H=2039:00 +CPU= 16.861
  00:42:05 STEP 2040 H=2040:00 +CPU= 30.658
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_84.zip
Uploading the intermediate file: upload_file_84.zip
  00:42:22 STEP 2041 H=2041:00 +CPU= 16.290
  00:42:38 STEP 2042 H=2042:00 +CPU= 16.560
  00:43:03 STEP 2043 H=2043:00 +CPU= 24.641
  00:43:18 STEP 2044 H=2044:00 +CPU= 15.666
  00:43:33 STEP 2045 H=2045:00 +CPU= 14.790
  00:43:57 STEP 2046 H=2046:00 +CPU= 23.966
  00:44:12 STEP 2047 H=2047:00 +CPU= 14.991
  00:44:27 STEP 2048 H=2048:00 +CPU= 15.218
  00:44:52 STEP 2049 H=2049:00 +CPU= 24.147
  00:45:07 STEP 2050 H=2050:00 +CPU= 15.349
  00:45:22 STEP 2051 H=2051:00 +CPU= 14.740
  00:45:50 STEP 2052 H=2052:00 +CPU= 27.958
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002052
  00:46:06 STEP 2053 H=2053:00 +CPU= 16.433
  00:46:24 STEP 2054 H=2054:00 +CPU= 17.795
  00:46:50 STEP 2055 H=2055:00 +CPU= 25.670
  00:47:06 STEP 2056 H=2056:00 +CPU= 16.544
  00:47:22 STEP 2057 H=2057:00 +CPU= 16.326
  00:47:49 STEP 2058 H=2058:00 +CPU= 26.083
  00:48:04 STEP 2059 H=2059:00 +CPU= 15.881
  00:48:22 STEP 2060 H=2060:00 +CPU= 17.352
  00:48:49 STEP 2061 H=2061:00 +CPU= 27.087
  00:49:06 STEP 2062 H=2062:00 +CPU= 17.198
  00:49:23 STEP 2063 H=2063:00 +CPU= 17.157
  00:49:56 STEP 2064 H=2064:00 +CPU= 32.836
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_85.zip
Uploading the intermediate file: upload_file_85.zip
  00:50:14 STEP 2065 H=2065:00 +CPU= 17.644
  00:50:31 STEP 2066 H=2066:00 +CPU= 16.806
  00:50:57 STEP 2067 H=2067:00 +CPU= 25.999
  00:51:12 STEP 2068 H=2068:00 +CPU= 15.585
  00:51:28 STEP 2069 H=2069:00 +CPU= 15.526
  00:51:55 STEP 2070 H=2070:00 +CPU= 26.918
  00:52:10 STEP 2071 H=2071:00 +CPU= 15.259
  00:52:26 STEP 2072 H=2072:00 +CPU= 16.639
  00:52:53 STEP 2073 H=2073:00 +CPU= 26.247
  00:53:10 STEP 2074 H=2074:00 +CPU= 16.884
  00:53:26 STEP 2075 H=2075:00 +CPU= 15.997
  00:53:55 STEP 2076 H=2076:00 +CPU= 29.167
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002076
  00:54:11 STEP 2077 H=2077:00 +CPU= 16.601
  00:54:29 STEP 2078 H=2078:00 +CPU= 17.333
  00:54:55 STEP 2079 H=2079:00 +CPU= 26.133
  00:55:12 STEP 2080 H=2080:00 +CPU= 16.814
  00:55:27 STEP 2081 H=2081:00 +CPU= 14.890
  00:55:51 STEP 2082 H=2082:00 +CPU= 24.700
  00:56:08 STEP 2083 H=2083:00 +CPU= 16.441
  00:56:24 STEP 2084 H=2084:00 +CPU= 16.763
  00:56:52 STEP 2085 H=2085:00 +CPU= 27.436
  00:57:10 STEP 2086 H=2086:00 +CPU= 17.817
  00:57:26 STEP 2087 H=2087:00 +CPU= 16.606
  00:57:58 STEP 2088 H=2088:00 +CPU= 31.222
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_86.zip
Uploading the intermediate file: upload_file_86.zip
  00:58:15 STEP 2089 H=2089:00 +CPU= 17.150
  00:58:32 STEP 2090 H=2090:00 +CPU= 17.469
  00:58:59 STEP 2091 H=2091:00 +CPU= 27.142
  00:59:16 STEP 2092 H=2092:00 +CPU= 16.489
  00:59:31 STEP 2093 H=2093:00 +CPU= 15.635
  00:59:58 STEP 2094 H=2094:00 +CPU= 26.655
  01:00:14 STEP 2095 H=2095:00 +CPU= 16.015
  01:00:30 STEP 2096 H=2096:00 +CPU= 16.237
  01:00:56 STEP 2097 H=2097:00 +CPU= 25.424
  01:01:13 STEP 2098 H=2098:00 +CPU= 16.703
  01:01:28 STEP 2099 H=2099:00 +CPU= 15.929
  01:01:58 STEP 2100 H=2100:00 +CPU= 29.200
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002100
  01:02:15 STEP 2101 H=2101:00 +CPU= 17.367
  01:02:33 STEP 2102 H=2102:00 +CPU= 18.357
  01:03:00 STEP 2103 H=2103:00 +CPU= 26.742
  01:03:17 STEP 2104 H=2104:00 +CPU= 17.140
  01:03:34 STEP 2105 H=2105:00 +CPU= 16.363
  01:04:00 STEP 2106 H=2106:00 +CPU= 26.764
  01:04:18 STEP 2107 H=2107:00 +CPU= 17.153
  01:04:34 STEP 2108 H=2108:00 +CPU= 16.563
  01:05:01 STEP 2109 H=2109:00 +CPU= 27.203
  01:05:20 STEP 2110 H=2110:00 +CPU= 18.411
  01:05:37 STEP 2111 H=2111:00 +CPU= 16.938
  01:06:09 STEP 2112 H=2112:00 +CPU= 31.908
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_87.zip
Uploading the intermediate file: upload_file_87.zip
  01:06:25 STEP 2113 H=2113:00 +CPU= 16.624
  01:06:42 STEP 2114 H=2114:00 +CPU= 16.566
  01:07:08 STEP 2115 H=2115:00 +CPU= 26.341
  01:07:25 STEP 2116 H=2116:00 +CPU= 16.501
  01:07:41 STEP 2117 H=2117:00 +CPU= 16.245
  01:08:08 STEP 2118 H=2118:00 +CPU= 26.870
  01:08:24 STEP 2119 H=2119:00 +CPU= 16.063
  01:08:40 STEP 2120 H=2120:00 +CPU= 16.626
  01:09:08 STEP 2121 H=2121:00 +CPU= 27.436
  01:09:25 STEP 2122 H=2122:00 +CPU= 16.874
  01:09:41 STEP 2123 H=2123:00 +CPU= 16.216
  01:10:11 STEP 2124 H=2124:00 +CPU= 29.492
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002124
  01:10:28 STEP 2125 H=2125:00 +CPU= 17.178
  01:10:44 STEP 2126 H=2126:00 +CPU= 16.469
  01:11:12 STEP 2127 H=2127:00 +CPU= 27.829
  01:11:33 STEP 2128 H=2128:00 +CPU= 21.378
  01:11:52 STEP 2129 H=2129:00 +CPU= 18.204
  01:12:25 STEP 2130 H=2130:00 +CPU= 33.237
  01:12:47 STEP 2131 H=2131:00 +CPU= 22.192
  01:13:10 STEP 2132 H=2132:00 +CPU= 22.278
  01:13:39 STEP 2133 H=2133:00 +CPU= 28.952
  01:14:00 STEP 2134 H=2134:00 +CPU= 20.968
  01:14:22 STEP 2135 H=2135:00 +CPU= 22.635
  01:15:06 STEP 2136 H=2136:00 +CPU= 43.323
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_88.zip
Uploading the intermediate file: upload_file_88.zip
  01:15:29 STEP 2137 H=2137:00 +CPU= 22.910
  01:15:53 STEP 2138 H=2138:00 +CPU= 23.752
  01:16:29 STEP 2139 H=2139:00 +CPU= 36.051
  01:16:52 STEP 2140 H=2140:00 +CPU= 22.970
  01:17:14 STEP 2141 H=2141:00 +CPU= 21.919
  01:17:50 STEP 2142 H=2142:00 +CPU= 36.064
  01:18:12 STEP 2143 H=2143:00 +CPU= 21.697
  01:18:34 STEP 2144 H=2144:00 +CPU= 22.202
  01:19:09 STEP 2145 H=2145:00 +CPU= 35.215
  01:19:32 STEP 2146 H=2146:00 +CPU= 22.370
  01:19:54 STEP 2147 H=2147:00 +CPU= 22.097
  01:20:33 STEP 2148 H=2148:00 +CPU= 39.480
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002148
  01:20:56 STEP 2149 H=2149:00 +CPU= 22.479
  01:21:18 STEP 2150 H=2150:00 +CPU= 21.904
  01:21:51 STEP 2151 H=2151:00 +CPU= 33.751
  01:22:13 STEP 2152 H=2152:00 +CPU= 22.028
  01:22:35 STEP 2153 H=2153:00 +CPU= 21.741
  01:23:12 STEP 2154 H=2154:00 +CPU= 36.470
  01:23:33 STEP 2155 H=2155:00 +CPU= 21.778
  01:23:55 STEP 2156 H=2156:00 +CPU= 22.153
  01:24:29 STEP 2157 H=2157:00 +CPU= 33.865
  01:24:52 STEP 2158 H=2158:00 +CPU= 22.471
  01:25:14 STEP 2159 H=2159:00 +CPU= 21.932
  01:25:55 STEP 2160 H=2160:00 +CPU= 40.776
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_89.zip
Uploading the intermediate file: upload_file_89.zip
  01:26:15 STEP 2161 H=2161:00 +CPU= 20.747
  01:26:39 STEP 2162 H=2162:00 +CPU= 23.668
  01:27:17 STEP 2163 H=2163:00 +CPU= 37.668
  01:27:40 STEP 2164 H=2164:00 +CPU= 23.152
  01:28:03 STEP 2165 H=2165:00 +CPU= 23.011
  01:28:38 STEP 2166 H=2166:00 +CPU= 35.306
  01:29:00 STEP 2167 H=2167:00 +CPU= 21.889
  01:29:22 STEP 2168 H=2168:00 +CPU= 21.787
  01:29:57 STEP 2169 H=2169:00 +CPU= 35.229
  01:30:20 STEP 2170 H=2170:00 +CPU= 22.951
  01:30:42 STEP 2171 H=2171:00 +CPU= 21.407
  01:31:21 STEP 2172 H=2172:00 +CPU= 39.426
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002172
  01:31:43 STEP 2173 H=2173:00 +CPU= 21.685
  01:32:05 STEP 2174 H=2174:00 +CPU= 21.894
  01:32:42 STEP 2175 H=2175:00 +CPU= 36.971
  01:33:05 STEP 2176 H=2176:00 +CPU= 23.328
  01:33:24 STEP 2177 H=2177:00 +CPU= 19.475
  01:33:57 STEP 2178 H=2178:00 +CPU= 32.257
  01:34:19 STEP 2179 H=2179:00 +CPU= 22.251
  01:34:42 STEP 2180 H=2180:00 +CPU= 23.105
  01:35:17 STEP 2181 H=2181:00 +CPU= 35.052
  01:35:37 STEP 2182 H=2182:00 +CPU= 19.827
  01:35:57 STEP 2183 H=2183:00 +CPU= 19.802
  01:36:32 STEP 2184 H=2184:00 +CPU= 35.610
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_90.zip
Uploading the intermediate file: upload_file_90.zip
  01:36:50 STEP 2185 H=2185:00 +CPU= 17.933
Uploading trickle at timestep: 7862400
  01:37:09 STEP 2186 H=2186:00 +CPU= 18.718
  01:37:41 STEP 2187 H=2187:00 +CPU= 31.567
  01:37:59 STEP 2188 H=2188:00 +CPU= 18.274
  01:38:21 STEP 2189 H=2189:00 +CPU= 22.093
  01:38:58 STEP 2190 H=2190:00 +CPU= 37.394
  01:39:21 STEP 2191 H=2191:00 +CPU= 22.825
  01:39:44 STEP 2192 H=2192:00 +CPU= 22.427
  01:40:14 STEP 2193 H=2193:00 +CPU= 30.432
  01:40:33 STEP 2194 H=2194:00 +CPU= 18.663
  01:40:50 STEP 2195 H=2195:00 +CPU= 17.036
  01:41:24 STEP 2196 H=2196:00 +CPU= 34.076
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002196
  01:41:41 STEP 2197 H=2197:00 +CPU= 17.540
  01:42:02 STEP 2198 H=2198:00 +CPU= 20.624
  01:42:33 STEP 2199 H=2199:00 +CPU= 30.513
  01:42:52 STEP 2200 H=2200:00 +CPU= 19.720
  01:43:12 STEP 2201 H=2201:00 +CPU= 19.981
  01:43:45 STEP 2202 H=2202:00 +CPU= 32.546
  01:44:08 STEP 2203 H=2203:00 +CPU= 23.302
  01:44:30 STEP 2204 H=2204:00 +CPU= 21.787
  01:44:57 STEP 2205 H=2205:00 +CPU= 27.064
  01:45:19 STEP 2206 H=2206:00 +CPU= 21.676
  01:45:41 STEP 2207 H=2207:00 +CPU= 21.910
  01:46:13 STEP 2208 H=2208:00 +CPU= 32.867
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_91.zip
Uploading the intermediate file: upload_file_91.zip
  01:46:30 STEP 2209 H=2209:00 +CPU= 16.825
  01:46:53 STEP 2210 H=2210:00 +CPU= 22.325
  01:47:19 STEP 2211 H=2211:00 +CPU= 26.341
  01:47:36 STEP 2212 H=2212:00 +CPU= 17.456
  01:47:52 STEP 2213 H=2213:00 +CPU= 15.936
  01:48:17 STEP 2214 H=2214:00 +CPU= 24.848
  01:48:34 STEP 2215 H=2215:00 +CPU= 16.462
  01:48:50 STEP 2216 H=2216:00 +CPU= 16.757
  01:49:19 STEP 2217 H=2217:00 +CPU= 28.433
  01:49:36 STEP 2218 H=2218:00 +CPU= 17.349
  01:49:53 STEP 2219 H=2219:00 +CPU= 16.898
  01:50:23 STEP 2220 H=2220:00 +CPU= 29.910
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002220
  01:50:39 STEP 2221 H=2221:00 +CPU= 16.312
  01:50:57 STEP 2222 H=2222:00 +CPU= 17.430
  01:51:23 STEP 2223 H=2223:00 +CPU= 26.523
  01:51:41 STEP 2224 H=2224:00 +CPU= 17.279
  01:51:57 STEP 2225 H=2225:00 +CPU= 16.504
  01:52:25 STEP 2226 H=2226:00 +CPU= 28.050
  01:52:42 STEP 2227 H=2227:00 +CPU= 16.475
  01:52:58 STEP 2228 H=2228:00 +CPU= 16.782
  01:53:24 STEP 2229 H=2229:00 +CPU= 25.900
  01:53:42 STEP 2230 H=2230:00 +CPU= 17.257
  01:54:00 STEP 2231 H=2231:00 +CPU= 18.288
  01:54:31 STEP 2232 H=2232:00 +CPU= 30.925
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_92.zip
Uploading the intermediate file: upload_file_92.zip
  01:54:47 STEP 2233 H=2233:00 +CPU= 16.171
  01:55:04 STEP 2234 H=2234:00 +CPU= 16.988
  01:55:30 STEP 2235 H=2235:00 +CPU= 26.183
  01:55:46 STEP 2236 H=2236:00 +CPU= 15.769
  01:56:02 STEP 2237 H=2237:00 +CPU= 16.291
  01:56:30 STEP 2238 H=2238:00 +CPU= 27.372
  01:56:47 STEP 2239 H=2239:00 +CPU= 16.859
  01:57:04 STEP 2240 H=2240:00 +CPU= 17.522
  01:57:31 STEP 2241 H=2241:00 +CPU= 26.939
  01:57:48 STEP 2242 H=2242:00 +CPU= 17.275
  01:58:06 STEP 2243 H=2243:00 +CPU= 17.230
  01:58:36 STEP 2244 H=2244:00 +CPU= 30.642
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002244
  01:58:53 STEP 2245 H=2245:00 +CPU= 17.287
  01:59:12 STEP 2246 H=2246:00 +CPU= 18.194
  01:59:39 STEP 2247 H=2247:00 +CPU= 27.404
  01:59:57 STEP 2248 H=2248:00 +CPU= 17.942
  02:00:14 STEP 2249 H=2249:00 +CPU= 16.575
  02:00:38 STEP 2250 H=2250:00 +CPU= 24.802
  02:00:56 STEP 2251 H=2251:00 +CPU= 17.348
  02:01:14 STEP 2252 H=2252:00 +CPU= 18.089
  02:01:39 STEP 2253 H=2253:00 +CPU= 25.314
  02:01:55 STEP 2254 H=2254:00 +CPU= 15.970
  02:02:11 STEP 2255 H=2255:00 +CPU= 16.098
  02:02:45 STEP 2256 H=2256:00 +CPU= 34.142
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_93.zip
Uploading the intermediate file: upload_file_93.zip
  02:03:06 STEP 2257 H=2257:00 +CPU= 20.900
  02:03:26 STEP 2258 H=2258:00 +CPU= 19.339
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0517
(argv4) batchid: 965
(argv5) wuid: 12182161
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9ef8f817669026e8eefbbffd113c2b84 to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0517_1996050100_123_965_12182161.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0517_1996050100_123_965_12182161.zip
ic_ancil_file: ic_ancil_12182161
ifsdata_file: ifsdata_12182161
climate_data_file: clim_data_12182161
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4d9886e6d0c2b1ba775a9e7d1e38b895 to: /var/lib/boinc-client/slots/23/ic_ancil_12182161.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12182161.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f6ecc9bccbb0135785351f3e24947933 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12182161.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12182161.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12182161.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12182161.zip
Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12182161.xml
Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12182161.xml
-- Model is restarting --
Adjusting last_iter, 2259, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12182161.xml
last_cpu_time: 48575
upload_file_number: 94
last_iter: 2255
last_upload: 8121600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232
The child process has been launched with process id: 588084
Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588084:588084] [20230106:024102:1672972862:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:41:53 STEP 2256 H=2256:00 +CPU= 48.156
  02:42:22 STEP 2257 H=2257:00 +CPU= 29.373
  02:42:54 STEP 2258 H=2258:00 +CPU= 31.929
  02:43:41 STEP 2259 H=2259:00 +CPU= 46.973
  02:44:12 STEP 2260 H=2260:00 +CPU= 30.291
  02:44:42 STEP 2261 H=2261:00 +CPU= 30.169
  02:45:27 STEP 2262 H=2262:00 +CPU= 45.248
  02:46:08 STEP 2263 H=2263:00 +CPU= 41.476
  02:46:40 STEP 2264 H=2264:00 +CPU= 31.645
  02:47:28 STEP 2265 H=2265:00 +CPU= 47.602
  02:48:06 STEP 2266 H=2266:00 +CPU= 38.345
  02:48:33 STEP 2267 H=2267:00 +CPU= 26.967
  02:49:22 STEP 2268 H=2268:00 +CPU= 49.107
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002268
  02:49:51 STEP 2269 H=2269:00 +CPU= 28.622
  02:50:25 STEP 2270 H=2270:00 +CPU= 34.479
  02:51:12 STEP 2271 H=2271:00 +CPU= 46.553
  02:51:39 STEP 2272 H=2272:00 +CPU= 27.511
  02:52:07 STEP 2273 H=2273:00 +CPU= 27.509
  02:52:57 STEP 2274 H=2274:00 +CPU= 49.546
  02:53:33 STEP 2275 H=2275:00 +CPU= 36.380
  02:54:03 STEP 2276 H=2276:00 +CPU= 30.500
  02:54:45 STEP 2277 H=2277:00 +CPU= 41.882
  02:55:08 STEP 2278 H=2278:00 +CPU= 22.880
  02:55:34 STEP 2279 H=2279:00 +CPU= 25.390
  02:56:22 STEP 2280 H=2280:00 +CPU= 48.223
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_94.zip
Uploading the intermediate file: upload_file_94.zip
  02:56:46 STEP 2281 H=2281:00 +CPU= 24.520
  02:57:16 STEP 2282 H=2282:00 +CPU= 29.552
  02:57:54 STEP 2283 H=2283:00 +CPU= 37.804
  02:58:24 STEP 2284 H=2284:00 +CPU= 30.494
  02:58:43 STEP 2285 H=2285:00 +CPU= 18.894
  02:59:20 STEP 2286 H=2286:00 +CPU= 37.062
  02:59:46 STEP 2287 H=2287:00 +CPU= 25.952
  03:00:11 STEP 2288 H=2288:00 +CPU= 24.723
  03:00:44 STEP 2289 H=2289:00 +CPU= 32.716
  03:01:12 STEP 2290 H=2290:00 +CPU= 28.824
  03:01:43 STEP 2291 H=2291:00 +CPU= 30.344
  03:02:33 STEP 2292 H=2292:00 +CPU= 50.267
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002292
  03:03:12 STEP 2293 H=2293:00 +CPU= 38.877
  03:04:04 STEP 2294 H=2294:00 +CPU= 51.950
  03:04:56 STEP 2295 H=2295:00 +CPU= 51.937
  03:05:28 STEP 2296 H=2296:00 +CPU= 31.307
  03:05:55 STEP 2297 H=2297:00 +CPU= 27.400
  03:06:43 STEP 2298 H=2298:00 +CPU= 47.413
  03:07:11 STEP 2299 H=2299:00 +CPU= 28.751
  03:07:40 STEP 2300 H=2300:00 +CPU= 28.300
  03:08:21 STEP 2301 H=2301:00 +CPU= 41.729
  03:08:56 STEP 2302 H=2302:00 +CPU= 34.160
  03:09:26 STEP 2303 H=2303:00 +CPU= 29.975
  03:10:26 STEP 2304 H=2304:00 +CPU= 60.168
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_95.zip
Uploading the intermediate file: upload_file_95.zip
  03:10:51 STEP 2305 H=2305:00 +CPU= 24.906
  03:11:24 STEP 2306 H=2306:00 +CPU= 33.214
  03:12:09 STEP 2307 H=2307:00 +CPU= 44.425
  03:12:36 STEP 2308 H=2308:00 +CPU= 27.248
  03:13:05 STEP 2309 H=2309:00 +CPU= 28.704
  03:13:48 STEP 2310 H=2310:00 +CPU= 43.507
  03:14:16 STEP 2311 H=2311:00 +CPU= 27.623
  03:14:44 STEP 2312 H=2312:00 +CPU= 28.041
  03:15:24 STEP 2313 H=2313:00 +CPU= 39.707
  03:15:48 STEP 2314 H=2314:00 +CPU= 24.493
  03:16:12 STEP 2315 H=2315:00 +CPU= 24.092
  03:16:56 STEP 2316 H=2316:00 +CPU= 43.547
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002316
  03:17:21 STEP 2317 H=2317:00 +CPU= 24.520
  03:17:46 STEP 2318 H=2318:00 +CPU= 25.703
  03:18:27 STEP 2319 H=2319:00 +CPU= 40.388
  03:18:54 STEP 2320 H=2320:00 +CPU= 27.345
  03:19:20 STEP 2321 H=2321:00 +CPU= 26.039
  03:20:03 STEP 2322 H=2322:00 +CPU= 42.567
  03:20:29 STEP 2323 H=2323:00 +CPU= 25.829
  03:20:54 STEP 2324 H=2324:00 +CPU= 25.839
  03:21:41 STEP 2325 H=2325:00 +CPU= 46.279
  03:22:09 STEP 2326 H=2326:00 +CPU= 28.397
  03:22:35 STEP 2327 H=2327:00 +CPU= 25.556
  03:23:21 STEP 2328 H=2328:00 +CPU= 45.976
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_96.zip
Uploading the intermediate file: upload_file_96.zip
  03:23:47 STEP 2329 H=2329:00 +CPU= 26.445
  03:24:12 STEP 2330 H=2330:00 +CPU= 24.990
  03:24:50 STEP 2331 H=2331:00 +CPU= 37.707
  03:25:13 STEP 2332 H=2332:00 +CPU= 23.252
  03:25:36 STEP 2333 H=2333:00 +CPU= 22.301
  03:26:12 STEP 2334 H=2334:00 +CPU= 36.456
  03:26:34 STEP 2335 H=2335:00 +CPU= 21.659
  03:26:58 STEP 2336 H=2336:00 +CPU= 24.147
  03:27:35 STEP 2337 H=2337:00 +CPU= 36.582
  03:27:58 STEP 2338 H=2338:00 +CPU= 23.431
  03:28:21 STEP 2339 H=2339:00 +CPU= 22.938
  03:29:02 STEP 2340 H=2340:00 +CPU= 40.617
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002340
  03:29:25 STEP 2341 H=2341:00 +CPU= 23.485
  03:29:48 STEP 2342 H=2342:00 +CPU= 22.810
  03:30:26 STEP 2343 H=2343:00 +CPU= 37.339
  03:30:51 STEP 2344 H=2344:00 +CPU= 25.414
  03:31:17 STEP 2345 H=2345:00 +CPU= 26.067
  03:31:59 STEP 2346 H=2346:00 +CPU= 41.907
  03:32:22 STEP 2347 H=2347:00 +CPU= 23.105
  03:32:46 STEP 2348 H=2348:00 +CPU= 22.970
  03:33:23 STEP 2349 H=2349:00 +CPU= 37.019
  03:33:47 STEP 2350 H=2350:00 +CPU= 24.045
  03:34:11 STEP 2351 H=2351:00 +CPU= 24.183
  03:34:55 STEP 2352 H=2352:00 +CPU= 43.505
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_97.zip
Uploading the intermediate file: upload_file_97.zip
  03:35:19 STEP 2353 H=2353:00 +CPU= 24.153
  03:35:43 STEP 2354 H=2354:00 +CPU= 23.899
  03:36:21 STEP 2355 H=2355:00 +CPU= 37.471
  03:36:45 STEP 2356 H=2356:00 +CPU= 23.783
  03:37:08 STEP 2357 H=2357:00 +CPU= 22.906
  03:37:45 STEP 2358 H=2358:00 +CPU= 37.248
  03:38:09 STEP 2359 H=2359:00 +CPU= 23.791
  03:38:32 STEP 2360 H=2360:00 +CPU= 23.617
  03:39:07 STEP 2361 H=2361:00 +CPU= 34.552
  03:39:33 STEP 2362 H=2362:00 +CPU= 25.730
  03:39:56 STEP 2363 H=2363:00 +CPU= 22.758
  03:40:37 STEP 2364 H=2364:00 +CPU= 40.964
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002364
  03:40:58 STEP 2365 H=2365:00 +CPU= 21.657
  03:41:22 STEP 2366 H=2366:00 +CPU= 23.810
  03:41:58 STEP 2367 H=2367:00 +CPU= 35.304
  03:42:19 STEP 2368 H=2368:00 +CPU= 21.970
  03:42:41 STEP 2369 H=2369:00 +CPU= 21.834
  03:43:19 STEP 2370 H=2370:00 +CPU= 38.108
  03:43:42 STEP 2371 H=2371:00 +CPU= 22.464
  03:44:05 STEP 2372 H=2372:00 +CPU= 22.851
  03:44:39 STEP 2373 H=2373:00 +CPU= 34.575
  03:45:01 STEP 2374 H=2374:00 +CPU= 21.457
  03:45:21 STEP 2375 H=2375:00 +CPU= 20.290
  03:46:04 STEP 2376 H=2376:00 +CPU= 42.762
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_98.zip
Uploading the intermediate file: upload_file_98.zip
  03:46:27 STEP 2377 H=2377:00 +CPU= 23.169
  03:46:52 STEP 2378 H=2378:00 +CPU= 25.339
  03:47:27 STEP 2379 H=2379:00 +CPU= 34.891
  03:47:51 STEP 2380 H=2380:00 +CPU= 23.435
  03:48:13 STEP 2381 H=2381:00 +CPU= 22.243
  03:48:45 STEP 2382 H=2382:00 +CPU= 31.495
  03:49:06 STEP 2383 H=2383:00 +CPU= 21.708
  03:49:25 STEP 2384 H=2384:00 +CPU= 18.351
  03:49:59 STEP 2385 H=2385:00 +CPU= 34.630
  03:50:21 STEP 2386 H=2386:00 +CPU= 21.327
  03:50:43 STEP 2387 H=2387:00 +CPU= 22.580
  03:51:20 STEP 2388 H=2388:00 +CPU= 36.593
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002388
  03:51:42 STEP 2389 H=2389:00 +CPU= 22.265
  03:52:05 STEP 2390 H=2390:00 +CPU= 22.948
  03:52:39 STEP 2391 H=2391:00 +CPU= 33.969
  03:53:03 STEP 2392 H=2392:00 +CPU= 23.866
  03:53:22 STEP 2393 H=2393:00 +CPU= 19.040
  03:53:55 STEP 2394 H=2394:00 +CPU= 33.411
  03:54:13 STEP 2395 H=2395:00 +CPU= 17.296
  03:54:29 STEP 2396 H=2396:00 +CPU= 16.290
  03:54:59 STEP 2397 H=2397:00 +CPU= 29.779
  03:55:18 STEP 2398 H=2398:00 +CPU= 19.003
  03:55:43 STEP 2399 H=2399:00 +CPU= 24.866
  03:56:23 STEP 2400 H=2400:00 +CPU= 40.380
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_99.zip
Uploading the intermediate file: upload_file_99.zip
  03:56:43 STEP 2401 H=2401:00 +CPU= 20.278
  03:57:03 STEP 2402 H=2402:00 +CPU= 19.767
  03:57:30 STEP 2403 H=2403:00 +CPU= 26.936
  03:57:47 STEP 2404 H=2404:00 +CPU= 16.768
  03:58:06 STEP 2405 H=2405:00 +CPU= 19.594
  03:58:44 STEP 2406 H=2406:00 +CPU= 37.310
  03:59:03 STEP 2407 H=2407:00 +CPU= 19.072
  03:59:23 STEP 2408 H=2408:00 +CPU= 20.436
  03:59:50 STEP 2409 H=2409:00 +CPU= 27.215
  04:00:10 STEP 2410 H=2410:00 +CPU= 19.302
  04:00:27 STEP 2411 H=2411:00 +CPU= 17.245
  04:01:01 STEP 2412 H=2412:00 +CPU= 33.983
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002412
  04:01:17 STEP 2413 H=2413:00 +CPU= 16.289
  04:01:35 STEP 2414 H=2414:00 +CPU= 17.524
  04:02:05 STEP 2415 H=2415:00 +CPU= 30.270
  04:02:22 STEP 2416 H=2416:00 +CPU= 16.966
  04:02:38 STEP 2417 H=2417:00 +CPU= 16.405
  04:03:07 STEP 2418 H=2418:00 +CPU= 28.782
  04:03:25 STEP 2419 H=2419:00 +CPU= 17.918
  04:03:44 STEP 2420 H=2420:00 +CPU= 18.536
  04:04:13 STEP 2421 H=2421:00 +CPU= 29.253
  04:04:31 STEP 2422 H=2422:00 +CPU= 18.182
  04:04:49 STEP 2423 H=2423:00 +CPU= 17.618
  04:05:22 STEP 2424 H=2424:00 +CPU= 33.223
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_100.zip
Uploading the intermediate file: upload_file_100.zip
  04:05:44 STEP 2425 H=2425:00 +CPU= 21.751
  04:06:07 STEP 2426 H=2426:00 +CPU= 22.997
  04:06:42 STEP 2427 H=2427:00 +CPU= 35.126
  04:07:04 STEP 2428 H=2428:00 +CPU= 22.575
  04:07:27 STEP 2429 H=2429:00 +CPU= 22.925
  04:08:06 STEP 2430 H=2430:00 +CPU= 38.681
  04:08:28 STEP 2431 H=2431:00 +CPU= 21.773
  04:08:49 STEP 2432 H=2432:00 +CPU= 20.692
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0517
(argv4) batchid: 965
(argv5) wuid: 12182161
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9ef8f817669026e8eefbbffd113c2b84 to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0517_1996050100_123_965_12182161.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0517_1996050100_123_965_12182161.zip
ic_ancil_file: ic_ancil_12182161
ifsdata_file: ifsdata_12182161
climate_data_file: clim_data_12182161
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4d9886e6d0c2b1ba775a9e7d1e38b895 to: /var/lib/boinc-client/slots/23/ic_ancil_12182161.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12182161.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f6ecc9bccbb0135785351f3e24947933 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12182161.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12182161.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12182161.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12182161.zip
Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12182161.xml
Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12182161.xml
-- Model is restarting --
Adjusting last_iter, 2433, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12182161.xml
last_cpu_time: 53848
upload_file_number: 101
last_iter: 2423
last_upload: 8726400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232
The child process has been launched with process id: 590768
Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590768:590768] [20230106:042005:1672978805:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:20:53 STEP 2424 H=2424:00 +CPU= 44.643
  04:21:18 STEP 2425 H=2425:00 +CPU= 25.414
  04:21:48 STEP 2426 H=2426:00 +CPU= 29.885
  04:22:37 STEP 2427 H=2427:00 +CPU= 48.782
  04:23:06 STEP 2428 H=2428:00 +CPU= 28.791
  04:23:32 STEP 2429 H=2429:00 +CPU= 26.828
  04:24:15 STEP 2430 H=2430:00 +CPU= 42.574
  04:24:41 STEP 2431 H=2431:00 +CPU= 26.229
  04:25:08 STEP 2432 H=2432:00 +CPU= 26.315
  04:25:51 STEP 2433 H=2433:00 +CPU= 43.438
  04:26:17 STEP 2434 H=2434:00 +CPU= 25.489
  04:26:45 STEP 2435 H=2435:00 +CPU= 28.359
  04:27:39 STEP 2436 H=2436:00 +CPU= 53.475
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002436
  04:28:09 STEP 2437 H=2437:00 +CPU= 30.355
  04:28:38 STEP 2438 H=2438:00 +CPU= 29.320
  04:29:20 STEP 2439 H=2439:00 +CPU= 41.681
  04:29:47 STEP 2440 H=2440:00 +CPU= 27.029
  04:30:15 STEP 2441 H=2441:00 +CPU= 28.080
  04:31:07 STEP 2442 H=2442:00 +CPU= 51.421
  04:31:41 STEP 2443 H=2443:00 +CPU= 34.831
  04:32:07 STEP 2444 H=2444:00 +CPU= 25.945
  04:32:51 STEP 2445 H=2445:00 +CPU= 43.302
  04:33:16 STEP 2446 H=2446:00 +CPU= 25.503
  04:33:42 STEP 2447 H=2447:00 +CPU= 26.159
  04:34:34 STEP 2448 H=2448:00 +CPU= 51.861
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182161/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0517_1996050100_123_965_12182161_1_r1750330232_101.zip
Uploading the intermediate file: upload_file_101.zip
  04:35:06 STEP 2449 H=2449:00 +CPU= 31.595
  04:35:37 STEP 2450 H=2450:00 +CPU= 30.616
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:46 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 7,862,400 46,960 0.0060
09 May 2023 16:09:43 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 6,998,400 41,847 0.0060
09 May 2023 16:09:39 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 6,134,400 36,704 0.0060
09 May 2023 16:09:39 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 6,134,400 36,704 0.0060
09 May 2023 16:09:11 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 3,456,000 17,255 0.0050
09 May 2023 16:09:11 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 3,456,000 17,255 0.0050
09 May 2023 16:09:07 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 2,592,000 13,243 0.0051
09 May 2023 16:09:04 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 1,728,000 8,410 0.0049
09 May 2023 16:09:04 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 1,728,000 8,410 0.0049
09 May 2023 16:09:00 1532338 22295768 oifs_43r3_ps_0517_1996050100_123_965_12182161_1 864,000 4,099 0.0047


©2024 cpdn.org