climateprediction.net home page
Task 22270450

Task 22270450

Name oifs_43r3_ps_0518_1999050100_123_968_12185162_0
Workunit 12185162
Created 22 Dec 2022, 19:01:42 UTC
Sent 19 Jan 2023, 7:53:20 UTC
Report deadline 20 Mar 2023, 7:53:20 UTC
Received 20 Jan 2023, 8:48:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 5 (0x00000005) Unknown error code
Computer ID 1534452
Run time 21 hours 12 min 43 sec
CPU time 20 hours 26 min 48 sec
Validate state Invalid
Credit 12,337.92
Device peak FLOPS 4.58 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,402.05 MB
Peak swap size 4,978.07 MB
Peak disk usage 5,251.72 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 5 (0x5, -251)</message>
<stderr_txt>
:00 +CPU= 24.376
  04:49:20 STEP 2034 H=2034:00 +CPU= 35.317
  04:49:41 STEP 2035 H=2035:00 +CPU= 19.916
  04:50:01 STEP 2036 H=2036:00 +CPU= 19.226
  04:50:33 STEP 2037 H=2037:00 +CPU= 31.769
  04:50:58 STEP 2038 H=2038:00 +CPU= 24.740
  04:51:20 STEP 2039 H=2039:00 +CPU= 21.396
  04:52:01 STEP 2040 H=2040:00 +CPU= 41.363
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_84.zip
Uploading the intermediate file: upload_file_84.zip
  04:52:25 STEP 2041 H=2041:00 +CPU= 23.977
  04:52:47 STEP 2042 H=2042:00 +CPU= 21.639
  04:53:19 STEP 2043 H=2043:00 +CPU= 30.321
  04:53:43 STEP 2044 H=2044:00 +CPU= 23.946
  04:54:08 STEP 2045 H=2045:00 +CPU= 24.161
  04:54:47 STEP 2046 H=2046:00 +CPU= 38.725
  04:55:11 STEP 2047 H=2047:00 +CPU= 23.805
  04:55:36 STEP 2048 H=2048:00 +CPU= 24.889
  04:56:14 STEP 2049 H=2049:00 +CPU= 37.875
  04:56:38 STEP 2050 H=2050:00 +CPU= 23.547
  04:57:02 STEP 2051 H=2051:00 +CPU= 23.826
  04:57:36 STEP 2052 H=2052:00 +CPU= 34.233
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002052
  04:57:56 STEP 2053 H=2053:00 +CPU= 19.623
  04:58:15 STEP 2054 H=2054:00 +CPU= 18.257
  04:58:53 STEP 2055 H=2055:00 +CPU= 37.640
  04:59:18 STEP 2056 H=2056:00 +CPU= 25.237
  04:59:42 STEP 2057 H=2057:00 +CPU= 24.007
  05:00:21 STEP 2058 H=2058:00 +CPU= 39.093
  05:00:46 STEP 2059 H=2059:00 +CPU= 24.579
  05:01:10 STEP 2060 H=2060:00 +CPU= 24.548
  05:01:48 STEP 2061 H=2061:00 +CPU= 37.692
  05:02:13 STEP 2062 H=2062:00 +CPU= 24.925
  05:02:38 STEP 2063 H=2063:00 +CPU= 24.762
  05:03:23 STEP 2064 H=2064:00 +CPU= 44.681
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_85.zip
Uploading the intermediate file: upload_file_85.zip
  05:03:45 STEP 2065 H=2065:00 +CPU= 22.123
  05:04:10 STEP 2066 H=2066:00 +CPU= 24.404
  05:04:42 STEP 2067 H=2067:00 +CPU= 31.854
  05:05:08 STEP 2068 H=2068:00 +CPU= 24.671
  05:05:32 STEP 2069 H=2069:00 +CPU= 23.957
  05:06:10 STEP 2070 H=2070:00 +CPU= 38.713
  05:06:34 STEP 2071 H=2071:00 +CPU= 24.054
  05:07:00 STEP 2072 H=2072:00 +CPU= 25.089
  05:07:38 STEP 2073 H=2073:00 +CPU= 38.102
  05:08:03 STEP 2074 H=2074:00 +CPU= 24.884
  05:08:27 STEP 2075 H=2075:00 +CPU= 24.133
  05:09:09 STEP 2076 H=2076:00 +CPU= 41.914
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002076
  05:09:34 STEP 2077 H=2077:00 +CPU= 24.176
  05:10:01 STEP 2078 H=2078:00 +CPU= 26.243
  05:10:40 STEP 2079 H=2079:00 +CPU= 38.987
  05:11:05 STEP 2080 H=2080:00 +CPU= 25.255
  05:11:27 STEP 2081 H=2081:00 +CPU= 21.838
  05:12:06 STEP 2082 H=2082:00 +CPU= 37.834
  05:12:30 STEP 2083 H=2083:00 +CPU= 24.105
  05:12:53 STEP 2084 H=2084:00 +CPU= 21.653
  05:13:31 STEP 2085 H=2085:00 +CPU= 38.085
  05:13:56 STEP 2086 H=2086:00 +CPU= 25.101
  05:14:21 STEP 2087 H=2087:00 +CPU= 25.284
  05:15:06 STEP 2088 H=2088:00 +CPU= 44.883
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_86.zip
Uploading the intermediate file: upload_file_86.zip
  05:15:31 STEP 2089 H=2089:00 +CPU= 24.446
  05:15:56 STEP 2090 H=2090:00 +CPU= 25.069
  05:16:31 STEP 2091 H=2091:00 +CPU= 34.870
  05:16:56 STEP 2092 H=2092:00 +CPU= 25.017
  05:17:20 STEP 2093 H=2093:00 +CPU= 24.185
  05:17:58 STEP 2094 H=2094:00 +CPU= 38.073
  05:18:23 STEP 2095 H=2095:00 +CPU= 24.589
  05:18:48 STEP 2096 H=2096:00 +CPU= 25.128
  05:19:26 STEP 2097 H=2097:00 +CPU= 37.281
  05:19:51 STEP 2098 H=2098:00 +CPU= 25.187
  05:20:16 STEP 2099 H=2099:00 +CPU= 24.574
  05:20:58 STEP 2100 H=2100:00 +CPU= 41.837
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002100
  05:21:20 STEP 2101 H=2101:00 +CPU= 21.494
  05:21:46 STEP 2102 H=2102:00 +CPU= 25.594
  05:22:24 STEP 2103 H=2103:00 +CPU= 37.887
  05:22:49 STEP 2104 H=2104:00 +CPU= 25.539
  05:23:15 STEP 2105 H=2105:00 +CPU= 25.320
  05:23:55 STEP 2106 H=2106:00 +CPU= 39.790
  05:24:20 STEP 2107 H=2107:00 +CPU= 24.441
  05:24:45 STEP 2108 H=2108:00 +CPU= 24.622
  05:25:19 STEP 2109 H=2109:00 +CPU= 33.963
  05:25:42 STEP 2110 H=2110:00 +CPU= 23.657
  05:26:08 STEP 2111 H=2111:00 +CPU= 25.237
  05:26:52 STEP 2112 H=2112:00 +CPU= 43.741
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_87.zip
Uploading the intermediate file: upload_file_87.zip
  05:27:16 STEP 2113 H=2113:00 +CPU= 24.454
  05:27:42 STEP 2114 H=2114:00 +CPU= 25.246
  05:28:20 STEP 2115 H=2115:00 +CPU= 37.544
  05:28:46 STEP 2116 H=2116:00 +CPU= 25.015
  05:29:03 STEP 2117 H=2117:00 +CPU= 17.472
  05:29:32 STEP 2118 H=2118:00 +CPU= 28.316
  05:29:51 STEP 2119 H=2119:00 +CPU= 17.805
  05:30:16 STEP 2120 H=2120:00 +CPU= 25.086
  05:30:55 STEP 2121 H=2121:00 +CPU= 38.581
  05:31:21 STEP 2122 H=2122:00 +CPU= 25.429
  05:31:44 STEP 2123 H=2123:00 +CPU= 22.982
  05:32:24 STEP 2124 H=2124:00 +CPU= 39.997
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002124
  05:32:49 STEP 2125 H=2125:00 +CPU= 24.437
  05:33:14 STEP 2126 H=2126:00 +CPU= 25.488
  05:33:53 STEP 2127 H=2127:00 +CPU= 37.769
  05:34:18 STEP 2128 H=2128:00 +CPU= 25.197
  05:34:43 STEP 2129 H=2129:00 +CPU= 24.865
  05:35:23 STEP 2130 H=2130:00 +CPU= 39.710
  05:35:49 STEP 2131 H=2131:00 +CPU= 25.094
  05:36:15 STEP 2132 H=2132:00 +CPU= 26.034
  05:36:53 STEP 2133 H=2133:00 +CPU= 38.392
  05:37:19 STEP 2134 H=2134:00 +CPU= 24.873
  05:37:44 STEP 2135 H=2135:00 +CPU= 25.145
  05:38:27 STEP 2136 H=2136:00 +CPU= 42.969
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_88.zip
Uploading the intermediate file: upload_file_88.zip
  05:38:51 STEP 2137 H=2137:00 +CPU= 23.931
  05:39:15 STEP 2138 H=2138:00 +CPU= 23.198
  05:39:54 STEP 2139 H=2139:00 +CPU= 38.790
  05:40:20 STEP 2140 H=2140:00 +CPU= 25.290
  05:40:44 STEP 2141 H=2141:00 +CPU= 24.616
  05:41:23 STEP 2142 H=2142:00 +CPU= 38.546
  05:41:48 STEP 2143 H=2143:00 +CPU= 24.712
  05:42:14 STEP 2144 H=2144:00 +CPU= 25.597
  05:42:52 STEP 2145 H=2145:00 +CPU= 37.571
  05:43:17 STEP 2146 H=2146:00 +CPU= 25.239
  05:43:36 STEP 2147 H=2147:00 +CPU= 18.981
  05:44:14 STEP 2148 H=2148:00 +CPU= 37.206
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002148
  05:44:39 STEP 2149 H=2149:00 +CPU= 25.282
  05:45:05 STEP 2150 H=2150:00 +CPU= 25.208
  05:45:38 STEP 2151 H=2151:00 +CPU= 33.734
  05:45:58 STEP 2152 H=2152:00 +CPU= 19.707
  05:46:23 STEP 2153 H=2153:00 +CPU= 24.218
  05:47:02 STEP 2154 H=2154:00 +CPU= 38.312
  05:47:26 STEP 2155 H=2155:00 +CPU= 24.365
  05:47:49 STEP 2156 H=2156:00 +CPU= 22.082
  05:48:24 STEP 2157 H=2157:00 +CPU= 35.105
  05:48:50 STEP 2158 H=2158:00 +CPU= 25.379
  05:49:09 STEP 2159 H=2159:00 +CPU= 18.861
  05:49:54 STEP 2160 H=2160:00 +CPU= 43.870
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_89.zip
Uploading the intermediate file: upload_file_89.zip
  05:50:19 STEP 2161 H=2161:00 +CPU= 24.527
  05:50:44 STEP 2162 H=2162:00 +CPU= 25.036
  05:51:16 STEP 2163 H=2163:00 +CPU= 32.439
  05:51:41 STEP 2164 H=2164:00 +CPU= 24.279
  05:52:05 STEP 2165 H=2165:00 +CPU= 23.559
  05:52:42 STEP 2166 H=2166:00 +CPU= 37.259
  05:53:07 STEP 2167 H=2167:00 +CPU= 24.485
  05:53:32 STEP 2168 H=2168:00 +CPU= 25.467
  05:54:07 STEP 2169 H=2169:00 +CPU= 34.133
  05:54:29 STEP 2170 H=2170:00 +CPU= 22.161
  05:54:52 STEP 2171 H=2171:00 +CPU= 23.310
  05:55:34 STEP 2172 H=2172:00 +CPU= 41.761
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002172
  05:55:59 STEP 2173 H=2173:00 +CPU= 24.609
  05:56:24 STEP 2174 H=2174:00 +CPU= 25.147
  05:57:03 STEP 2175 H=2175:00 +CPU= 38.013
  05:57:29 STEP 2176 H=2176:00 +CPU= 25.964
  05:57:52 STEP 2177 H=2177:00 +CPU= 23.470
  05:58:28 STEP 2178 H=2178:00 +CPU= 34.806
  05:58:53 STEP 2179 H=2179:00 +CPU= 25.213
  05:59:18 STEP 2180 H=2180:00 +CPU= 24.804
  05:59:56 STEP 2181 H=2181:00 +CPU= 37.901
  06:00:22 STEP 2182 H=2182:00 +CPU= 25.102
  06:00:46 STEP 2183 H=2183:00 +CPU= 24.375
  06:01:32 STEP 2184 H=2184:00 +CPU= 45.212
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_90.zip
Uploading the intermediate file: upload_file_90.zip
  06:01:53 STEP 2185 H=2185:00 +CPU= 21.108
  06:02:19 STEP 2186 H=2186:00 +CPU= 25.609
  06:02:57 STEP 2187 H=2187:00 +CPU= 38.339
  06:03:21 STEP 2188 H=2188:00 +CPU= 24.424
  06:03:47 STEP 2189 H=2189:00 +CPU= 25.021
  06:04:26 STEP 2190 H=2190:00 +CPU= 39.263
  06:04:51 STEP 2191 H=2191:00 +CPU= 24.422
  06:05:15 STEP 2192 H=2192:00 +CPU= 24.377
  06:05:53 STEP 2193 H=2193:00 +CPU= 37.623
  06:06:17 STEP 2194 H=2194:00 +CPU= 23.892
  06:06:42 STEP 2195 H=2195:00 +CPU= 24.265
  06:07:21 STEP 2196 H=2196:00 +CPU= 37.679
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002196
  06:07:45 STEP 2197 H=2197:00 +CPU= 23.572
  06:08:06 STEP 2198 H=2198:00 +CPU= 20.317
  06:08:42 STEP 2199 H=2199:00 +CPU= 36.628
  06:09:07 STEP 2200 H=2200:00 +CPU= 24.572
  06:09:29 STEP 2201 H=2201:00 +CPU= 21.862
  06:10:06 STEP 2202 H=2202:00 +CPU= 37.543
  06:10:28 STEP 2203 H=2203:00 +CPU= 21.697
  06:10:51 STEP 2204 H=2204:00 +CPU= 22.034
  06:11:27 STEP 2205 H=2205:00 +CPU= 36.072
  06:11:53 STEP 2206 H=2206:00 +CPU= 25.341
  06:12:13 STEP 2207 H=2207:00 +CPU= 20.449
  06:13:01 STEP 2208 H=2208:00 +CPU= 46.554
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_91.zip
Uploading the intermediate file: upload_file_91.zip
  06:13:27 STEP 2209 H=2209:00 +CPU= 26.092
Uploading trickle at timestep: 7948800
  06:13:54 STEP 2210 H=2210:00 +CPU= 26.454
  06:14:35 STEP 2211 H=2211:00 +CPU= 40.870
  06:15:02 STEP 2212 H=2212:00 +CPU= 26.708
  06:15:27 STEP 2213 H=2213:00 +CPU= 25.562
  06:16:10 STEP 2214 H=2214:00 +CPU= 42.099
  06:16:36 STEP 2215 H=2215:00 +CPU= 25.880
  06:17:02 STEP 2216 H=2216:00 +CPU= 26.463
  06:17:43 STEP 2217 H=2217:00 +CPU= 40.663
  06:18:10 STEP 2218 H=2218:00 +CPU= 26.879
  06:18:35 STEP 2219 H=2219:00 +CPU= 24.429
  06:19:12 STEP 2220 H=2220:00 +CPU= 36.779
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002220
  06:19:39 STEP 2221 H=2221:00 +CPU= 26.562
  06:20:05 STEP 2222 H=2222:00 +CPU= 26.602
  06:20:47 STEP 2223 H=2223:00 +CPU= 40.849
  06:21:08 STEP 2224 H=2224:00 +CPU= 20.442
  06:21:28 STEP 2225 H=2225:00 +CPU= 20.272
  06:22:05 STEP 2226 H=2226:00 +CPU= 36.442
  06:22:31 STEP 2227 H=2227:00 +CPU= 25.846
  06:22:54 STEP 2228 H=2228:00 +CPU= 22.785
  06:23:34 STEP 2229 H=2229:00 +CPU= 39.417
  06:24:01 STEP 2230 H=2230:00 +CPU= 26.444
  06:24:29 STEP 2231 H=2231:00 +CPU= 28.091
  06:25:19 STEP 2232 H=2232:00 +CPU= 48.194
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_92.zip
Uploading the intermediate file: upload_file_92.zip
  06:25:42 STEP 2233 H=2233:00 +CPU= 23.005
  06:26:03 STEP 2234 H=2234:00 +CPU= 20.645
  06:26:36 STEP 2235 H=2235:00 +CPU= 32.431
  06:27:03 STEP 2236 H=2236:00 +CPU= 27.489
  06:27:30 STEP 2237 H=2237:00 +CPU= 26.499
  06:28:05 STEP 2238 H=2238:00 +CPU= 34.753
  06:28:32 STEP 2239 H=2239:00 +CPU= 26.520
  06:28:54 STEP 2240 H=2240:00 +CPU= 22.357
  06:29:34 STEP 2241 H=2241:00 +CPU= 39.214
  06:30:03 STEP 2242 H=2242:00 +CPU= 28.997
  06:30:30 STEP 2243 H=2243:00 +CPU= 27.329
  06:31:16 STEP 2244 H=2244:00 +CPU= 45.560
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002244
  06:31:43 STEP 2245 H=2245:00 +CPU= 27.056
  06:32:11 STEP 2246 H=2246:00 +CPU= 27.060
  06:32:51 STEP 2247 H=2247:00 +CPU= 40.560
  06:33:13 STEP 2248 H=2248:00 +CPU= 21.334
  06:33:38 STEP 2249 H=2249:00 +CPU= 25.034
  06:34:20 STEP 2250 H=2250:00 +CPU= 41.328
  06:34:40 STEP 2251 H=2251:00 +CPU= 19.938
  06:35:07 STEP 2252 H=2252:00 +CPU= 27.205
  06:35:42 STEP 2253 H=2253:00 +CPU= 34.183
  06:36:10 STEP 2254 H=2254:00 +CPU= 27.680
  06:36:37 STEP 2255 H=2255:00 +CPU= 26.221
  06:37:23 STEP 2256 H=2256:00 +CPU= 46.532
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_93.zip
Uploading the intermediate file: upload_file_93.zip
  06:37:49 STEP 2257 H=2257:00 +CPU= 26.089
  06:38:16 STEP 2258 H=2258:00 +CPU= 26.378
  06:38:57 STEP 2259 H=2259:00 +CPU= 40.839
  06:39:25 STEP 2260 H=2260:00 +CPU= 28.458
  06:39:51 STEP 2261 H=2261:00 +CPU= 25.452
  06:40:31 STEP 2262 H=2262:00 +CPU= 40.160
  06:40:56 STEP 2263 H=2263:00 +CPU= 25.629
  06:41:24 STEP 2264 H=2264:00 +CPU= 26.905
  06:41:59 STEP 2265 H=2265:00 +CPU= 35.549
  06:42:27 STEP 2266 H=2266:00 +CPU= 27.013
  06:42:54 STEP 2267 H=2267:00 +CPU= 26.646
  06:43:39 STEP 2268 H=2268:00 +CPU= 45.477
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002268
  06:44:06 STEP 2269 H=2269:00 +CPU= 26.597
  06:44:34 STEP 2270 H=2270:00 +CPU= 28.135
  06:45:17 STEP 2271 H=2271:00 +CPU= 41.477
  06:45:44 STEP 2272 H=2272:00 +CPU= 27.209
  06:46:12 STEP 2273 H=2273:00 +CPU= 27.448
  06:46:49 STEP 2274 H=2274:00 +CPU= 36.433
  06:47:08 STEP 2275 H=2275:00 +CPU= 18.912
  06:47:27 STEP 2276 H=2276:00 +CPU= 18.676
  06:48:05 STEP 2277 H=2277:00 +CPU= 37.952
  06:48:32 STEP 2278 H=2278:00 +CPU= 26.455
  06:48:59 STEP 2279 H=2279:00 +CPU= 26.699
  06:49:40 STEP 2280 H=2280:00 +CPU= 40.396
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_94.zip
Uploading the intermediate file: upload_file_94.zip
  06:50:06 STEP 2281 H=2281:00 +CPU= 25.907
  06:50:33 STEP 2282 H=2282:00 +CPU= 27.201
  06:51:14 STEP 2283 H=2283:00 +CPU= 40.693
  06:51:41 STEP 2284 H=2284:00 +CPU= 26.672
  06:52:06 STEP 2285 H=2285:00 +CPU= 24.742
  06:52:47 STEP 2286 H=2286:00 +CPU= 40.216
  06:53:12 STEP 2287 H=2287:00 +CPU= 25.076
  06:53:38 STEP 2288 H=2288:00 +CPU= 26.399
  06:54:17 STEP 2289 H=2289:00 +CPU= 38.921
  06:54:45 STEP 2290 H=2290:00 +CPU= 27.362
  06:55:07 STEP 2291 H=2291:00 +CPU= 21.743
  06:55:52 STEP 2292 H=2292:00 +CPU= 44.728
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002292
  06:56:19 STEP 2293 H=2293:00 +CPU= 26.691
  06:56:47 STEP 2294 H=2294:00 +CPU= 27.831
  06:57:27 STEP 2295 H=2295:00 +CPU= 40.242
  06:57:53 STEP 2296 H=2296:00 +CPU= 25.630
  06:58:21 STEP 2297 H=2297:00 +CPU= 26.866
  06:59:01 STEP 2298 H=2298:00 +CPU= 40.308
  06:59:28 STEP 2299 H=2299:00 +CPU= 27.018
  06:59:56 STEP 2300 H=2300:00 +CPU= 27.077
  07:00:36 STEP 2301 H=2301:00 +CPU= 39.812
  07:01:03 STEP 2302 H=2302:00 +CPU= 27.090
  07:01:30 STEP 2303 H=2303:00 +CPU= 27.178
  07:02:17 STEP 2304 H=2304:00 +CPU= 46.691
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_95.zip
Uploading the intermediate file: upload_file_95.zip
  07:02:43 STEP 2305 H=2305:00 +CPU= 25.635
  07:03:07 STEP 2306 H=2306:00 +CPU= 23.165
  07:03:44 STEP 2307 H=2307:00 +CPU= 37.035
  07:04:12 STEP 2308 H=2308:00 +CPU= 28.271
  07:04:40 STEP 2309 H=2309:00 +CPU= 27.031
  07:05:22 STEP 2310 H=2310:00 +CPU= 41.801
  07:05:49 STEP 2311 H=2311:00 +CPU= 27.487
  07:06:17 STEP 2312 H=2312:00 +CPU= 28.315
  07:06:59 STEP 2313 H=2313:00 +CPU= 40.715
  07:07:22 STEP 2314 H=2314:00 +CPU= 23.489
  07:07:40 STEP 2315 H=2315:00 +CPU= 18.246
  07:08:15 STEP 2316 H=2316:00 +CPU= 33.964
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002316
  07:08:40 STEP 2317 H=2317:00 +CPU= 24.603
  07:09:07 STEP 2318 H=2318:00 +CPU= 27.460
  07:09:45 STEP 2319 H=2319:00 +CPU= 37.214
  07:10:12 STEP 2320 H=2320:00 +CPU= 26.418
  07:10:37 STEP 2321 H=2321:00 +CPU= 24.963
  07:11:18 STEP 2322 H=2322:00 +CPU= 40.575
  07:11:45 STEP 2323 H=2323:00 +CPU= 26.540
  07:12:13 STEP 2324 H=2324:00 +CPU= 27.673
  07:12:53 STEP 2325 H=2325:00 +CPU= 40.100
  07:13:22 STEP 2326 H=2326:00 +CPU= 28.484
  07:13:49 STEP 2327 H=2327:00 +CPU= 27.724
  07:14:38 STEP 2328 H=2328:00 +CPU= 48.885
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_96.zip
Uploading the intermediate file: upload_file_96.zip
  07:15:06 STEP 2329 H=2329:00 +CPU= 27.086
  07:15:33 STEP 2330 H=2330:00 +CPU= 27.216
  07:16:14 STEP 2331 H=2331:00 +CPU= 40.590
  07:16:41 STEP 2332 H=2332:00 +CPU= 27.006
  07:17:06 STEP 2333 H=2333:00 +CPU= 25.547
  07:17:47 STEP 2334 H=2334:00 +CPU= 40.641
  07:18:13 STEP 2335 H=2335:00 +CPU= 26.028
  07:18:40 STEP 2336 H=2336:00 +CPU= 26.825
  07:19:20 STEP 2337 H=2337:00 +CPU= 40.521
  07:19:48 STEP 2338 H=2338:00 +CPU= 27.513
  07:20:15 STEP 2339 H=2339:00 +CPU= 26.226
  07:21:00 STEP 2340 H=2340:00 +CPU= 44.849
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002340
  07:21:25 STEP 2341 H=2341:00 +CPU= 25.222
  07:21:52 STEP 2342 H=2342:00 +CPU= 26.924
  07:22:33 STEP 2343 H=2343:00 +CPU= 40.410
  07:22:59 STEP 2344 H=2344:00 +CPU= 26.460
  07:23:25 STEP 2345 H=2345:00 +CPU= 25.865
  07:24:04 STEP 2346 H=2346:00 +CPU= 38.026
  07:24:30 STEP 2347 H=2347:00 +CPU= 25.913
  07:24:57 STEP 2348 H=2348:00 +CPU= 27.401
  07:25:38 STEP 2349 H=2349:00 +CPU= 40.064
  07:26:05 STEP 2350 H=2350:00 +CPU= 27.341
  07:26:32 STEP 2351 H=2351:00 +CPU= 26.849
  07:27:20 STEP 2352 H=2352:00 +CPU= 47.649
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_97.zip
Uploading the intermediate file: upload_file_97.zip
  07:27:46 STEP 2353 H=2353:00 +CPU= 25.558
  07:28:12 STEP 2354 H=2354:00 +CPU= 26.292
  07:28:49 STEP 2355 H=2355:00 +CPU= 36.907
  07:29:09 STEP 2356 H=2356:00 +CPU= 19.810
  07:29:28 STEP 2357 H=2357:00 +CPU= 18.834
  07:30:01 STEP 2358 H=2358:00 +CPU= 31.857
  07:30:26 STEP 2359 H=2359:00 +CPU= 24.487
  07:30:53 STEP 2360 H=2360:00 +CPU= 26.450
  07:31:34 STEP 2361 H=2361:00 +CPU= 41.142
  07:32:02 STEP 2362 H=2362:00 +CPU= 27.706
  07:32:28 STEP 2363 H=2363:00 +CPU= 26.080
  07:33:14 STEP 2364 H=2364:00 +CPU= 45.381
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002364
  07:33:41 STEP 2365 H=2365:00 +CPU= 26.744
  07:34:08 STEP 2366 H=2366:00 +CPU= 27.104
  07:34:44 STEP 2367 H=2367:00 +CPU= 35.222
  07:35:10 STEP 2368 H=2368:00 +CPU= 26.032
  07:35:36 STEP 2369 H=2369:00 +CPU= 25.645
  07:36:11 STEP 2370 H=2370:00 +CPU= 35.033
  07:36:31 STEP 2371 H=2371:00 +CPU= 19.327
  07:36:55 STEP 2372 H=2372:00 +CPU= 23.776
  07:37:35 STEP 2373 H=2373:00 +CPU= 39.570
  07:38:02 STEP 2374 H=2374:00 +CPU= 26.622
  07:38:28 STEP 2375 H=2375:00 +CPU= 26.004
  07:39:13 STEP 2376 H=2376:00 +CPU= 44.310
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_98.zip
Uploading the intermediate file: upload_file_98.zip
  07:39:41 STEP 2377 H=2377:00 +CPU= 27.008
  07:40:08 STEP 2378 H=2378:00 +CPU= 26.759
  07:40:49 STEP 2379 H=2379:00 +CPU= 40.566
  07:41:16 STEP 2380 H=2380:00 +CPU= 27.050
  07:41:41 STEP 2381 H=2381:00 +CPU= 25.208
  07:42:23 STEP 2382 H=2382:00 +CPU= 41.639
  07:42:49 STEP 2383 H=2383:00 +CPU= 26.057
  07:43:15 STEP 2384 H=2384:00 +CPU= 26.081
  07:43:56 STEP 2385 H=2385:00 +CPU= 40.873
  07:44:24 STEP 2386 H=2386:00 +CPU= 27.722
  07:44:49 STEP 2387 H=2387:00 +CPU= 25.463
  07:45:36 STEP 2388 H=2388:00 +CPU= 46.127
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002388
  07:45:57 STEP 2389 H=2389:00 +CPU= 21.709
  07:46:17 STEP 2390 H=2390:00 +CPU= 19.747
  07:46:59 STEP 2391 H=2391:00 +CPU= 41.272
  07:47:22 STEP 2392 H=2392:00 +CPU= 22.359
  07:47:43 STEP 2393 H=2393:00 +CPU= 20.265
  07:48:24 STEP 2394 H=2394:00 +CPU= 41.298
  07:48:51 STEP 2395 H=2395:00 +CPU= 26.505
  07:49:16 STEP 2396 H=2396:00 +CPU= 24.390
  07:49:54 STEP 2397 H=2397:00 +CPU= 37.511
  07:50:21 STEP 2398 H=2398:00 +CPU= 27.032
  07:50:47 STEP 2399 H=2399:00 +CPU= 26.199
  07:51:30 STEP 2400 H=2400:00 +CPU= 40.778
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_99.zip
Uploading the intermediate file: upload_file_99.zip
  07:51:56 STEP 2401 H=2401:00 +CPU= 26.298
  07:52:23 STEP 2402 H=2402:00 +CPU= 26.996
  07:53:05 STEP 2403 H=2403:00 +CPU= 41.212
  07:53:32 STEP 2404 H=2404:00 +CPU= 27.025
  07:53:58 STEP 2405 H=2405:00 +CPU= 25.036
  07:54:39 STEP 2406 H=2406:00 +CPU= 41.064
  07:55:05 STEP 2407 H=2407:00 +CPU= 26.229
  07:55:32 STEP 2408 H=2408:00 +CPU= 27.003
  07:56:13 STEP 2409 H=2409:00 +CPU= 40.826
  07:56:41 STEP 2410 H=2410:00 +CPU= 26.991
  07:57:07 STEP 2411 H=2411:00 +CPU= 26.007
  07:57:52 STEP 2412 H=2412:00 +CPU= 45.025
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002412
  07:58:13 STEP 2413 H=2413:00 +CPU= 20.151
  07:58:32 STEP 2414 H=2414:00 +CPU= 19.410
  07:59:12 STEP 2415 H=2415:00 +CPU= 39.138
  07:59:39 STEP 2416 H=2416:00 +CPU= 26.766
  08:00:05 STEP 2417 H=2417:00 +CPU= 26.418
  08:00:37 STEP 2418 H=2418:00 +CPU= 31.429
  08:01:01 STEP 2419 H=2419:00 +CPU= 23.744
  08:01:29 STEP 2420 H=2420:00 +CPU= 27.214
  08:02:09 STEP 2421 H=2421:00 +CPU= 40.383
  08:02:36 STEP 2422 H=2422:00 +CPU= 27.190
  08:03:04 STEP 2423 H=2423:00 +CPU= 27.185
  08:03:51 STEP 2424 H=2424:00 +CPU= 46.481
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_100.zip
Uploading the intermediate file: upload_file_100.zip
  08:04:15 STEP 2425 H=2425:00 +CPU= 23.507
  08:04:37 STEP 2426 H=2426:00 +CPU= 21.631
  08:05:10 STEP 2427 H=2427:00 +CPU= 32.906
  08:05:37 STEP 2428 H=2428:00 +CPU= 26.465
  08:06:03 STEP 2429 H=2429:00 +CPU= 26.453
  08:06:40 STEP 2430 H=2430:00 +CPU= 36.220
  08:07:06 STEP 2431 H=2431:00 +CPU= 26.451
  08:07:31 STEP 2432 H=2432:00 +CPU= 24.786
  08:08:12 STEP 2433 H=2433:00 +CPU= 40.197
  08:08:40 STEP 2434 H=2434:00 +CPU= 27.695
  08:09:06 STEP 2435 H=2435:00 +CPU= 26.620
  08:09:53 STEP 2436 H=2436:00 +CPU= 46.312
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002436
  08:10:21 STEP 2437 H=2437:00 +CPU= 26.959
  08:10:45 STEP 2438 H=2438:00 +CPU= 24.180
  08:11:25 STEP 2439 H=2439:00 +CPU= 39.823
  08:11:54 STEP 2440 H=2440:00 +CPU= 28.668
  08:12:21 STEP 2441 H=2441:00 +CPU= 27.093
  08:13:00 STEP 2442 H=2442:00 +CPU= 38.034
  08:13:26 STEP 2443 H=2443:00 +CPU= 26.187
  08:13:54 STEP 2444 H=2444:00 +CPU= 27.430
  08:14:33 STEP 2445 H=2445:00 +CPU= 39.785
  08:15:00 STEP 2446 H=2446:00 +CPU= 26.793
  08:15:28 STEP 2447 H=2447:00 +CPU= 27.393
  08:16:16 STEP 2448 H=2448:00 +CPU= 47.643
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_101.zip
Uploading the intermediate file: upload_file_101.zip
  08:16:42 STEP 2449 H=2449:00 +CPU= 25.848
Uploading trickle at timestep: 8812800
  08:17:10 STEP 2450 H=2450:00 +CPU= 28.203
  08:17:51 STEP 2451 H=2451:00 +CPU= 41.096
  08:18:18 STEP 2452 H=2452:00 +CPU= 26.642
  08:18:37 STEP 2453 H=2453:00 +CPU= 19.395
  08:19:20 STEP 2454 H=2454:00 +CPU= 42.488
  08:19:47 STEP 2455 H=2455:00 +CPU= 26.495
  08:20:14 STEP 2456 H=2456:00 +CPU= 27.086
  08:20:49 STEP 2457 H=2457:00 +CPU= 34.495
  08:21:10 STEP 2458 H=2458:00 +CPU= 20.797
  08:21:37 STEP 2459 H=2459:00 +CPU= 26.584
  08:22:23 STEP 2460 H=2460:00 +CPU= 45.621
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002460
  08:22:48 STEP 2461 H=2461:00 +CPU= 25.891
  08:23:14 STEP 2462 H=2462:00 +CPU= 25.420
  08:23:56 STEP 2463 H=2463:00 +CPU= 41.319
  08:24:19 STEP 2464 H=2464:00 +CPU= 22.576
  08:24:45 STEP 2465 H=2465:00 +CPU= 25.778
  08:25:27 STEP 2466 H=2466:00 +CPU= 42.232
  08:25:52 STEP 2467 H=2467:00 +CPU= 25.368
  08:26:18 STEP 2468 H=2468:00 +CPU= 25.775
  08:26:58 STEP 2469 H=2469:00 +CPU= 39.741
  08:27:26 STEP 2470 H=2470:00 +CPU= 27.540
  08:27:53 STEP 2471 H=2471:00 +CPU= 26.996
  08:28:38 STEP 2472 H=2472:00 +CPU= 44.718
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_102.zip
Uploading the intermediate file: upload_file_102.zip
  08:29:01 STEP 2473 H=2473:00 +CPU= 22.586
  08:29:28 STEP 2474 H=2474:00 +CPU= 26.411
  08:30:07 STEP 2475 H=2475:00 +CPU= 39.013
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.818] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4404MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.818] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.818] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.818] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.818] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.818] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4404 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.818] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.923] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.981] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.981] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083026:1674199826:49222.999] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                   VDFDIFM 
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083027:1674199827:49223.999] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:25264:25264] [20230120:083027:1674199827:49223.999] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FC6BDD  vdfdifm_                  139  vdfdifm.F90
oifs_43r3_model.e  0000000000FADC81  vdfmain_                 1162  vdfmain.F90
oifs_43r3_model.e  0000000000F92699  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  0000000000F88512  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000F0B683  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0518
(argv4) batchid: 968
(argv5) wuid: 12185162
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_eaa928c35b0715014b5210d03721876b to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0518_1999050100_123_968_12185162.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0518_1999050100_123_968_12185162.zip
ic_ancil_file: ic_ancil_12185162
ifsdata_file: ifsdata_12185162
climate_data_file: clim_data_12185162
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_86d8c259774ae8a295a46d7271fddb4c to: /var/lib/boinc-client/slots/11/ic_ancil_12185162.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12185162.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_03b2ff7cfd917e1627d21c2c7cf25a9f to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12185162.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12185162.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12185162.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12185162.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12185162.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12185162.xml
-- Model is restarting --
Adjusting last_iter, 2476, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12185162.xml
last_cpu_time: 72792
upload_file_number: 103
last_iter: 2471
last_upload: 8899200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507
The child process has been launched with process id: 1802
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:1802:1802] [20230120:083247:1674199967:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:34:25 STEP 2472 H=2472:00 +CPU= 44.560
  08:34:50 STEP 2473 H=2473:00 +CPU= 25.237
  08:35:18 STEP 2474 H=2474:00 +CPU= 27.218
  08:35:57 STEP 2475 H=2475:00 +CPU= 39.088
  08:36:24 STEP 2476 H=2476:00 +CPU= 25.692
  08:36:49 STEP 2477 H=2477:00 +CPU= 24.944
  08:37:29 STEP 2478 H=2478:00 +CPU= 39.416
  08:37:55 STEP 2479 H=2479:00 +CPU= 25.146
  08:38:22 STEP 2480 H=2480:00 +CPU= 25.770
  08:39:05 STEP 2481 H=2481:00 +CPU= 38.997
  08:39:33 STEP 2482 H=2482:00 +CPU= 25.821
  08:39:59 STEP 2483 H=2483:00 +CPU= 25.365
  08:40:46 STEP 2484 H=2484:00 +CPU= 42.842
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002484
  08:41:14 STEP 2485 H=2485:00 +CPU= 25.355
  08:41:40 STEP 2486 H=2486:00 +CPU= 25.834
  08:42:20 STEP 2487 H=2487:00 +CPU= 38.635
  08:42:47 STEP 2488 H=2488:00 +CPU= 26.005
  08:43:13 STEP 2489 H=2489:00 +CPU= 25.661
  08:43:54 STEP 2490 H=2490:00 +CPU= 40.425
  08:44:21 STEP 2491 H=2491:00 +CPU= 25.403
  08:44:48 STEP 2492 H=2492:00 +CPU= 26.284
  08:45:27 STEP 2493 H=2493:00 +CPU= 38.692
  08:45:55 STEP 2494 H=2494:00 +CPU= 25.849
  08:46:23 STEP 2495 H=2495:00 +CPU= 26.539
  08:47:13 STEP 2496 H=2496:00 +CPU= 46.753
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185162/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0518_1999050100_123_968_12185162_0_r352656507_103.zip
Uploading the intermediate file: upload_file_103.zip
  08:47:41 STEP 2497 H=2497:00 +CPU= 25.447
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:30:21 1534452 22270450 oifs_43r3_ps_0518_1999050100_123_968_12185162_0 8,812,800 71,962 0.0082
09 May 2023 16:30:12 1534452 22270450 oifs_43r3_ps_0518_1999050100_123_968_12185162_0 7,948,800 64,629 0.0081
09 May 2023 16:29:55 1534452 22270450 oifs_43r3_ps_0518_1999050100_123_968_12185162_0 7,084,800 57,780 0.0082
09 May 2023 16:29:45 1534452 22270450 oifs_43r3_ps_0518_1999050100_123_968_12185162_0 6,220,800 51,197 0.0082
09 May 2023 16:29:35 1534452 22270450 oifs_43r3_ps_0518_1999050100_123_968_12185162_0 5,356,800 45,029 0.0084
09 May 2023 16:29:25 1534452 22270450 oifs_43r3_ps_0518_1999050100_123_968_12185162_0 4,492,800 38,788 0.0086
09 May 2023 16:29:15 1534452 22270450 oifs_43r3_ps_0518_1999050100_123_968_12185162_0 3,628,800 32,681 0.0090
09 May 2023 16:28:51 1534452 22270450 oifs_43r3_ps_0518_1999050100_123_968_12185162_0 2,246,400 19,813 0.0088


©2024 cpdn.org