climateprediction.net home page
Task 22282386

Task 22282386

Name oifs_43r3_ps_0760_2010050100_123_979_12196404_0
Workunit 12196404
Created 24 Dec 2022, 0:32:00 UTC
Sent 27 Dec 2022, 18:23:15 UTC
Report deadline 26 Jan 2023, 18:23:15 UTC
Received 6 Jan 2023, 11:37:40 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1489681
Run time 15 hours 27 min 30 sec
CPU time 10 hours 16 min 50 sec
Validate state Invalid
Credit 11,249.28
Device peak FLOPS 6.59 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,759.39 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,935.54 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_64.zip
Uploading the intermediate file: upload_file_64.zip
  04:06:03 STEP 2233 H=2233:00 +CPU= 16.614
Uploading trickle at timestep: 8035200
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:06:23 STEP 2234 H=2234:00 +CPU= 17.506
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:06:52 STEP 2235 H=2235:00 +CPU= 27.268
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:07:11 STEP 2236 H=2236:00 +CPU= 16.818
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:07:30 STEP 2237 H=2237:00 +CPU= 17.125
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:08:02 STEP 2238 H=2238:00 +CPU= 28.003
  04:08:19 STEP 2239 H=2239:00 +CPU= 17.139
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:08:39 STEP 2240 H=2240:00 +CPU= 17.640
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:09:11 STEP 2241 H=2241:00 +CPU= 27.773
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:09:30 STEP 2242 H=2242:00 +CPU= 17.661
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:09:50 STEP 2243 H=2243:00 +CPU= 17.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:10:25 STEP 2244 H=2244:00 +CPU= 30.959
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002244
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002244
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002244
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:10:44 STEP 2245 H=2245:00 +CPU= 17.240
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:11:04 STEP 2246 H=2246:00 +CPU= 17.676
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:11:33 STEP 2247 H=2247:00 +CPU= 27.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:11:53 STEP 2248 H=2248:00 +CPU= 17.736
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:12:12 STEP 2249 H=2249:00 +CPU= 17.186
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:12:45 STEP 2250 H=2250:00 +CPU= 28.247
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:13:04 STEP 2251 H=2251:00 +CPU= 17.170
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:13:23 STEP 2252 H=2252:00 +CPU= 17.558
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:13:53 STEP 2253 H=2253:00 +CPU= 27.779
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:14:13 STEP 2254 H=2254:00 +CPU= 17.694
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:14:33 STEP 2255 H=2255:00 +CPU= 17.653
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:15:09 STEP 2256 H=2256:00 +CPU= 31.684
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002256
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002256
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002256
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002232
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002232
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002232
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002244
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002244
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002244
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_65.zip
Uploading the intermediate file: upload_file_65.zip
  04:15:26 STEP 2257 H=2257:00 +CPU= 17.015
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:15:45 STEP 2258 H=2258:00 +CPU= 17.581
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:16:15 STEP 2259 H=2259:00 +CPU= 27.718
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:16:35 STEP 2260 H=2260:00 +CPU= 17.595
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:16:54 STEP 2261 H=2261:00 +CPU= 17.108
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:17:26 STEP 2262 H=2262:00 +CPU= 28.226
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:17:45 STEP 2263 H=2263:00 +CPU= 16.728
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:18:04 STEP 2264 H=2264:00 +CPU= 17.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:18:34 STEP 2265 H=2265:00 +CPU= 27.789
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:18:54 STEP 2266 H=2266:00 +CPU= 17.723
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:19:13 STEP 2267 H=2267:00 +CPU= 17.175
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:19:47 STEP 2268 H=2268:00 +CPU= 30.104
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002268
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002268
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002268
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:20:06 STEP 2269 H=2269:00 +CPU= 16.457
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:20:25 STEP 2270 H=2270:00 +CPU= 16.967
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:20:53 STEP 2271 H=2271:00 +CPU= 25.683
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:21:12 STEP 2272 H=2272:00 +CPU= 17.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:21:31 STEP 2273 H=2273:00 +CPU= 16.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:22:00 STEP 2274 H=2274:00 +CPU= 27.112
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:22:19 STEP 2275 H=2275:00 +CPU= 17.242
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:22:39 STEP 2276 H=2276:00 +CPU= 17.322
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:23:09 STEP 2277 H=2277:00 +CPU= 26.242
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:23:28 STEP 2278 H=2278:00 +CPU= 17.447
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:23:48 STEP 2279 H=2279:00 +CPU= 17.440
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:24:24 STEP 2280 H=2280:00 +CPU= 31.700
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002280
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002280
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002280
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002256
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002256
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002256
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002268
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002268
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002268
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_66.zip
Uploading the intermediate file: upload_file_66.zip
  04:24:41 STEP 2281 H=2281:00 +CPU= 16.868
  04:24:59 STEP 2282 H=2282:00 +CPU= 17.443
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:25:28 STEP 2283 H=2283:00 +CPU= 25.589
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:25:47 STEP 2284 H=2284:00 +CPU= 16.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:26:06 STEP 2285 H=2285:00 +CPU= 16.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:26:36 STEP 2286 H=2286:00 +CPU= 27.541
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:26:55 STEP 2287 H=2287:00 +CPU= 16.896
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:27:14 STEP 2288 H=2288:00 +CPU= 17.540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:27:46 STEP 2289 H=2289:00 +CPU= 27.299
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:28:05 STEP 2290 H=2290:00 +CPU= 17.495
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:28:24 STEP 2291 H=2291:00 +CPU= 16.964
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:28:57 STEP 2292 H=2292:00 +CPU= 30.293
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002292
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002292
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002292
  04:29:16 STEP 2293 H=2293:00 +CPU= 17.177
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:29:35 STEP 2294 H=2294:00 +CPU= 17.424
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:30:07 STEP 2295 H=2295:00 +CPU= 27.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:30:26 STEP 2296 H=2296:00 +CPU= 17.283
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:30:45 STEP 2297 H=2297:00 +CPU= 16.935
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:31:15 STEP 2298 H=2298:00 +CPU= 28.074
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:31:34 STEP 2299 H=2299:00 +CPU= 17.069
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:31:54 STEP 2300 H=2300:00 +CPU= 17.363
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:32:25 STEP 2301 H=2301:00 +CPU= 27.372
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:32:45 STEP 2302 H=2302:00 +CPU= 17.339
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:33:03 STEP 2303 H=2303:00 +CPU= 16.689
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:33:37 STEP 2304 H=2304:00 +CPU= 31.635
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002304
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002304
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002304
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002280
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002280
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002280
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002292
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002292
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002292
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_67.zip
Uploading the intermediate file: upload_file_67.zip
  04:33:54 STEP 2305 H=2305:00 +CPU= 17.198
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:34:14 STEP 2306 H=2306:00 +CPU= 17.566
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:34:44 STEP 2307 H=2307:00 +CPU= 26.023
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:35:03 STEP 2308 H=2308:00 +CPU= 16.917
  04:35:19 STEP 2309 H=2309:00 +CPU= 16.516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:35:51 STEP 2310 H=2310:00 +CPU= 27.927
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:36:10 STEP 2311 H=2311:00 +CPU= 17.020
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:36:29 STEP 2312 H=2312:00 +CPU= 16.554
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:36:58 STEP 2313 H=2313:00 +CPU= 26.915
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:37:17 STEP 2314 H=2314:00 +CPU= 17.071
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:37:36 STEP 2315 H=2315:00 +CPU= 17.139
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:38:10 STEP 2316 H=2316:00 +CPU= 29.488
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002316
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002316
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002316
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:38:28 STEP 2317 H=2317:00 +CPU= 16.272
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:38:48 STEP 2318 H=2318:00 +CPU= 17.633
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:39:17 STEP 2319 H=2319:00 +CPU= 27.439
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:39:37 STEP 2320 H=2320:00 +CPU= 17.626
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:39:56 STEP 2321 H=2321:00 +CPU= 16.839
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:40:28 STEP 2322 H=2322:00 +CPU= 28.082
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:40:47 STEP 2323 H=2323:00 +CPU= 17.063
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:41:06 STEP 2324 H=2324:00 +CPU= 16.869
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:41:35 STEP 2325 H=2325:00 +CPU= 26.522
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:41:53 STEP 2326 H=2326:00 +CPU= 16.563
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:42:13 STEP 2327 H=2327:00 +CPU= 17.539
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:42:48 STEP 2328 H=2328:00 +CPU= 31.597
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002328
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002328
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002328
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002304
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002304
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002304
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002316
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002316
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002316
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_68.zip
Uploading the intermediate file: upload_file_68.zip
  04:43:06 STEP 2329 H=2329:00 +CPU= 17.103
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:43:25 STEP 2330 H=2330:00 +CPU= 17.261
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:43:55 STEP 2331 H=2331:00 +CPU= 27.418
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:44:14 STEP 2332 H=2332:00 +CPU= 17.493
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:44:33 STEP 2333 H=2333:00 +CPU= 17.049
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:45:04 STEP 2334 H=2334:00 +CPU= 27.025
  04:45:19 STEP 2335 H=2335:00 +CPU= 14.706
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:45:37 STEP 2336 H=2336:00 +CPU= 16.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:46:07 STEP 2337 H=2337:00 +CPU= 25.874
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:46:27 STEP 2338 H=2338:00 +CPU= 17.377
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:46:46 STEP 2339 H=2339:00 +CPU= 17.142
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:47:19 STEP 2340 H=2340:00 +CPU= 30.772
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002340
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002340
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002340
  04:47:38 STEP 2341 H=2341:00 +CPU= 17.247
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:47:58 STEP 2342 H=2342:00 +CPU= 17.676
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:48:29 STEP 2343 H=2343:00 +CPU= 27.115
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:48:49 STEP 2344 H=2344:00 +CPU= 17.697
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:49:08 STEP 2345 H=2345:00 +CPU= 17.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:49:38 STEP 2346 H=2346:00 +CPU= 27.941
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:49:56 STEP 2347 H=2347:00 +CPU= 16.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:50:16 STEP 2348 H=2348:00 +CPU= 17.511
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:50:48 STEP 2349 H=2349:00 +CPU= 27.557
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:51:07 STEP 2350 H=2350:00 +CPU= 17.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:51:27 STEP 2351 H=2351:00 +CPU= 17.549
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:52:03 STEP 2352 H=2352:00 +CPU= 31.892
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002352
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002352
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002352
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002328
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002328
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002328
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002340
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002340
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002340
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_69.zip
Uploading the intermediate file: upload_file_69.zip
  04:52:20 STEP 2353 H=2353:00 +CPU= 17.051
  04:52:37 STEP 2354 H=2354:00 +CPU= 17.201
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:53:08 STEP 2355 H=2355:00 +CPU= 26.478
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:53:27 STEP 2356 H=2356:00 +CPU= 17.535
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:53:46 STEP 2357 H=2357:00 +CPU= 17.070
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:54:16 STEP 2358 H=2358:00 +CPU= 27.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:54:35 STEP 2359 H=2359:00 +CPU= 17.131
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:54:55 STEP 2360 H=2360:00 +CPU= 17.574
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:55:26 STEP 2361 H=2361:00 +CPU= 26.834
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:55:46 STEP 2362 H=2362:00 +CPU= 17.702
  04:56:03 STEP 2363 H=2363:00 +CPU= 17.002
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:56:34 STEP 2364 H=2364:00 +CPU= 29.511
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002364
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002364
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002364
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:56:53 STEP 2365 H=2365:00 +CPU= 17.169
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:57:13 STEP 2366 H=2366:00 +CPU= 17.583
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:57:44 STEP 2367 H=2367:00 +CPU= 27.353
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:58:04 STEP 2368 H=2368:00 +CPU= 17.409
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:58:22 STEP 2369 H=2369:00 +CPU= 16.150
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:58:50 STEP 2370 H=2370:00 +CPU= 25.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:59:08 STEP 2371 H=2371:00 +CPU= 16.471
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:59:28 STEP 2372 H=2372:00 +CPU= 17.123
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:59:56 STEP 2373 H=2373:00 +CPU= 26.102
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:00:15 STEP 2374 H=2374:00 +CPU= 16.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:00:33 STEP 2375 H=2375:00 +CPU= 15.813
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:01:03 STEP 2376 H=2376:00 +CPU= 26.856
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002376
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002376
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002376
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002352
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002352
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002352
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002364
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002364
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002364
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_70.zip
Uploading the intermediate file: upload_file_70.zip
  05:01:17 STEP 2377 H=2377:00 +CPU= 13.797
  05:01:31 STEP 2378 H=2378:00 +CPU= 13.939
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:01:59 STEP 2379 H=2379:00 +CPU= 25.257
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:02:18 STEP 2380 H=2380:00 +CPU= 17.276
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:02:36 STEP 2381 H=2381:00 +CPU= 15.988
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:03:07 STEP 2382 H=2382:00 +CPU= 27.571
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:03:26 STEP 2383 H=2383:00 +CPU= 16.918
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:03:46 STEP 2384 H=2384:00 +CPU= 17.465
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:04:15 STEP 2385 H=2385:00 +CPU= 26.779
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:04:34 STEP 2386 H=2386:00 +CPU= 17.316
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:04:52 STEP 2387 H=2387:00 +CPU= 16.370
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:05:27 STEP 2388 H=2388:00 +CPU= 30.057
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002388
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002388
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002388
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:05:45 STEP 2389 H=2389:00 +CPU= 16.484
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:06:04 STEP 2390 H=2390:00 +CPU= 16.459
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:06:29 STEP 2391 H=2391:00 +CPU= 23.374
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:06:48 STEP 2392 H=2392:00 +CPU= 17.162
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:07:07 STEP 2393 H=2393:00 +CPU= 16.628
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:07:37 STEP 2394 H=2394:00 +CPU= 27.743
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:07:56 STEP 2395 H=2395:00 +CPU= 16.945
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:08:10 STEP 2396 H=2396:00 +CPU= 12.467
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:08:36 STEP 2397 H=2397:00 +CPU= 23.371
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:08:53 STEP 2398 H=2398:00 +CPU= 15.286
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:09:12 STEP 2399 H=2399:00 +CPU= 16.639
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:09:47 STEP 2400 H=2400:00 +CPU= 31.346
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002400
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002400
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002400
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002376
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002376
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002376
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002388
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002388
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002388
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_71.zip
Uploading the intermediate file: upload_file_71.zip
  05:10:04 STEP 2401 H=2401:00 +CPU= 16.702
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:10:23 STEP 2402 H=2402:00 +CPU= 17.353
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:10:52 STEP 2403 H=2403:00 +CPU= 26.226
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:11:10 STEP 2404 H=2404:00 +CPU= 16.722
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:11:29 STEP 2405 H=2405:00 +CPU= 16.739
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:11:58 STEP 2406 H=2406:00 +CPU= 26.805
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:12:16 STEP 2407 H=2407:00 +CPU= 15.824
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:12:35 STEP 2408 H=2408:00 +CPU= 17.243
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:13:06 STEP 2409 H=2409:00 +CPU= 26.910
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:13:26 STEP 2410 H=2410:00 +CPU= 17.524
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:13:44 STEP 2411 H=2411:00 +CPU= 16.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:14:15 STEP 2412 H=2412:00 +CPU= 28.804
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002412
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002412
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002412
  05:14:34 STEP 2413 H=2413:00 +CPU= 17.043
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:14:53 STEP 2414 H=2414:00 +CPU= 17.553
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:15:25 STEP 2415 H=2415:00 +CPU= 27.349
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:15:44 STEP 2416 H=2416:00 +CPU= 17.452
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:16:03 STEP 2417 H=2417:00 +CPU= 17.055
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:16:34 STEP 2418 H=2418:00 +CPU= 28.210
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:16:53 STEP 2419 H=2419:00 +CPU= 17.235
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:17:11 STEP 2420 H=2420:00 +CPU= 16.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:17:42 STEP 2421 H=2421:00 +CPU= 26.612
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:18:02 STEP 2422 H=2422:00 +CPU= 17.441
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:18:21 STEP 2423 H=2423:00 +CPU= 17.768
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:18:54 STEP 2424 H=2424:00 +CPU= 30.652
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002424
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002424
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002424
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002400
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002400
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002400
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002412
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002412
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002412
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_72.zip
Uploading the intermediate file: upload_file_72.zip
  05:19:11 STEP 2425 H=2425:00 +CPU= 16.763
  05:19:28 STEP 2426 H=2426:00 +CPU= 17.154
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:19:57 STEP 2427 H=2427:00 +CPU= 26.878
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:20:15 STEP 2428 H=2428:00 +CPU= 16.270
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:20:34 STEP 2429 H=2429:00 +CPU= 16.930
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:21:07 STEP 2430 H=2430:00 +CPU= 28.258
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:21:26 STEP 2431 H=2431:00 +CPU= 17.052
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:21:45 STEP 2432 H=2432:00 +CPU= 17.303
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:22:14 STEP 2433 H=2433:00 +CPU= 26.246
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:22:33 STEP 2434 H=2434:00 +CPU= 17.403
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:22:51 STEP 2435 H=2435:00 +CPU= 16.352
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:23:26 STEP 2436 H=2436:00 +CPU= 30.602
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002436
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002436
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:23:44 STEP 2437 H=2437:00 +CPU= 15.817
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:24:03 STEP 2438 H=2438:00 +CPU= 17.243
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:24:33 STEP 2439 H=2439:00 +CPU= 27.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:24:52 STEP 2440 H=2440:00 +CPU= 17.385
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:25:11 STEP 2441 H=2441:00 +CPU= 17.221
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:25:44 STEP 2442 H=2442:00 +CPU= 28.104
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:26:03 STEP 2443 H=2443:00 +CPU= 17.195
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:26:22 STEP 2444 H=2444:00 +CPU= 17.477
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:26:51 STEP 2445 H=2445:00 +CPU= 26.208
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:27:10 STEP 2446 H=2446:00 +CPU= 16.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:27:28 STEP 2447 H=2447:00 +CPU= 16.789
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:28:04 STEP 2448 H=2448:00 +CPU= 31.952
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002448
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002448
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002448
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002424
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002424
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002424
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002436
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002436
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002436
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_73.zip
Uploading the intermediate file: upload_file_73.zip
  05:28:21 STEP 2449 H=2449:00 +CPU= 17.009
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:28:41 STEP 2450 H=2450:00 +CPU= 17.567
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0760
(argv4) batchid: 979
(argv5) wuid: 12196404
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/nvme/scratch1/boinc/slots/1
Project directory is: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404
..mkdir for temp folder for results failed
Copying: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_175a20700ec0a4b06faf15647e96986c to: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_0760_2010050100_123_979_12196404.zip
Unzipping the namelist zip file: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_0760_2010050100_123_979_12196404.zip
ic_ancil_file: ic_ancil_12196404
ifsdata_file: ifsdata_12196404
climate_data_file: clim_data_12196404
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b54035023075d27e0cd65a1acf95760a to: /mnt/nvme/scratch1/boinc/slots/1/ic_ancil_12196404.zip
Unzipping the IC ancils zip file: /mnt/nvme/scratch1/boinc/slots/1/ic_ancil_12196404.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5e40eb891275bb837f87cd1690b01ecb to: /mnt/nvme/scratch1/boinc/slots/1/ifsdata/ifsdata_12196404.zip
Unzipping the ifsdata_zip file: /mnt/nvme/scratch1/boinc/slots/1/ifsdata/ifsdata_12196404.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/nvme/scratch1/boinc/slots/1/159l_2/clim_data_12196404.zip
Unzipping the climate data zip file: /mnt/nvme/scratch1/boinc/slots/1/159l_2/clim_data_12196404.zip
Checking for progress XML file: /mnt/nvme/scratch1/boinc/slots/1/progress_file_12196404.xml
Opened progress file ok : /mnt/nvme/scratch1/boinc/slots/1/progress_file_12196404.xml
-- Model is restarting --
Adjusting last_iter, 2451, to previous model restart step.
Creating progress file: /mnt/nvme/scratch1/boinc/slots/1/progress_file_12196404.xml
last_cpu_time: 36219
upload_file_number: 74
last_iter: 2447
last_upload: 8812800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216
The child process has been launched with process id: 29950
Executing the command: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:29950:29950] [20230106:112119:1673000479:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:21:58 STEP 2448 H=2448:00 +CPU= 32.756
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:22:18 STEP 2449 H=2449:00 +CPU= 17.933
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:22:39 STEP 2450 H=2450:00 +CPU= 18.748
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:23:12 STEP 2451 H=2451:00 +CPU= 28.828
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:23:32 STEP 2452 H=2452:00 +CPU= 18.469
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:23:52 STEP 2453 H=2453:00 +CPU= 17.970
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:24:24 STEP 2454 H=2454:00 +CPU= 29.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:24:44 STEP 2455 H=2455:00 +CPU= 18.058
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:25:04 STEP 2456 H=2456:00 +CPU= 18.489
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:25:33 STEP 2457 H=2457:00 +CPU= 24.527
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:25:53 STEP 2458 H=2458:00 +CPU= 18.407
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:26:13 STEP 2459 H=2459:00 +CPU= 17.844
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:26:45 STEP 2460 H=2460:00 +CPU= 30.170
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002460
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002460
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002460
  11:27:06 STEP 2461 H=2461:00 +CPU= 17.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:27:26 STEP 2462 H=2462:00 +CPU= 18.337
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:27:59 STEP 2463 H=2463:00 +CPU= 28.794
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:28:19 STEP 2464 H=2464:00 +CPU= 18.449
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:28:39 STEP 2465 H=2465:00 +CPU= 18.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:29:12 STEP 2466 H=2466:00 +CPU= 29.049
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:29:32 STEP 2467 H=2467:00 +CPU= 17.926
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:29:53 STEP 2468 H=2468:00 +CPU= 18.272
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:30:23 STEP 2469 H=2469:00 +CPU= 28.771
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:30:44 STEP 2470 H=2470:00 +CPU= 18.531
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:31:05 STEP 2471 H=2471:00 +CPU= 18.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:31:42 STEP 2472 H=2472:00 +CPU= 33.649
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+002472
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+002472
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+002472
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002448
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002448
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002448
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMGGhq0f+002460
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMSHhq0f+002460
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12196404/ICMUAhq0f+002460
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0760_2010050100_123_979_12196404_0_r450511216_74.zip
Uploading the intermediate file: upload_file_74.zip
  11:32:02 STEP 2473 H=2473:00 +CPU= 17.771
  11:32:21 STEP 2474 H=2474:00 +CPU= 18.296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:32:53 STEP 2475 H=2475:00 +CPU= 28.217
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:33:13 STEP 2476 H=2476:00 +CPU= 18.226
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:33:33 STEP 2477 H=2477:00 +CPU= 17.860
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:34:04 STEP 2478 H=2478:00 +CPU= 29.277
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:34:24 STEP 2479 H=2479:00 +CPU= 17.821
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:34:45 STEP 2480 H=2480:00 +CPU= 18.441
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:17 STEP 2481 H=2481:00 +CPU= 28.714
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:38 STEP 2482 H=2482:00 +CPU= 18.229
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:58 STEP 2483 H=2483:00 +CPU= 17.869
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:52 1489681 22282386 oifs_43r3_ps_0760_2010050100_123_979_12196404_0 8,035,200 31,722 0.0039
09 May 2023 16:09:26 1489681 22282386 oifs_43r3_ps_0760_2010050100_123_979_12196404_0 6,739,200 23,985 0.0036
09 May 2023 16:09:22 1489681 22282386 oifs_43r3_ps_0760_2010050100_123_979_12196404_0 5,875,200 19,116 0.0033
09 May 2023 16:09:18 1489681 22282386 oifs_43r3_ps_0760_2010050100_123_979_12196404_0 5,011,200 14,253 0.0028
09 May 2023 16:09:10 1489681 22282386 oifs_43r3_ps_0760_2010050100_123_979_12196404_0 4,147,200 9,309 0.0022
09 May 2023 16:09:07 1489681 22282386 oifs_43r3_ps_0760_2010050100_123_979_12196404_0 3,283,200 4,398 0.0013
09 May 2023 16:07:14 1489681 22282386 oifs_43r3_ps_0760_2010050100_123_979_12196404_0 2,160,000 8,434 0.0039


©2024 climateprediction.net