climateprediction.net home page
Task 22295505

Task 22295505

Name oifs_43r3_ps_0405_1997050100_123_966_12183049_1
Workunit 12183049
Created 4 Jan 2023, 18:12:39 UTC
Sent 4 Jan 2023, 18:41:18 UTC
Report deadline 3 Feb 2023, 18:41:18 UTC
Received 5 Jan 2023, 12:48:36 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1505480
Run time 14 hours 28 min 41 sec
CPU time 13 hours 46 min 51 sec
Validate state Invalid
Credit 8,346.24
Device peak FLOPS 6.09 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.05 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,528.14 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
-client/slots/6/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_53.zip
Uploading the intermediate file: upload_file_53.zip
  00:37:18 STEP 1297 H=1297:00 +CPU= 34.241
  00:38:00 STEP 1298 H=1298:00 +CPU= 38.488
  00:38:45 STEP 1299 H=1299:00 +CPU= 41.549
  00:39:05 STEP 1300 H=1300:00 +CPU= 19.547
  00:39:23 STEP 1301 H=1301:00 +CPU= 18.054
  00:39:51 STEP 1302 H=1302:00 +CPU= 27.954
  00:40:13 STEP 1303 H=1303:00 +CPU= 21.922
  00:40:36 STEP 1304 H=1304:00 +CPU= 22.818
  00:41:05 STEP 1305 H=1305:00 +CPU= 28.353
  00:41:23 STEP 1306 H=1306:00 +CPU= 18.288
  00:41:41 STEP 1307 H=1307:00 +CPU= 17.875
  00:42:10 STEP 1308 H=1308:00 +CPU= 28.837
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001308
  00:42:37 STEP 1309 H=1309:00 +CPU= 26.087
  00:43:19 STEP 1310 H=1310:00 +CPU= 40.939
  00:44:09 STEP 1311 H=1311:00 +CPU= 44.798
  00:44:29 STEP 1312 H=1312:00 +CPU= 20.117
  00:44:50 STEP 1313 H=1313:00 +CPU= 21.340
  00:45:17 STEP 1314 H=1314:00 +CPU= 26.463
  00:45:31 STEP 1315 H=1315:00 +CPU= 13.728
  00:45:45 STEP 1316 H=1316:00 +CPU= 14.261
  00:46:07 STEP 1317 H=1317:00 +CPU= 21.806
  00:46:21 STEP 1318 H=1318:00 +CPU= 14.804
  00:46:40 STEP 1319 H=1319:00 +CPU= 17.968
  00:47:10 STEP 1320 H=1320:00 +CPU= 30.673
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_54.zip
Uploading the intermediate file: upload_file_54.zip
  00:47:24 STEP 1321 H=1321:00 +CPU= 13.741
  00:47:38 STEP 1322 H=1322:00 +CPU= 14.267
  00:48:00 STEP 1323 H=1323:00 +CPU= 21.811
  00:48:22 STEP 1324 H=1324:00 +CPU= 21.299
  00:49:02 STEP 1325 H=1325:00 +CPU= 39.036
  00:49:58 STEP 1326 H=1326:00 +CPU= 55.369
  00:50:36 STEP 1327 H=1327:00 +CPU= 37.455
  00:51:19 STEP 1328 H=1328:00 +CPU= 42.299
  00:52:23 STEP 1329 H=1329:00 +CPU= 61.315
  00:53:09 STEP 1330 H=1330:00 +CPU= 45.572
  00:53:44 STEP 1331 H=1331:00 +CPU= 33.468
  00:54:44 STEP 1332 H=1332:00 +CPU= 58.459
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001332
  00:55:27 STEP 1333 H=1333:00 +CPU= 41.919
  00:56:10 STEP 1334 H=1334:00 +CPU= 41.978
  00:57:06 STEP 1335 H=1335:00 +CPU= 55.063
  00:57:47 STEP 1336 H=1336:00 +CPU= 40.553
  00:58:27 STEP 1337 H=1337:00 +CPU= 38.873
  00:59:27 STEP 1338 H=1338:00 +CPU= 57.999
  01:00:10 STEP 1339 H=1339:00 +CPU= 42.233
  01:01:02 STEP 1340 H=1340:00 +CPU= 44.264
  01:01:46 STEP 1341 H=1341:00 +CPU= 43.117
  01:02:02 STEP 1342 H=1342:00 +CPU= 16.032
  01:02:17 STEP 1343 H=1343:00 +CPU= 14.407
  01:02:43 STEP 1344 H=1344:00 +CPU= 26.555
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_55.zip
Uploading the intermediate file: upload_file_55.zip
  01:02:58 STEP 1345 H=1345:00 +CPU= 15.062
  01:03:17 STEP 1346 H=1346:00 +CPU= 18.492
  01:03:45 STEP 1347 H=1347:00 +CPU= 27.715
  01:04:03 STEP 1348 H=1348:00 +CPU= 18.252
  01:04:21 STEP 1349 H=1349:00 +CPU= 17.761
  01:04:50 STEP 1350 H=1350:00 +CPU= 29.024
  01:05:08 STEP 1351 H=1351:00 +CPU= 18.409
  01:05:29 STEP 1352 H=1352:00 +CPU= 20.226
  01:05:56 STEP 1353 H=1353:00 +CPU= 27.347
  01:06:10 STEP 1354 H=1354:00 +CPU= 13.929
  01:06:24 STEP 1355 H=1355:00 +CPU= 13.488
  01:06:48 STEP 1356 H=1356:00 +CPU= 24.463
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001356
  01:07:09 STEP 1357 H=1357:00 +CPU= 20.941
  01:07:29 STEP 1358 H=1358:00 +CPU= 19.360
  01:07:51 STEP 1359 H=1359:00 +CPU= 22.593
  01:08:06 STEP 1360 H=1360:00 +CPU= 14.641
  01:08:20 STEP 1361 H=1361:00 +CPU= 13.835
  01:08:45 STEP 1362 H=1362:00 +CPU= 25.052
  01:08:59 STEP 1363 H=1363:00 +CPU= 14.215
  01:09:14 STEP 1364 H=1364:00 +CPU= 14.653
  01:09:37 STEP 1365 H=1365:00 +CPU= 22.829
  01:09:51 STEP 1366 H=1366:00 +CPU= 13.923
  01:10:05 STEP 1367 H=1367:00 +CPU= 13.851
  01:10:30 STEP 1368 H=1368:00 +CPU= 25.553
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_56.zip
Uploading the intermediate file: upload_file_56.zip
  01:10:52 STEP 1369 H=1369:00 +CPU= 21.953
  01:11:09 STEP 1370 H=1370:00 +CPU= 16.953
  01:11:32 STEP 1371 H=1371:00 +CPU= 22.536
  01:11:49 STEP 1372 H=1372:00 +CPU= 17.166
  01:12:08 STEP 1373 H=1373:00 +CPU= 18.912
  01:12:37 STEP 1374 H=1374:00 +CPU= 28.168
  01:12:54 STEP 1375 H=1375:00 +CPU= 17.573
  01:13:14 STEP 1376 H=1376:00 +CPU= 19.428
  01:13:46 STEP 1377 H=1377:00 +CPU= 32.154
  01:14:01 STEP 1378 H=1378:00 +CPU= 14.992
  01:14:14 STEP 1379 H=1379:00 +CPU= 13.379
  01:14:38 STEP 1380 H=1380:00 +CPU= 23.792
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001380
  01:14:52 STEP 1381 H=1381:00 +CPU= 13.633
  01:15:10 STEP 1382 H=1382:00 +CPU= 18.300
  01:15:35 STEP 1383 H=1383:00 +CPU= 24.595
  01:15:50 STEP 1384 H=1384:00 +CPU= 15.201
  01:16:05 STEP 1385 H=1385:00 +CPU= 14.511
  01:16:29 STEP 1386 H=1386:00 +CPU= 24.657
  01:16:47 STEP 1387 H=1387:00 +CPU= 17.455
  01:17:05 STEP 1388 H=1388:00 +CPU= 18.226
  01:17:33 STEP 1389 H=1389:00 +CPU= 27.320
  01:17:53 STEP 1390 H=1390:00 +CPU= 19.685
  01:18:07 STEP 1391 H=1391:00 +CPU= 14.295
  01:18:34 STEP 1392 H=1392:00 +CPU= 27.450
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_57.zip
Uploading the intermediate file: upload_file_57.zip
  01:18:49 STEP 1393 H=1393:00 +CPU= 14.770
  01:19:04 STEP 1394 H=1394:00 +CPU= 14.918
Uploading trickle at timestep: 5011200
  01:19:28 STEP 1395 H=1395:00 +CPU= 23.504
  01:19:44 STEP 1396 H=1396:00 +CPU= 16.084
  01:19:58 STEP 1397 H=1397:00 +CPU= 14.080
  01:20:21 STEP 1398 H=1398:00 +CPU= 22.661
  01:20:35 STEP 1399 H=1399:00 +CPU= 14.643
  01:20:51 STEP 1400 H=1400:00 +CPU= 15.072
  01:21:14 STEP 1401 H=1401:00 +CPU= 23.340
  01:21:30 STEP 1402 H=1402:00 +CPU= 15.878
  01:21:45 STEP 1403 H=1403:00 +CPU= 15.356
  01:22:09 STEP 1404 H=1404:00 +CPU= 23.708
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001404
  01:22:22 STEP 1405 H=1405:00 +CPU= 13.440
  01:22:36 STEP 1406 H=1406:00 +CPU= 13.842
  01:23:02 STEP 1407 H=1407:00 +CPU= 25.869
  01:23:30 STEP 1408 H=1408:00 +CPU= 27.824
  01:24:05 STEP 1409 H=1409:00 +CPU= 33.769
  01:25:00 STEP 1410 H=1410:00 +CPU= 51.669
  01:25:32 STEP 1411 H=1411:00 +CPU= 31.277
  01:25:55 STEP 1412 H=1412:00 +CPU= 22.928
  01:26:19 STEP 1413 H=1413:00 +CPU= 23.828
  01:26:59 STEP 1414 H=1414:00 +CPU= 38.010
  01:27:41 STEP 1415 H=1415:00 +CPU= 37.605
  01:28:48 STEP 1416 H=1416:00 +CPU= 63.098
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_58.zip
Uploading the intermediate file: upload_file_58.zip
  01:29:18 STEP 1417 H=1417:00 +CPU= 29.693
  01:29:44 STEP 1418 H=1418:00 +CPU= 25.690
  01:30:21 STEP 1419 H=1419:00 +CPU= 36.774
  01:30:48 STEP 1420 H=1420:00 +CPU= 26.426
  01:31:09 STEP 1421 H=1421:00 +CPU= 21.188
  01:31:37 STEP 1422 H=1422:00 +CPU= 27.745
  01:31:52 STEP 1423 H=1423:00 +CPU= 15.164
  01:32:10 STEP 1424 H=1424:00 +CPU= 17.686
  01:32:37 STEP 1425 H=1425:00 +CPU= 26.416
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0405
(argv4) batchid: 966
(argv5) wuid: 12183049
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_840558d09c5d5c7f0120126d143bc8ec to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0405_1997050100_123_966_12183049.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0405_1997050100_123_966_12183049.zip
ic_ancil_file: ic_ancil_12183049
ifsdata_file: ifsdata_12183049
climate_data_file: clim_data_12183049
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_24c79f97dddc016a1ac687392d4e955d to: /var/lib/boinc-client/slots/6/ic_ancil_12183049.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12183049.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_76326dc65c4cd56ec61d218ec8d726b9 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12183049.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12183049.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12183049.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12183049.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12183049.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12183049.xml
-- Model is restarting --
Adjusting last_iter, 1426, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12183049.xml
last_cpu_time: 41761
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668
The child process has been launched with process id: 456047
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:456047:456047] [20230105:013350:1672911230:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:34:51 STEP 1416 H=1416:00 +CPU= 54.124
  01:35:17 STEP 1417 H=1417:00 +CPU= 25.940
  01:35:49 STEP 1418 H=1418:00 +CPU= 31.167
  01:36:49 STEP 1419 H=1419:00 +CPU= 54.564
  01:37:26 STEP 1420 H=1420:00 +CPU= 34.148
  01:37:49 STEP 1421 H=1421:00 +CPU= 22.909
  01:38:15 STEP 1422 H=1422:00 +CPU= 26.353
  01:38:36 STEP 1423 H=1423:00 +CPU= 21.332
  01:38:58 STEP 1424 H=1424:00 +CPU= 21.543
  01:39:28 STEP 1425 H=1425:00 +CPU= 30.150
  01:39:46 STEP 1426 H=1426:00 +CPU= 17.253
  01:40:04 STEP 1427 H=1427:00 +CPU= 18.139
  01:40:32 STEP 1428 H=1428:00 +CPU= 28.034
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001428
  01:41:08 STEP 1429 H=1429:00 +CPU= 35.490
  01:41:43 STEP 1430 H=1430:00 +CPU= 33.253
  01:42:33 STEP 1431 H=1431:00 +CPU= 47.595
  01:42:51 STEP 1432 H=1432:00 +CPU= 17.862
  01:43:11 STEP 1433 H=1433:00 +CPU= 20.094
  01:43:39 STEP 1434 H=1434:00 +CPU= 27.426
  01:43:52 STEP 1435 H=1435:00 +CPU= 13.684
  01:44:06 STEP 1436 H=1436:00 +CPU= 14.041
  01:44:28 STEP 1437 H=1437:00 +CPU= 21.718
  01:44:43 STEP 1438 H=1438:00 +CPU= 14.606
  01:45:01 STEP 1439 H=1439:00 +CPU= 17.657
  01:45:44 STEP 1440 H=1440:00 +CPU= 42.931
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_59.zip
Uploading the intermediate file: upload_file_59.zip
  01:46:24 STEP 1441 H=1441:00 +CPU= 37.839
  01:47:04 STEP 1442 H=1442:00 +CPU= 35.461
  01:47:43 STEP 1443 H=1443:00 +CPU= 39.543
  01:48:01 STEP 1444 H=1444:00 +CPU= 17.277
  01:48:23 STEP 1445 H=1445:00 +CPU= 21.929
  01:48:52 STEP 1446 H=1446:00 +CPU= 29.520
  01:49:11 STEP 1447 H=1447:00 +CPU= 18.080
  01:49:28 STEP 1448 H=1448:00 +CPU= 17.812
  01:49:51 STEP 1449 H=1449:00 +CPU= 22.634
  01:50:06 STEP 1450 H=1450:00 +CPU= 14.457
  01:50:20 STEP 1451 H=1451:00 +CPU= 13.998
  01:50:48 STEP 1452 H=1452:00 +CPU= 28.448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001452
  01:51:08 STEP 1453 H=1453:00 +CPU= 20.015
  01:51:26 STEP 1454 H=1454:00 +CPU= 17.729
  01:51:49 STEP 1455 H=1455:00 +CPU= 22.500
  01:52:03 STEP 1456 H=1456:00 +CPU= 14.409
  01:52:17 STEP 1457 H=1457:00 +CPU= 13.850
  01:52:56 STEP 1458 H=1458:00 +CPU= 37.779
  01:53:36 STEP 1459 H=1459:00 +CPU= 38.450
  01:54:13 STEP 1460 H=1460:00 +CPU= 34.453
  01:54:53 STEP 1461 H=1461:00 +CPU= 38.998
  01:55:12 STEP 1462 H=1462:00 +CPU= 18.959
  01:55:31 STEP 1463 H=1463:00 +CPU= 19.235
  01:56:04 STEP 1464 H=1464:00 +CPU= 32.266
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_60.zip
Uploading the intermediate file: upload_file_60.zip
  01:56:18 STEP 1465 H=1465:00 +CPU= 14.359
  01:56:34 STEP 1466 H=1466:00 +CPU= 16.212
  01:57:07 STEP 1467 H=1467:00 +CPU= 32.422
  01:57:30 STEP 1468 H=1468:00 +CPU= 22.442
  01:57:47 STEP 1469 H=1469:00 +CPU= 16.951
  01:58:09 STEP 1470 H=1470:00 +CPU= 22.065
  01:58:23 STEP 1471 H=1471:00 +CPU= 13.733
  01:58:37 STEP 1472 H=1472:00 +CPU= 13.963
  01:58:59 STEP 1473 H=1473:00 +CPU= 22.371
  01:59:14 STEP 1474 H=1474:00 +CPU= 14.635
  01:59:28 STEP 1475 H=1475:00 +CPU= 14.138
  01:59:58 STEP 1476 H=1476:00 +CPU= 29.826
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001476
  02:00:20 STEP 1477 H=1477:00 +CPU= 21.730
  02:00:41 STEP 1478 H=1478:00 +CPU= 21.044
  02:01:03 STEP 1479 H=1479:00 +CPU= 21.944
  02:01:18 STEP 1480 H=1480:00 +CPU= 14.454
  02:01:31 STEP 1481 H=1481:00 +CPU= 13.875
  02:01:55 STEP 1482 H=1482:00 +CPU= 23.324
  02:02:26 STEP 1483 H=1483:00 +CPU= 30.185
  02:03:04 STEP 1484 H=1484:00 +CPU= 37.639
  02:03:57 STEP 1485 H=1485:00 +CPU= 49.148
  02:04:22 STEP 1486 H=1486:00 +CPU= 25.159
  02:04:46 STEP 1487 H=1487:00 +CPU= 23.869
  02:05:18 STEP 1488 H=1488:00 +CPU= 31.519
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_61.zip
Uploading the intermediate file: upload_file_61.zip
  02:05:33 STEP 1489 H=1489:00 +CPU= 15.597
  02:05:50 STEP 1490 H=1490:00 +CPU= 16.853
  02:06:16 STEP 1491 H=1491:00 +CPU= 26.075
  02:06:32 STEP 1492 H=1492:00 +CPU= 15.705
  02:06:50 STEP 1493 H=1493:00 +CPU= 17.615
  02:07:17 STEP 1494 H=1494:00 +CPU= 27.439
  02:07:33 STEP 1495 H=1495:00 +CPU= 15.640
  02:07:48 STEP 1496 H=1496:00 +CPU= 15.294
  02:08:12 STEP 1497 H=1497:00 +CPU= 23.218
  02:08:27 STEP 1498 H=1498:00 +CPU= 15.713
  02:08:45 STEP 1499 H=1499:00 +CPU= 17.683
  02:09:14 STEP 1500 H=1500:00 +CPU= 28.794
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001500
  02:09:35 STEP 1501 H=1501:00 +CPU= 20.431
  02:09:54 STEP 1502 H=1502:00 +CPU= 19.769
  02:10:26 STEP 1503 H=1503:00 +CPU= 31.286
  02:10:43 STEP 1504 H=1504:00 +CPU= 17.150
  02:11:01 STEP 1505 H=1505:00 +CPU= 17.818
  02:11:28 STEP 1506 H=1506:00 +CPU= 27.311
  02:11:43 STEP 1507 H=1507:00 +CPU= 14.904
  02:11:59 STEP 1508 H=1508:00 +CPU= 15.237
  02:12:22 STEP 1509 H=1509:00 +CPU= 23.138
  02:12:40 STEP 1510 H=1510:00 +CPU= 17.972
  02:12:58 STEP 1511 H=1511:00 +CPU= 18.074
  02:13:30 STEP 1512 H=1512:00 +CPU= 31.605
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_62.zip
Uploading the intermediate file: upload_file_62.zip
  02:13:58 STEP 1513 H=1513:00 +CPU= 27.418
  02:14:40 STEP 1514 H=1514:00 +CPU= 41.256
  02:15:38 STEP 1515 H=1515:00 +CPU= 53.161
  02:16:06 STEP 1516 H=1516:00 +CPU= 27.497
  02:16:26 STEP 1517 H=1517:00 +CPU= 19.789
  02:16:53 STEP 1518 H=1518:00 +CPU= 26.992
  02:17:11 STEP 1519 H=1519:00 +CPU= 18.389
  02:17:31 STEP 1520 H=1520:00 +CPU= 19.900
  02:17:56 STEP 1521 H=1521:00 +CPU= 24.222
  02:18:14 STEP 1522 H=1522:00 +CPU= 17.834
  02:18:31 STEP 1523 H=1523:00 +CPU= 17.470
  02:19:13 STEP 1524 H=1524:00 +CPU= 41.962
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001524
  02:19:50 STEP 1525 H=1525:00 +CPU= 35.984
  02:20:32 STEP 1526 H=1526:00 +CPU= 37.475
  02:21:15 STEP 1527 H=1527:00 +CPU= 43.218
  02:21:34 STEP 1528 H=1528:00 +CPU= 18.743
  02:21:48 STEP 1529 H=1529:00 +CPU= 13.550
  02:22:10 STEP 1530 H=1530:00 +CPU= 21.975
  02:22:23 STEP 1531 H=1531:00 +CPU= 13.457
  02:22:39 STEP 1532 H=1532:00 +CPU= 15.581
  02:23:05 STEP 1533 H=1533:00 +CPU= 26.280
  02:23:19 STEP 1534 H=1534:00 +CPU= 13.886
  02:23:33 STEP 1535 H=1535:00 +CPU= 13.817
  02:23:58 STEP 1536 H=1536:00 +CPU= 25.275
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_63.zip
Uploading the intermediate file: upload_file_63.zip
  02:24:13 STEP 1537 H=1537:00 +CPU= 15.304
  02:24:31 STEP 1538 H=1538:00 +CPU= 17.637
  02:24:58 STEP 1539 H=1539:00 +CPU= 26.736
  02:25:16 STEP 1540 H=1540:00 +CPU= 17.831
  02:25:33 STEP 1541 H=1541:00 +CPU= 17.196
  02:26:01 STEP 1542 H=1542:00 +CPU= 28.050
  02:26:18 STEP 1543 H=1543:00 +CPU= 16.531
  02:26:33 STEP 1544 H=1544:00 +CPU= 15.071
  02:26:56 STEP 1545 H=1545:00 +CPU= 23.202
  02:27:12 STEP 1546 H=1546:00 +CPU= 15.296
  02:27:29 STEP 1547 H=1547:00 +CPU= 17.714
  02:28:24 STEP 1548 H=1548:00 +CPU= 53.544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001548
  02:29:08 STEP 1549 H=1549:00 +CPU= 38.856
  02:29:43 STEP 1550 H=1550:00 +CPU= 32.402
  02:30:14 STEP 1551 H=1551:00 +CPU= 31.452
  02:30:29 STEP 1552 H=1552:00 +CPU= 14.597
  02:30:45 STEP 1553 H=1553:00 +CPU= 15.587
  02:31:12 STEP 1554 H=1554:00 +CPU= 27.169
  02:31:26 STEP 1555 H=1555:00 +CPU= 13.693
  02:31:57 STEP 1556 H=1556:00 +CPU= 31.007
  02:33:02 STEP 1557 H=1557:00 +CPU= 57.324
  02:33:35 STEP 1558 H=1558:00 +CPU= 31.517
  02:33:55 STEP 1559 H=1559:00 +CPU= 20.103
  02:34:21 STEP 1560 H=1560:00 +CPU= 26.183
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_64.zip
Uploading the intermediate file: upload_file_64.zip
  02:34:36 STEP 1561 H=1561:00 +CPU= 14.360
  02:34:52 STEP 1562 H=1562:00 +CPU= 16.075
  02:35:17 STEP 1563 H=1563:00 +CPU= 25.056
  02:35:31 STEP 1564 H=1564:00 +CPU= 14.140
  02:35:45 STEP 1565 H=1565:00 +CPU= 13.745
  02:36:07 STEP 1566 H=1566:00 +CPU= 22.141
  02:36:21 STEP 1567 H=1567:00 +CPU= 13.784
  02:36:35 STEP 1568 H=1568:00 +CPU= 14.139
  02:36:57 STEP 1569 H=1569:00 +CPU= 21.838
  02:37:27 STEP 1570 H=1570:00 +CPU= 29.257
  02:38:03 STEP 1571 H=1571:00 +CPU= 35.056
  02:38:58 STEP 1572 H=1572:00 +CPU= 49.784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001572
  02:39:16 STEP 1573 H=1573:00 +CPU= 17.849
  02:39:31 STEP 1574 H=1574:00 +CPU= 14.587
  02:39:53 STEP 1575 H=1575:00 +CPU= 22.250
  02:40:07 STEP 1576 H=1576:00 +CPU= 14.210
  02:40:36 STEP 1577 H=1577:00 +CPU= 27.934
  02:41:36 STEP 1578 H=1578:00 +CPU= 56.295
  02:42:11 STEP 1579 H=1579:00 +CPU= 34.372
  02:42:36 STEP 1580 H=1580:00 +CPU= 25.330
  02:43:08 STEP 1581 H=1581:00 +CPU= 31.572
  02:43:23 STEP 1582 H=1582:00 +CPU= 14.672
  02:43:37 STEP 1583 H=1583:00 +CPU= 14.524
  02:44:03 STEP 1584 H=1584:00 +CPU= 26.020
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_65.zip
Uploading the intermediate file: upload_file_65.zip
  02:44:17 STEP 1585 H=1585:00 +CPU= 13.811
  02:44:47 STEP 1586 H=1586:00 +CPU= 29.410
  02:45:47 STEP 1587 H=1587:00 +CPU= 55.382
  02:46:26 STEP 1588 H=1588:00 +CPU= 35.822
  02:46:49 STEP 1589 H=1589:00 +CPU= 22.048
  02:47:14 STEP 1590 H=1590:00 +CPU= 25.231
  02:47:31 STEP 1591 H=1591:00 +CPU= 17.160
  02:47:50 STEP 1592 H=1592:00 +CPU= 18.360
  02:48:27 STEP 1593 H=1593:00 +CPU= 36.599
  02:49:05 STEP 1594 H=1594:00 +CPU= 37.452
  02:49:44 STEP 1595 H=1595:00 +CPU= 34.856
  02:50:31 STEP 1596 H=1596:00 +CPU= 47.058
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001596
  02:50:52 STEP 1597 H=1597:00 +CPU= 21.114
  02:51:11 STEP 1598 H=1598:00 +CPU= 18.129
  02:51:35 STEP 1599 H=1599:00 +CPU= 24.439
  02:51:51 STEP 1600 H=1600:00 +CPU= 16.216
  02:52:06 STEP 1601 H=1601:00 +CPU= 14.211
  02:52:28 STEP 1602 H=1602:00 +CPU= 22.659
  02:52:43 STEP 1603 H=1603:00 +CPU= 14.050
  02:52:57 STEP 1604 H=1604:00 +CPU= 14.279
  02:53:20 STEP 1605 H=1605:00 +CPU= 23.527
  02:53:36 STEP 1606 H=1606:00 +CPU= 15.829
  02:53:52 STEP 1607 H=1607:00 +CPU= 15.599
  02:54:34 STEP 1608 H=1608:00 +CPU= 41.988
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_66.zip
Uploading the intermediate file: upload_file_66.zip
  02:55:13 STEP 1609 H=1609:00 +CPU= 36.693
  02:55:54 STEP 1610 H=1610:00 +CPU= 35.353
  02:56:29 STEP 1611 H=1611:00 +CPU= 34.667
  02:56:45 STEP 1612 H=1612:00 +CPU= 15.830
  02:57:02 STEP 1613 H=1613:00 +CPU= 17.175
  02:57:34 STEP 1614 H=1614:00 +CPU= 31.452
  02:57:58 STEP 1615 H=1615:00 +CPU= 24.392
  02:58:17 STEP 1616 H=1616:00 +CPU= 19.256
  02:58:45 STEP 1617 H=1617:00 +CPU= 27.049
  02:59:02 STEP 1618 H=1618:00 +CPU= 16.970
  02:59:24 STEP 1619 H=1619:00 +CPU= 22.108
  02:59:54 STEP 1620 H=1620:00 +CPU= 30.581
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001620
  03:00:09 STEP 1621 H=1621:00 +CPU= 14.336
  03:00:23 STEP 1622 H=1622:00 +CPU= 14.522
  03:00:46 STEP 1623 H=1623:00 +CPU= 22.684
  03:01:08 STEP 1624 H=1624:00 +CPU= 22.066
  03:01:29 STEP 1625 H=1625:00 +CPU= 20.834
  03:01:59 STEP 1626 H=1626:00 +CPU= 30.161
  03:02:22 STEP 1627 H=1627:00 +CPU= 22.505
  03:02:45 STEP 1628 H=1628:00 +CPU= 22.992
  03:03:15 STEP 1629 H=1629:00 +CPU= 29.374
  03:03:30 STEP 1630 H=1630:00 +CPU= 15.127
  03:03:45 STEP 1631 H=1631:00 +CPU= 15.169
  03:04:14 STEP 1632 H=1632:00 +CPU= 28.981
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_67.zip
Uploading the intermediate file: upload_file_67.zip
  03:04:34 STEP 1633 H=1633:00 +CPU= 19.806
  03:04:52 STEP 1634 H=1634:00 +CPU= 18.308
  03:05:19 STEP 1635 H=1635:00 +CPU= 26.891
  03:05:39 STEP 1636 H=1636:00 +CPU= 19.396
  03:06:01 STEP 1637 H=1637:00 +CPU= 21.764
  03:06:32 STEP 1638 H=1638:00 +CPU= 30.897
  03:06:52 STEP 1639 H=1639:00 +CPU= 19.922
  03:07:11 STEP 1640 H=1640:00 +CPU= 19.595
  03:07:39 STEP 1641 H=1641:00 +CPU= 27.782
  03:07:58 STEP 1642 H=1642:00 +CPU= 18.282
  03:08:16 STEP 1643 H=1643:00 +CPU= 18.186
  03:08:48 STEP 1644 H=1644:00 +CPU= 32.154
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001644
  03:09:05 STEP 1645 H=1645:00 +CPU= 16.916
  03:09:23 STEP 1646 H=1646:00 +CPU= 18.123
  03:09:55 STEP 1647 H=1647:00 +CPU= 32.037
  03:10:11 STEP 1648 H=1648:00 +CPU= 15.137
  03:10:24 STEP 1649 H=1649:00 +CPU= 13.316
  03:10:46 STEP 1650 H=1650:00 +CPU= 21.994
  03:10:59 STEP 1651 H=1651:00 +CPU= 13.524
  03:11:17 STEP 1652 H=1652:00 +CPU= 17.557
  03:11:44 STEP 1653 H=1653:00 +CPU= 26.508
  03:12:02 STEP 1654 H=1654:00 +CPU= 17.772
  03:12:19 STEP 1655 H=1655:00 +CPU= 17.830
  03:12:51 STEP 1656 H=1656:00 +CPU= 31.213
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_68.zip
Uploading the intermediate file: upload_file_68.zip
  03:13:05 STEP 1657 H=1657:00 +CPU= 14.117
  03:13:19 STEP 1658 H=1658:00 +CPU= 14.491
Uploading trickle at timestep: 5961600
  03:13:45 STEP 1659 H=1659:00 +CPU= 25.465
  03:14:07 STEP 1660 H=1660:00 +CPU= 21.952
  03:14:28 STEP 1661 H=1661:00 +CPU= 21.030
  03:15:11 STEP 1662 H=1662:00 +CPU= 42.862
  03:15:49 STEP 1663 H=1663:00 +CPU= 37.085
  03:16:34 STEP 1664 H=1664:00 +CPU= 38.921
  03:17:21 STEP 1665 H=1665:00 +CPU= 45.853
  03:17:41 STEP 1666 H=1666:00 +CPU= 20.352
  03:17:58 STEP 1667 H=1667:00 +CPU= 16.849
  03:18:29 STEP 1668 H=1668:00 +CPU= 30.756
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001668
  03:18:50 STEP 1669 H=1669:00 +CPU= 21.397
  03:19:05 STEP 1670 H=1670:00 +CPU= 15.045
  03:19:32 STEP 1671 H=1671:00 +CPU= 26.489
  03:19:50 STEP 1672 H=1672:00 +CPU= 17.907
  03:20:09 STEP 1673 H=1673:00 +CPU= 18.676
  03:20:35 STEP 1674 H=1674:00 +CPU= 26.399
  03:20:56 STEP 1675 H=1675:00 +CPU= 21.068
  03:21:17 STEP 1676 H=1676:00 +CPU= 20.569
  03:21:53 STEP 1677 H=1677:00 +CPU= 35.756
  03:22:16 STEP 1678 H=1678:00 +CPU= 23.374
  03:22:40 STEP 1679 H=1679:00 +CPU= 23.199
  03:23:26 STEP 1680 H=1680:00 +CPU= 45.824
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_69.zip
Uploading the intermediate file: upload_file_69.zip
  03:23:48 STEP 1681 H=1681:00 +CPU= 22.157
  03:24:05 STEP 1682 H=1682:00 +CPU= 17.422
  03:24:32 STEP 1683 H=1683:00 +CPU= 25.970
  03:24:53 STEP 1684 H=1684:00 +CPU= 21.838
  03:25:20 STEP 1685 H=1685:00 +CPU= 26.384
  03:25:54 STEP 1686 H=1686:00 +CPU= 33.777
  03:26:14 STEP 1687 H=1687:00 +CPU= 20.119
  03:26:36 STEP 1688 H=1688:00 +CPU= 22.188
  03:27:04 STEP 1689 H=1689:00 +CPU= 27.432
  03:27:19 STEP 1690 H=1690:00 +CPU= 15.163
  03:27:34 STEP 1691 H=1691:00 +CPU= 14.620
  03:27:59 STEP 1692 H=1692:00 +CPU= 25.160
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001692
  03:28:13 STEP 1693 H=1693:00 +CPU= 13.590
  03:28:27 STEP 1694 H=1694:00 +CPU= 13.922
  03:28:48 STEP 1695 H=1695:00 +CPU= 21.658
  03:29:05 STEP 1696 H=1696:00 +CPU= 16.473
  03:29:26 STEP 1697 H=1697:00 +CPU= 20.594
  03:29:51 STEP 1698 H=1698:00 +CPU= 25.051
  03:30:08 STEP 1699 H=1699:00 +CPU= 17.067
  03:30:25 STEP 1700 H=1700:00 +CPU= 17.648
  03:30:53 STEP 1701 H=1701:00 +CPU= 27.310
  03:31:11 STEP 1702 H=1702:00 +CPU= 18.309
  03:31:29 STEP 1703 H=1703:00 +CPU= 17.246
  03:31:56 STEP 1704 H=1704:00 +CPU= 27.169
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_70.zip
Uploading the intermediate file: upload_file_70.zip
  03:32:11 STEP 1705 H=1705:00 +CPU= 14.793
  03:32:26 STEP 1706 H=1706:00 +CPU= 15.143
  03:32:50 STEP 1707 H=1707:00 +CPU= 24.143
  03:33:04 STEP 1708 H=1708:00 +CPU= 13.946
  03:33:18 STEP 1709 H=1709:00 +CPU= 13.539
  03:33:39 STEP 1710 H=1710:00 +CPU= 21.888
  03:33:55 STEP 1711 H=1711:00 +CPU= 15.222
  03:34:13 STEP 1712 H=1712:00 +CPU= 18.439
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0405
(argv4) batchid: 966
(argv5) wuid: 12183049
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_840558d09c5d5c7f0120126d143bc8ec to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0405_1997050100_123_966_12183049.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0405_1997050100_123_966_12183049.zip
ic_ancil_file: ic_ancil_12183049
ifsdata_file: ifsdata_12183049
climate_data_file: clim_data_12183049
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_24c79f97dddc016a1ac687392d4e955d to: /var/lib/boinc-client/slots/6/ic_ancil_12183049.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12183049.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_76326dc65c4cd56ec61d218ec8d726b9 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12183049.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12183049.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12183049.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12183049.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12183049.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12183049.xml
-- Model is restarting --
Adjusting last_iter, 1713, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12183049.xml
last_cpu_time: 48858
upload_file_number: 71
last_iter: 1703
last_upload: 6134400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668
The child process has been launched with process id: 473754
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Valeri2:1:1:473754:473754] [20230105:033526:1672918526:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:36:36 STEP 1704 H=1704:00 +CPU= 59.800
  03:36:57 STEP 1705 H=1705:00 +CPU= 21.212
  03:37:12 STEP 1706 H=1706:00 +CPU= 14.759
  03:37:35 STEP 1707 H=1707:00 +CPU= 22.548
  03:37:50 STEP 1708 H=1708:00 +CPU= 14.855
  03:38:03 STEP 1709 H=1709:00 +CPU= 13.605
  03:38:25 STEP 1710 H=1710:00 +CPU= 21.921
  03:38:39 STEP 1711 H=1711:00 +CPU= 13.524
  03:38:56 STEP 1712 H=1712:00 +CPU= 16.680
  03:39:47 STEP 1713 H=1713:00 +CPU= 50.746
  03:40:32 STEP 1714 H=1714:00 +CPU= 38.629
  03:41:04 STEP 1715 H=1715:00 +CPU= 30.496
  03:41:45 STEP 1716 H=1716:00 +CPU= 41.556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001716
  03:42:05 STEP 1717 H=1717:00 +CPU= 19.733
  03:42:23 STEP 1718 H=1718:00 +CPU= 17.593
  03:42:54 STEP 1719 H=1719:00 +CPU= 31.551
  03:43:16 STEP 1720 H=1720:00 +CPU= 21.628
  03:43:36 STEP 1721 H=1721:00 +CPU= 19.374
  03:44:01 STEP 1722 H=1722:00 +CPU= 25.736
  03:44:16 STEP 1723 H=1723:00 +CPU= 14.637
  03:44:31 STEP 1724 H=1724:00 +CPU= 15.214
  03:44:55 STEP 1725 H=1725:00 +CPU= 23.531
  03:45:13 STEP 1726 H=1726:00 +CPU= 18.346
  03:45:32 STEP 1727 H=1727:00 +CPU= 18.852
  03:46:05 STEP 1728 H=1728:00 +CPU= 32.420
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183049/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0405_1997050100_123_966_12183049_1_r28527668_71.zip
Uploading the intermediate file: upload_file_71.zip
  03:46:26 STEP 1729 H=1729:00 +CPU= 21.222
  03:46:48 STEP 1730 H=1730:00 +CPU= 21.383
  03:47:15 STEP 1731 H=1731:00 +CPU= 26.933
  03:47:37 STEP 1732 H=1732:00 +CPU= 21.826
  03:47:58 STEP 1733 H=1733:00 +CPU= 21.595
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:08:53 1505480 22295505 oifs_43r3_ps_0405_1997050100_123_966_12183049_1 5,961,600 47,596 0.0080
09 May 2023 16:08:49 1505480 22295505 oifs_43r3_ps_0405_1997050100_123_966_12183049_1 5,011,200 40,946 0.0082
09 May 2023 16:08:39 1505480 22295505 oifs_43r3_ps_0405_1997050100_123_966_12183049_1 3,974,400 33,150 0.0083
09 May 2023 16:08:32 1505480 22295505 oifs_43r3_ps_0405_1997050100_123_966_12183049_1 2,937,600 26,169 0.0089
09 May 2023 16:08:27 1505480 22295505 oifs_43r3_ps_0405_1997050100_123_966_12183049_1 2,073,600 20,049 0.0097


©2024 climateprediction.net