climateprediction.net home page
Task 22443595

Task 22443595

Name oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0
Workunit 12290006
Created 12 Jun 2024, 22:23:28 UTC
Sent 14 Jun 2024, 4:19:30 UTC
Report deadline 13 Aug 2024, 4:19:30 UTC
Received 16 Jun 2024, 7:32:45 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1496046
Run time 14 hours 58 min 19 sec
CPU time 14 hours 58 min 19 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 5.56 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,022.97 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,356.74 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
ICMUAh7zg+001428
  20:57:37 STEP 1429 H= 357:15 +CPU= 27.054
  20:58:04 STEP 1430 H= 357:30 +CPU= 27.674
  20:58:32 STEP 1431 H= 357:45 +CPU= 26.971
  20:59:00 STEP 1432 H= 358:00 +CPU= 28.355
  20:59:27 STEP 1433 H= 358:15 +CPU= 27.148
  20:59:55 STEP 1434 H= 358:30 +CPU= 27.280
  21:00:21 STEP 1435 H= 358:45 +CPU= 26.616
  21:00:51 STEP 1436 H= 359:00 +CPU= 29.371
  21:01:18 STEP 1437 H= 359:15 +CPU= 27.486
  21:01:45 STEP 1438 H= 359:30 +CPU= 26.552
  21:02:13 STEP 1439 H= 359:45 +CPU= 27.989
  21:02:51 STEP 1440 H= 360:00 +CPU= 38.056
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001440
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001440
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001440
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001344
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001344
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001344
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001356
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001356
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001356
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001368
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001368
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001368
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001380
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001380
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001380
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001392
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001392
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001392
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001404
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001404
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001404
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001416
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001416
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001416
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001428
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001428
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001428
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0_r2117891975_14.zip
Uploading the intermediate file: upload_file_14.zip
  21:03:19 STEP 1441 H= 360:15 +CPU= 27.439
  21:03:47 STEP 1442 H= 360:30 +CPU= 27.923
  21:04:15 STEP 1443 H= 360:45 +CPU= 27.989
  21:04:44 STEP 1444 H= 361:00 +CPU= 28.533
  21:05:11 STEP 1445 H= 361:15 +CPU= 27.448
  21:05:38 STEP 1446 H= 361:30 +CPU= 27.114
  21:06:06 STEP 1447 H= 361:45 +CPU= 27.932
  21:06:35 STEP 1448 H= 362:00 +CPU= 28.317
  21:07:02 STEP 1449 H= 362:15 +CPU= 27.151
  21:07:29 STEP 1450 H= 362:30 +CPU= 27.462
  21:07:56 STEP 1451 H= 362:45 +CPU= 26.477
  21:08:30 STEP 1452 H= 363:00 +CPU= 33.993
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001452
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001452
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001452
  21:08:58 STEP 1453 H= 363:15 +CPU= 27.783
  21:09:25 STEP 1454 H= 363:30 +CPU= 27.183
  21:09:52 STEP 1455 H= 363:45 +CPU= 27.001
  21:10:20 STEP 1456 H= 364:00 +CPU= 27.870
  21:10:48 STEP 1457 H= 364:15 +CPU= 27.343
  21:11:14 STEP 1458 H= 364:30 +CPU= 25.962
  21:11:40 STEP 1459 H= 364:45 +CPU= 25.944
  21:12:07 STEP 1460 H= 365:00 +CPU= 26.849
  21:12:33 STEP 1461 H= 365:15 +CPU= 26.631
  21:13:01 STEP 1462 H= 365:30 +CPU= 27.904
  21:13:29 STEP 1463 H= 365:45 +CPU= 27.694
  21:14:04 STEP 1464 H= 366:00 +CPU= 34.505
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001464
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001464
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001464
  21:14:32 STEP 1465 H= 366:15 +CPU= 28.032
  21:14:59 STEP 1466 H= 366:30 +CPU= 26.670
  21:15:26 STEP 1467 H= 366:45 +CPU= 27.328
  21:15:54 STEP 1468 H= 367:00 +CPU= 27.511
  21:16:21 STEP 1469 H= 367:15 +CPU= 27.068
  21:16:47 STEP 1470 H= 367:30 +CPU= 26.085
  21:17:14 STEP 1471 H= 367:45 +CPU= 26.472
  21:17:41 STEP 1472 H= 368:00 +CPU= 27.790
  21:18:09 STEP 1473 H= 368:15 +CPU= 27.746
  21:18:35 STEP 1474 H= 368:30 +CPU= 26.095
  21:19:02 STEP 1475 H= 368:45 +CPU= 26.801
  21:19:37 STEP 1476 H= 369:00 +CPU= 34.579
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001476
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001476
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001476
  21:20:03 STEP 1477 H= 369:15 +CPU= 26.088
  21:20:31 STEP 1478 H= 369:30 +CPU= 27.936
  21:20:59 STEP 1479 H= 369:45 +CPU= 27.948
  21:21:27 STEP 1480 H= 370:00 +CPU= 27.833
  21:21:55 STEP 1481 H= 370:15 +CPU= 27.878
  21:22:23 STEP 1482 H= 370:30 +CPU= 28.191
  21:22:52 STEP 1483 H= 370:45 +CPU= 28.215
  21:23:20 STEP 1484 H= 371:00 +CPU= 28.213
  21:23:47 STEP 1485 H= 371:15 +CPU= 26.399
  21:24:14 STEP 1486 H= 371:30 +CPU= 27.260
  21:24:41 STEP 1487 H= 371:45 +CPU= 26.980
  21:25:19 STEP 1488 H= 372:00 +CPU= 38.130
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001488
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001488
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001488
  21:25:47 STEP 1489 H= 372:15 +CPU= 27.301
  21:26:14 STEP 1490 H= 372:30 +CPU= 27.604
  21:26:41 STEP 1491 H= 372:45 +CPU= 27.003
  21:27:09 STEP 1492 H= 373:00 +CPU= 27.542
  21:27:36 STEP 1493 H= 373:15 +CPU= 27.383
  21:28:04 STEP 1494 H= 373:30 +CPU= 27.719
  21:28:31 STEP 1495 H= 373:45 +CPU= 26.766
  21:29:00 STEP 1496 H= 374:00 +CPU= 28.578
  21:29:27 STEP 1497 H= 374:15 +CPU= 27.141
  21:29:54 STEP 1498 H= 374:30 +CPU= 26.416
  21:30:21 STEP 1499 H= 374:45 +CPU= 26.976
  21:30:56 STEP 1500 H= 375:00 +CPU= 35.321
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001500
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001500
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001500
  21:31:23 STEP 1501 H= 375:15 +CPU= 26.901
  21:31:50 STEP 1502 H= 375:30 +CPU= 27.206
  21:32:17 STEP 1503 H= 375:45 +CPU= 26.775
  21:32:45 STEP 1504 H= 376:00 +CPU= 28.321
  21:33:12 STEP 1505 H= 376:15 +CPU= 26.947
  21:33:39 STEP 1506 H= 376:30 +CPU= 26.742
  21:34:07 STEP 1507 H= 376:45 +CPU= 27.899
  21:34:35 STEP 1508 H= 377:00 +CPU= 27.453
  21:35:01 STEP 1509 H= 377:15 +CPU= 26.459
  21:35:29 STEP 1510 H= 377:30 +CPU= 27.681
  21:35:56 STEP 1511 H= 377:45 +CPU= 26.987
  21:36:31 STEP 1512 H= 378:00 +CPU= 34.212
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001512
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001512
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001512
  21:36:58 STEP 1513 H= 378:15 +CPU= 27.236
  21:37:25 STEP 1514 H= 378:30 +CPU= 27.123
  21:37:52 STEP 1515 H= 378:45 +CPU= 26.670
  21:38:19 STEP 1516 H= 379:00 +CPU= 27.535
  21:38:47 STEP 1517 H= 379:15 +CPU= 27.992
  21:39:15 STEP 1518 H= 379:30 +CPU= 27.936
  21:39:43 STEP 1519 H= 379:45 +CPU= 27.057
  21:40:10 STEP 1520 H= 380:00 +CPU= 27.217
  21:40:38 STEP 1521 H= 380:15 +CPU= 27.667
  21:41:04 STEP 1522 H= 380:30 +CPU= 26.539
  21:41:31 STEP 1523 H= 380:45 +CPU= 26.723
  21:42:06 STEP 1524 H= 381:00 +CPU= 34.513
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001524
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001524
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001524
  21:42:33 STEP 1525 H= 381:15 +CPU= 27.775
  21:43:00 STEP 1526 H= 381:30 +CPU= 26.342
  21:43:27 STEP 1527 H= 381:45 +CPU= 27.117
  21:43:55 STEP 1528 H= 382:00 +CPU= 27.834
  21:44:22 STEP 1529 H= 382:15 +CPU= 26.678
  21:44:50 STEP 1530 H= 382:30 +CPU= 27.769
  21:45:17 STEP 1531 H= 382:45 +CPU= 27.330
  21:45:45 STEP 1532 H= 383:00 +CPU= 28.448
  21:46:12 STEP 1533 H= 383:15 +CPU= 26.330
  21:46:40 STEP 1534 H= 383:30 +CPU= 28.196
  21:47:08 STEP 1535 H= 383:45 +CPU= 27.491
  21:47:46 STEP 1536 H= 384:00 +CPU= 38.620
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001536
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001536
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001440
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001440
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001440
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001452
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001452
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001452
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001464
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001464
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001464
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001476
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001476
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001476
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001488
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001488
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001488
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001500
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001500
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001500
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001512
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001512
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001512
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001524
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001524
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001524
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0_r2117891975_15.zip
Uploading the intermediate file: upload_file_15.zip
  21:48:14 STEP 1537 H= 384:15 +CPU= 28.035
  21:48:41 STEP 1538 H= 384:30 +CPU= 26.909
  21:49:09 STEP 1539 H= 384:45 +CPU= 27.323
  21:49:37 STEP 1540 H= 385:00 +CPU= 28.066
  21:50:04 STEP 1541 H= 385:15 +CPU= 26.918
  21:50:32 STEP 1542 H= 385:30 +CPU= 27.828
  21:50:59 STEP 1543 H= 385:45 +CPU= 26.879
  21:51:27 STEP 1544 H= 386:00 +CPU= 27.801
  21:51:55 STEP 1545 H= 386:15 +CPU= 27.471
  21:52:22 STEP 1546 H= 386:30 +CPU= 27.559
  21:52:50 STEP 1547 H= 386:45 +CPU= 27.918
  21:53:26 STEP 1548 H= 387:00 +CPU= 35.204
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001548
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001548
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001548
  21:53:53 STEP 1549 H= 387:15 +CPU= 27.524
  21:54:21 STEP 1550 H= 387:30 +CPU= 27.888
  21:54:49 STEP 1551 H= 387:45 +CPU= 27.995
  21:55:17 STEP 1552 H= 388:00 +CPU= 27.613
  21:55:44 STEP 1553 H= 388:15 +CPU= 27.284
  21:56:11 STEP 1554 H= 388:30 +CPU= 26.747
  21:56:39 STEP 1555 H= 388:45 +CPU= 27.497
  21:57:07 STEP 1556 H= 389:00 +CPU= 27.897
  21:57:34 STEP 1557 H= 389:15 +CPU= 26.810
  21:58:02 STEP 1558 H= 389:30 +CPU= 28.649
  21:58:29 STEP 1559 H= 389:45 +CPU= 26.516
  21:59:04 STEP 1560 H= 390:00 +CPU= 34.536
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001560
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001560
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001560
  21:59:30 STEP 1561 H= 390:15 +CPU= 26.404
  21:59:57 STEP 1562 H= 390:30 +CPU= 26.281
  22:00:25 STEP 1563 H= 390:45 +CPU= 27.885
  22:00:54 STEP 1564 H= 391:00 +CPU= 28.873
  22:01:22 STEP 1565 H= 391:15 +CPU= 27.947
  22:01:49 STEP 1566 H= 391:30 +CPU= 27.487
  22:02:17 STEP 1567 H= 391:45 +CPU= 27.725
  22:02:45 STEP 1568 H= 392:00 +CPU= 27.942
  22:03:12 STEP 1569 H= 392:15 +CPU= 26.479
  22:03:39 STEP 1570 H= 392:30 +CPU= 27.525
  22:04:06 STEP 1571 H= 392:45 +CPU= 26.717
  22:04:40 STEP 1572 H= 393:00 +CPU= 33.866
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001572
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001572
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001572
  22:05:08 STEP 1573 H= 393:15 +CPU= 27.544
  22:05:35 STEP 1574 H= 393:30 +CPU= 27.091
  22:06:02 STEP 1575 H= 393:45 +CPU= 26.916
  22:06:29 STEP 1576 H= 394:00 +CPU= 26.543
  22:06:56 STEP 1577 H= 394:15 +CPU= 27.170
  22:07:23 STEP 1578 H= 394:30 +CPU= 27.364
  22:07:51 STEP 1579 H= 394:45 +CPU= 27.965
  22:08:19 STEP 1580 H= 395:00 +CPU= 27.075
  22:08:46 STEP 1581 H= 395:15 +CPU= 27.372
  22:09:14 STEP 1582 H= 395:30 +CPU= 27.802
  22:09:41 STEP 1583 H= 395:45 +CPU= 27.019
  22:10:20 STEP 1584 H= 396:00 +CPU= 39.346
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001584
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001584
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001584
  22:10:47 STEP 1585 H= 396:15 +CPU= 26.722
  22:11:14 STEP 1586 H= 396:30 +CPU= 26.475
  22:11:41 STEP 1587 H= 396:45 +CPU= 27.395
  22:12:07 STEP 1588 H= 397:00 +CPU= 25.845
  22:12:36 STEP 1589 H= 397:15 +CPU= 28.966
  22:13:04 STEP 1590 H= 397:30 +CPU= 27.121
  22:13:31 STEP 1591 H= 397:45 +CPU= 27.375
  22:14:00 STEP 1592 H= 398:00 +CPU= 29.013
  22:14:28 STEP 1593 H= 398:15 +CPU= 27.563
  22:14:55 STEP 1594 H= 398:30 +CPU= 27.399
  22:15:21 STEP 1595 H= 398:45 +CPU= 26.216
  22:15:55 STEP 1596 H= 399:00 +CPU= 33.746
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001596
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001596
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001596
  22:16:23 STEP 1597 H= 399:15 +CPU= 27.108
  22:16:42 STEP 1598 H= 399:30 +CPU= 19.664
  22:17:02 STEP 1599 H= 399:45 +CPU= 19.630
  22:17:28 STEP 1600 H= 400:00 +CPU= 26.034
  22:17:54 STEP 1601 H= 400:15 +CPU= 25.808
  22:18:19 STEP 1602 H= 400:30 +CPU= 25.210
  22:18:46 STEP 1603 H= 400:45 +CPU= 26.465
  22:19:13 STEP 1604 H= 401:00 +CPU= 27.314
  22:19:40 STEP 1605 H= 401:15 +CPU= 26.493
  22:20:06 STEP 1606 H= 401:30 +CPU= 26.341
  22:20:31 STEP 1607 H= 401:45 +CPU= 24.709
  22:21:05 STEP 1608 H= 402:00 +CPU= 33.686
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001608
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001608
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001608
  22:21:31 STEP 1609 H= 402:15 +CPU= 26.310
  22:21:57 STEP 1610 H= 402:30 +CPU= 26.157
  22:22:23 STEP 1611 H= 402:45 +CPU= 25.595
  22:22:50 STEP 1612 H= 403:00 +CPU= 27.371
  22:23:15 STEP 1613 H= 403:15 +CPU= 24.833
  22:23:41 STEP 1614 H= 403:30 +CPU= 25.990
  22:24:08 STEP 1615 H= 403:45 +CPU= 26.891
  22:24:35 STEP 1616 H= 404:00 +CPU= 26.952
  22:25:02 STEP 1617 H= 404:15 +CPU= 26.664
  22:25:29 STEP 1618 H= 404:30 +CPU= 26.962
  22:25:55 STEP 1619 H= 404:45 +CPU= 25.967
  22:26:30 STEP 1620 H= 405:00 +CPU= 34.439
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001620
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001620
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001620
  22:26:59 STEP 1621 H= 405:15 +CPU= 28.559
  22:27:26 STEP 1622 H= 405:30 +CPU= 27.235
  22:27:53 STEP 1623 H= 405:45 +CPU= 26.984
  22:28:20 STEP 1624 H= 406:00 +CPU= 27.107
  22:28:49 STEP 1625 H= 406:15 +CPU= 27.979
  22:29:15 STEP 1626 H= 406:30 +CPU= 26.843
  22:29:42 STEP 1627 H= 406:45 +CPU= 26.349
  22:30:09 STEP 1628 H= 407:00 +CPU= 26.811
  22:30:35 STEP 1629 H= 407:15 +CPU= 25.692
  22:31:03 STEP 1630 H= 407:30 +CPU= 27.766
  22:31:30 STEP 1631 H= 407:45 +CPU= 26.992
  22:32:08 STEP 1632 H= 408:00 +CPU= 37.704
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001632
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001632
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001620
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001620
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001620
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0_r2117891975_16.zip
Uploading the intermediate file: upload_file_16.zip
  22:32:37 STEP 1633 H= 408:15 +CPU= 28.295
  22:33:06 STEP 1634 H= 408:30 +CPU= 28.396
  22:33:35 STEP 1635 H= 408:45 +CPU= 28.492
  22:34:06 STEP 1636 H= 409:00 +CPU= 29.662
  22:34:35 STEP 1637 H= 409:15 +CPU= 28.506
  22:35:01 STEP 1638 H= 409:30 +CPU= 25.817
  22:35:27 STEP 1639 H= 409:45 +CPU= 25.985
  22:35:56 STEP 1640 H= 410:00 +CPU= 28.786
  22:36:24 STEP 1641 H= 410:15 +CPU= 27.573
  22:36:51 STEP 1642 H= 410:30 +CPU= 26.976
  22:37:19 STEP 1643 H= 410:45 +CPU= 27.530
  22:37:53 STEP 1644 H= 411:00 +CPU= 34.275
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001644
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001644
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001644
  22:38:21 STEP 1645 H= 411:15 +CPU= 27.641
  22:38:49 STEP 1646 H= 411:30 +CPU= 27.383
  22:39:17 STEP 1647 H= 411:45 +CPU= 27.799
  22:39:42 STEP 1648 H= 412:00 +CPU= 25.341
  22:40:03 STEP 1649 H= 412:15 +CPU= 21.157
  22:40:25 STEP 1650 H= 412:30 +CPU= 21.450
  22:40:53 STEP 1651 H= 412:45 +CPU= 27.999
  22:41:22 STEP 1652 H= 413:00 +CPU= 28.943
  22:41:51 STEP 1653 H= 413:15 +CPU= 28.172
  22:42:19 STEP 1654 H= 413:30 +CPU= 27.961
  22:42:46 STEP 1655 H= 413:45 +CPU= 26.982
  22:43:22 STEP 1656 H= 414:00 +CPU= 35.071
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001656
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001656
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001656
  22:43:50 STEP 1657 H= 414:15 +CPU= 27.829
  22:44:17 STEP 1658 H= 414:30 +CPU= 26.945
  22:44:45 STEP 1659 H= 414:45 +CPU= 27.919
  22:45:14 STEP 1660 H= 415:00 +CPU= 28.409
  22:45:41 STEP 1661 H= 415:15 +CPU= 27.440
  22:46:11 STEP 1662 H= 415:30 +CPU= 28.996
  22:46:37 STEP 1663 H= 415:45 +CPU= 26.172
  22:47:06 STEP 1664 H= 416:00 +CPU= 28.830
  22:47:36 STEP 1665 H= 416:15 +CPU= 29.078
  22:48:04 STEP 1666 H= 416:30 +CPU= 27.813
  22:48:32 STEP 1667 H= 416:45 +CPU= 28.184
  22:49:08 STEP 1668 H= 417:00 +CPU= 36.266
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001668
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001668
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001668
  22:49:37 STEP 1669 H= 417:15 +CPU= 28.388
  22:50:05 STEP 1670 H= 417:30 +CPU= 27.707
  22:50:31 STEP 1671 H= 417:45 +CPU= 25.909
  22:50:52 STEP 1672 H= 418:00 +CPU= 21.094
  22:51:12 STEP 1673 H= 418:15 +CPU= 19.924
  22:51:39 STEP 1674 H= 418:30 +CPU= 26.715
  22:52:07 STEP 1675 H= 418:45 +CPU= 27.894
  22:52:35 STEP 1676 H= 419:00 +CPU= 27.902
  22:53:03 STEP 1677 H= 419:15 +CPU= 27.457
  22:53:30 STEP 1678 H= 419:30 +CPU= 26.788
  22:53:57 STEP 1679 H= 419:45 +CPU= 26.977
  22:54:36 STEP 1680 H= 420:00 +CPU= 38.731
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001680
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001680
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001680
  22:55:04 STEP 1681 H= 420:15 +CPU= 27.288
  22:55:31 STEP 1682 H= 420:30 +CPU= 26.973
  22:56:01 STEP 1683 H= 420:45 +CPU= 28.759
  22:56:30 STEP 1684 H= 421:00 +CPU= 28.504
  22:56:59 STEP 1685 H= 421:15 +CPU= 28.447
  22:57:27 STEP 1686 H= 421:30 +CPU= 27.725
  22:57:55 STEP 1687 H= 421:45 +CPU= 27.575
  22:58:25 STEP 1688 H= 422:00 +CPU= 29.062
  22:58:53 STEP 1689 H= 422:15 +CPU= 28.044
  22:59:20 STEP 1690 H= 422:30 +CPU= 26.831
  22:59:49 STEP 1691 H= 422:45 +CPU= 28.778
  23:00:25 STEP 1692 H= 423:00 +CPU= 35.090
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001692
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001692
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001692
  23:00:52 STEP 1693 H= 423:15 +CPU= 27.300
  23:01:22 STEP 1694 H= 423:30 +CPU= 28.665
  23:01:51 STEP 1695 H= 423:45 +CPU= 28.654
  23:02:21 STEP 1696 H= 424:00 +CPU= 29.339
  23:02:50 STEP 1697 H= 424:15 +CPU= 28.321
  23:03:19 STEP 1698 H= 424:30 +CPU= 28.604
  23:03:47 STEP 1699 H= 424:45 +CPU= 28.335
  23:04:17 STEP 1700 H= 425:00 +CPU= 29.289
  23:04:45 STEP 1701 H= 425:15 +CPU= 27.958
  23:05:14 STEP 1702 H= 425:30 +CPU= 28.260
  23:05:42 STEP 1703 H= 425:45 +CPU= 28.042
  23:06:17 STEP 1704 H= 426:00 +CPU= 34.215
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001704
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001704
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001704
  23:06:45 STEP 1705 H= 426:15 +CPU= 28.016
  23:07:13 STEP 1706 H= 426:30 +CPU= 27.704
  23:07:40 STEP 1707 H= 426:45 +CPU= 27.187
  23:08:09 STEP 1708 H= 427:00 +CPU= 28.427
  23:08:38 STEP 1709 H= 427:15 +CPU= 28.639
  23:09:06 STEP 1710 H= 427:30 +CPU= 28.043
  23:09:35 STEP 1711 H= 427:45 +CPU= 28.316
  23:10:04 STEP 1712 H= 428:00 +CPU= 28.930
  23:10:33 STEP 1713 H= 428:15 +CPU= 28.095
  23:11:01 STEP 1714 H= 428:30 +CPU= 28.343
  23:11:29 STEP 1715 H= 428:45 +CPU= 27.958
  23:12:05 STEP 1716 H= 429:00 +CPU= 35.479
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001716
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001716
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001716
  23:12:34 STEP 1717 H= 429:15 +CPU= 28.305
  23:13:02 STEP 1718 H= 429:30 +CPU= 28.192
  23:13:31 STEP 1719 H= 429:45 +CPU= 28.578
  23:13:56 STEP 1720 H= 430:00 +CPU= 24.208
  23:14:18 STEP 1721 H= 430:15 +CPU= 22.290
  23:14:45 STEP 1722 H= 430:30 +CPU= 26.228
  23:15:13 STEP 1723 H= 430:45 +CPU= 28.219
  23:15:40 STEP 1724 H= 431:00 +CPU= 26.729
  23:16:05 STEP 1725 H= 431:15 +CPU= 25.344
  23:16:32 STEP 1726 H= 431:30 +CPU= 26.849
  23:16:59 STEP 1727 H= 431:45 +CPU= 26.612
  23:17:35 STEP 1728 H= 432:00 +CPU= 35.897
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001728
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001728
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001716
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001716
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001716
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0_r2117891975_17.zip
Uploading the intermediate file: upload_file_17.zip
  23:18:00 STEP 1729 H= 432:15 +CPU= 25.563
  23:18:26 STEP 1730 H= 432:30 +CPU= 26.028
  23:18:52 STEP 1731 H= 432:45 +CPU= 25.857
  23:19:19 STEP 1732 H= 433:00 +CPU= 26.578
  23:19:46 STEP 1733 H= 433:15 +CPU= 26.995
  23:20:12 STEP 1734 H= 433:30 +CPU= 25.618
  23:20:38 STEP 1735 H= 433:45 +CPU= 26.507
  23:21:05 STEP 1736 H= 434:00 +CPU= 26.634
  23:21:31 STEP 1737 H= 434:15 +CPU= 25.788
  23:21:58 STEP 1738 H= 434:30 +CPU= 26.879
  23:22:24 STEP 1739 H= 434:45 +CPU= 26.641
  23:22:57 STEP 1740 H= 435:00 +CPU= 32.949
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001740
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001740
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001740
  23:23:24 STEP 1741 H= 435:15 +CPU= 26.566
  23:23:51 STEP 1742 H= 435:30 +CPU= 26.400
  23:24:17 STEP 1743 H= 435:45 +CPU= 26.691
  23:24:45 STEP 1744 H= 436:00 +CPU= 27.615
  23:25:12 STEP 1745 H= 436:15 +CPU= 26.629
  23:25:38 STEP 1746 H= 436:30 +CPU= 26.364
  23:26:05 STEP 1747 H= 436:45 +CPU= 26.741
  23:26:32 STEP 1748 H= 437:00 +CPU= 26.936
  23:26:58 STEP 1749 H= 437:15 +CPU= 26.105
  23:27:25 STEP 1750 H= 437:30 +CPU= 26.426
  23:27:51 STEP 1751 H= 437:45 +CPU= 26.783
  23:28:24 STEP 1752 H= 438:00 +CPU= 32.856
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001752
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001752
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001752
  23:28:52 STEP 1753 H= 438:15 +CPU= 27.202
  23:29:18 STEP 1754 H= 438:30 +CPU= 26.420
  23:29:45 STEP 1755 H= 438:45 +CPU= 26.867
  23:30:12 STEP 1756 H= 439:00 +CPU= 27.201
  23:30:38 STEP 1757 H= 439:15 +CPU= 25.881
  23:31:06 STEP 1758 H= 439:30 +CPU= 27.382
  23:31:33 STEP 1759 H= 439:45 +CPU= 27.220
  23:31:59 STEP 1760 H= 440:00 +CPU= 26.313
  23:32:26 STEP 1761 H= 440:15 +CPU= 26.721
  23:32:52 STEP 1762 H= 440:30 +CPU= 26.137
  23:33:17 STEP 1763 H= 440:45 +CPU= 25.106
  23:33:50 STEP 1764 H= 441:00 +CPU= 32.717
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001764
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001764
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001764
  23:34:17 STEP 1765 H= 441:15 +CPU= 26.418
  23:34:43 STEP 1766 H= 441:30 +CPU= 26.294
  23:35:09 STEP 1767 H= 441:45 +CPU= 25.680
  23:35:35 STEP 1768 H= 442:00 +CPU= 26.280
  23:36:01 STEP 1769 H= 442:15 +CPU= 25.391
  23:36:27 STEP 1770 H= 442:30 +CPU= 26.291
  23:36:51 STEP 1771 H= 442:45 +CPU= 24.450
  23:37:18 STEP 1772 H= 443:00 +CPU= 26.890
  23:37:45 STEP 1773 H= 443:15 +CPU= 26.646
  23:38:13 STEP 1774 H= 443:30 +CPU= 27.719
  23:38:39 STEP 1775 H= 443:45 +CPU= 26.181
  23:39:17 STEP 1776 H= 444:00 +CPU= 37.313
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001776
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001776
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001776
  23:39:42 STEP 1777 H= 444:15 +CPU= 25.667
  23:40:08 STEP 1778 H= 444:30 +CPU= 25.217
  23:40:34 STEP 1779 H= 444:45 +CPU= 26.335
  23:41:00 STEP 1780 H= 445:00 +CPU= 26.041
  23:41:27 STEP 1781 H= 445:15 +CPU= 27.088
  23:41:52 STEP 1782 H= 445:30 +CPU= 25.026
  23:42:18 STEP 1783 H= 445:45 +CPU= 25.795
  23:42:45 STEP 1784 H= 446:00 +CPU= 26.872
  23:43:11 STEP 1785 H= 446:15 +CPU= 25.521
  23:43:37 STEP 1786 H= 446:30 +CPU= 26.469
  23:44:04 STEP 1787 H= 446:45 +CPU= 27.026
  23:44:38 STEP 1788 H= 447:00 +CPU= 33.652
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001788
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001788
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001788
  23:45:04 STEP 1789 H= 447:15 +CPU= 25.516
  23:45:28 STEP 1790 H= 447:30 +CPU= 24.497
  23:45:48 STEP 1791 H= 447:45 +CPU= 19.368
  23:46:08 STEP 1792 H= 448:00 +CPU= 20.451
  23:46:36 STEP 1793 H= 448:15 +CPU= 27.806
  23:47:04 STEP 1794 H= 448:30 +CPU= 27.907
  23:47:32 STEP 1795 H= 448:45 +CPU= 27.746
  23:48:01 STEP 1796 H= 449:00 +CPU= 28.262
  23:48:29 STEP 1797 H= 449:15 +CPU= 27.695
  23:48:57 STEP 1798 H= 449:30 +CPU= 27.837
  23:49:25 STEP 1799 H= 449:45 +CPU= 27.342
  23:50:00 STEP 1800 H= 450:00 +CPU= 34.562
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001800
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001800
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001800
  23:50:28 STEP 1801 H= 450:15 +CPU= 28.429
  23:50:57 STEP 1802 H= 450:30 +CPU= 28.418
  23:51:26 STEP 1803 H= 450:45 +CPU= 28.338
  23:51:56 STEP 1804 H= 451:00 +CPU= 28.971
  23:52:25 STEP 1805 H= 451:15 +CPU= 28.140
  23:52:53 STEP 1806 H= 451:30 +CPU= 27.951
  23:53:22 STEP 1807 H= 451:45 +CPU= 28.278
  23:53:51 STEP 1808 H= 452:00 +CPU= 28.509
  23:54:19 STEP 1809 H= 452:15 +CPU= 27.957
  23:54:47 STEP 1810 H= 452:30 +CPU= 28.054
  23:55:16 STEP 1811 H= 452:45 +CPU= 28.156
  23:55:53 STEP 1812 H= 453:00 +CPU= 36.404
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001812
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001812
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001812
  23:56:22 STEP 1813 H= 453:15 +CPU= 28.045
  23:56:50 STEP 1814 H= 453:30 +CPU= 27.408
  23:57:18 STEP 1815 H= 453:45 +CPU= 28.060
  23:57:45 STEP 1816 H= 454:00 +CPU= 26.989
  23:58:13 STEP 1817 H= 454:15 +CPU= 28.130
  23:58:41 STEP 1818 H= 454:30 +CPU= 27.395
  23:59:08 STEP 1819 H= 454:45 +CPU= 27.011
  23:59:36 STEP 1820 H= 455:00 +CPU= 27.756
  00:00:04 STEP 1821 H= 455:15 +CPU= 27.819
  00:00:32 STEP 1822 H= 455:30 +CPU= 27.120
  00:00:59 STEP 1823 H= 455:45 +CPU= 26.950
  00:01:37 STEP 1824 H= 456:00 +CPU= 38.298
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001824
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001824
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001812
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001812
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001812
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0_r2117891975_18.zip
Uploading the intermediate file: upload_file_18.zip
  00:02:06 STEP 1825 H= 456:15 +CPU= 28.237
  00:02:35 STEP 1826 H= 456:30 +CPU= 28.552
  00:03:03 STEP 1827 H= 456:45 +CPU= 28.165
  00:03:31 STEP 1828 H= 457:00 +CPU= 27.578
  00:03:58 STEP 1829 H= 457:15 +CPU= 27.261
  00:04:26 STEP 1830 H= 457:30 +CPU= 27.587
  00:04:54 STEP 1831 H= 457:45 +CPU= 27.332
  00:05:24 STEP 1832 H= 458:00 +CPU= 29.492
  00:05:53 STEP 1833 H= 458:15 +CPU= 28.435
  00:06:21 STEP 1834 H= 458:30 +CPU= 27.980
  00:06:50 STEP 1835 H= 458:45 +CPU= 28.422
  00:07:27 STEP 1836 H= 459:00 +CPU= 35.585
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001836
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001836
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001836
  00:07:56 STEP 1837 H= 459:15 +CPU= 28.577
  00:08:24 STEP 1838 H= 459:30 +CPU= 28.326
  00:08:53 STEP 1839 H= 459:45 +CPU= 28.474
  00:09:23 STEP 1840 H= 460:00 +CPU= 29.114
  00:09:52 STEP 1841 H= 460:15 +CPU= 28.713
  00:10:21 STEP 1842 H= 460:30 +CPU= 28.077
  00:10:49 STEP 1843 H= 460:45 +CPU= 28.135
  00:11:19 STEP 1844 H= 461:00 +CPU= 28.840
  00:11:47 STEP 1845 H= 461:15 +CPU= 28.224
  00:12:16 STEP 1846 H= 461:30 +CPU= 28.600
  00:12:45 STEP 1847 H= 461:45 +CPU= 28.399
  00:13:22 STEP 1848 H= 462:00 +CPU= 35.617
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001848
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001848
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001848
  00:13:50 STEP 1849 H= 462:15 +CPU= 27.922
  00:14:18 STEP 1850 H= 462:30 +CPU= 27.751
  00:14:47 STEP 1851 H= 462:45 +CPU= 28.135
  00:15:16 STEP 1852 H= 463:00 +CPU= 29.302
  00:15:45 STEP 1853 H= 463:15 +CPU= 28.056
  00:16:13 STEP 1854 H= 463:30 +CPU= 28.708
  00:16:41 STEP 1855 H= 463:45 +CPU= 27.449
  00:17:10 STEP 1856 H= 464:00 +CPU= 28.445
  00:17:38 STEP 1857 H= 464:15 +CPU= 28.621
  00:18:04 STEP 1858 H= 464:30 +CPU= 25.874
  00:18:32 STEP 1859 H= 464:45 +CPU= 27.422
  00:19:08 STEP 1860 H= 465:00 +CPU= 35.640
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001860
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001860
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001860
  00:19:34 STEP 1861 H= 465:15 +CPU= 26.476
  00:20:02 STEP 1862 H= 465:30 +CPU= 27.810
  00:20:30 STEP 1863 H= 465:45 +CPU= 27.404
  00:20:59 STEP 1864 H= 466:00 +CPU= 28.858
  00:21:27 STEP 1865 H= 466:15 +CPU= 27.449
  00:21:55 STEP 1866 H= 466:30 +CPU= 28.329
  00:22:20 STEP 1867 H= 466:45 +CPU= 23.829
  00:22:42 STEP 1868 H= 467:00 +CPU= 21.935
  00:23:06 STEP 1869 H= 467:15 +CPU= 24.189
  00:23:34 STEP 1870 H= 467:30 +CPU= 27.880
  00:24:04 STEP 1871 H= 467:45 +CPU= 28.937
  00:24:46 STEP 1872 H= 468:00 +CPU= 40.904
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001872
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001872
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001872
  00:25:15 STEP 1873 H= 468:15 +CPU= 28.027
  00:25:44 STEP 1874 H= 468:30 +CPU= 28.191
  00:26:11 STEP 1875 H= 468:45 +CPU= 27.358
  00:26:41 STEP 1876 H= 469:00 +CPU= 29.074
  00:27:08 STEP 1877 H= 469:15 +CPU= 26.969
  00:27:36 STEP 1878 H= 469:30 +CPU= 27.577
  00:28:04 STEP 1879 H= 469:45 +CPU= 27.713
  00:28:33 STEP 1880 H= 470:00 +CPU= 28.388
  00:29:01 STEP 1881 H= 470:15 +CPU= 28.168
  00:29:30 STEP 1882 H= 470:30 +CPU= 28.361
  00:29:58 STEP 1883 H= 470:45 +CPU= 27.708
  00:30:34 STEP 1884 H= 471:00 +CPU= 35.348
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001884
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001884
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001884
  00:31:02 STEP 1885 H= 471:15 +CPU= 28.421
  00:31:31 STEP 1886 H= 471:30 +CPU= 28.641
  00:31:59 STEP 1887 H= 471:45 +CPU= 27.625
  00:32:27 STEP 1888 H= 472:00 +CPU= 28.032
  00:32:55 STEP 1889 H= 472:15 +CPU= 27.658
  00:33:23 STEP 1890 H= 472:30 +CPU= 27.641
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1an
(argv4) batchid: 1018
(argv5) wuid: 12290006
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /boinc_data/boinc/slots/4
Project directory is: /boinc_data/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006
..mkdir for temp folder for results failed
Copying: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /boinc_data/boinc/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /boinc_data/boinc/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7464a078b3e4c56941e7e9ab932a378e to: /boinc_data/boinc/slots/4/oifs_43r3_bl_a1an_2016092300_20_1018_12290006.zip
Unzipping the namelist zip file: /boinc_data/boinc/slots/4/oifs_43r3_bl_a1an_2016092300_20_1018_12290006.zip
ic_ancil_file: ic_ancil_12290006
ifsdata_file: ifsdata_12290006
climate_data_file: clim_data_12290006
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /boinc_data/boinc/slots/4/ic_ancil_12290006.zip
Unzipping the IC ancils zip file: /boinc_data/boinc/slots/4/ic_ancil_12290006.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_24ff58205e240807058755be99233c35 to: /boinc_data/boinc/slots/4/ifsdata/ifsdata_12290006.zip
Unzipping the ifsdata_zip file: /boinc_data/boinc/slots/4/ifsdata/ifsdata_12290006.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /boinc_data/boinc/slots/4/159l_2/clim_data_12290006.zip
Unzipping the climate data zip file: /boinc_data/boinc/slots/4/159l_2/clim_data_12290006.zip
Checking for progress XML file: /boinc_data/boinc/slots/4/progress_file_12290006.xml
Opened progress file ok : /boinc_data/boinc/slots/4/progress_file_12290006.xml
-- Model is restarting --
Adjusting last_iter, 1891, to previous model restart step.
Creating progress file: /boinc_data/boinc/slots/4/progress_file_12290006.xml
last_cpu_time: 52560
upload_file_number: 19
last_iter: 1873
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0_r2117891975
The child process has been launched with process id: 865254
Executing the command: /boinc_data/boinc/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865254:865254] [20240616:010756:1718518076:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:08:27 STEP 1872 H= 468:00 +CPU= 27.312
  01:08:52 STEP 1873 H= 468:15 +CPU= 24.674
  01:09:17 STEP 1874 H= 468:30 +CPU= 25.022
  01:09:42 STEP 1875 H= 468:45 +CPU= 24.579
  01:10:08 STEP 1876 H= 469:00 +CPU= 25.969
  01:10:35 STEP 1877 H= 469:15 +CPU= 27.232
  01:11:02 STEP 1878 H= 469:30 +CPU= 27.014
  01:11:28 STEP 1879 H= 469:45 +CPU= 26.439
  01:11:56 STEP 1880 H= 470:00 +CPU= 27.938
  01:12:23 STEP 1881 H= 470:15 +CPU= 26.100
  01:12:47 STEP 1882 H= 470:30 +CPU= 24.849
  01:13:12 STEP 1883 H= 470:45 +CPU= 24.344
  01:13:45 STEP 1884 H= 471:00 +CPU= 32.728
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001884
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001884
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001884
  01:14:11 STEP 1885 H= 471:15 +CPU= 26.694
  01:14:38 STEP 1886 H= 471:30 +CPU= 26.527
  01:15:06 STEP 1887 H= 471:45 +CPU= 27.452
  01:15:33 STEP 1888 H= 472:00 +CPU= 27.759
  01:15:59 STEP 1889 H= 472:15 +CPU= 25.751
  01:16:26 STEP 1890 H= 472:30 +CPU= 26.704
  01:16:52 STEP 1891 H= 472:45 +CPU= 25.642
  01:17:19 STEP 1892 H= 473:00 +CPU= 27.703
  01:17:47 STEP 1893 H= 473:15 +CPU= 27.687
  01:18:14 STEP 1894 H= 473:30 +CPU= 27.373
  01:18:42 STEP 1895 H= 473:45 +CPU= 27.108
  01:19:16 STEP 1896 H= 474:00 +CPU= 34.144
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001896
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001896
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001896
  01:19:43 STEP 1897 H= 474:15 +CPU= 26.892
  01:20:10 STEP 1898 H= 474:30 +CPU= 27.287
  01:20:36 STEP 1899 H= 474:45 +CPU= 26.065
  01:21:04 STEP 1900 H= 475:00 +CPU= 27.700
  01:21:30 STEP 1901 H= 475:15 +CPU= 25.523
  01:21:56 STEP 1902 H= 475:30 +CPU= 25.907
  01:22:23 STEP 1903 H= 475:45 +CPU= 27.320
  01:22:51 STEP 1904 H= 476:00 +CPU= 27.374
  01:23:17 STEP 1905 H= 476:15 +CPU= 26.784
  01:23:46 STEP 1906 H= 476:30 +CPU= 28.273
  01:24:13 STEP 1907 H= 476:45 +CPU= 27.266
  01:24:46 STEP 1908 H= 477:00 +CPU= 32.755
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001908
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001908
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001908
  01:25:12 STEP 1909 H= 477:15 +CPU= 26.413
  01:25:39 STEP 1910 H= 477:30 +CPU= 27.008
  01:26:06 STEP 1911 H= 477:45 +CPU= 26.902
  01:26:34 STEP 1912 H= 478:00 +CPU= 27.749
  01:27:02 STEP 1913 H= 478:15 +CPU= 27.959
  01:27:28 STEP 1914 H= 478:30 +CPU= 26.215
  01:27:55 STEP 1915 H= 478:45 +CPU= 26.379
  01:28:23 STEP 1916 H= 479:00 +CPU= 27.642
  01:28:50 STEP 1917 H= 479:15 +CPU= 26.867
  01:29:16 STEP 1918 H= 479:30 +CPU= 26.105
  01:29:43 STEP 1919 H= 479:45 +CPU= 27.141
  01:30:18 STEP 1920 H= 480:00 +CPU= 35.441
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.5297708874E-21      I       0.7755783184E-06      I
ICloud fractionI      0.0000000000E+00      I       0.1878151793E-01      I
I  Relat. Hum. I      0.5552187656E-01      I       0.1410075250E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1415591364E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    393927876 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS656546.5 Mbytes/s, TOTAL TIME=    0.0105( 5.7%)
 01:30:18 STEP 1920 H= 480:00 +CPU= 35.441
 END CNT3
 NSTEP =  1920 CNT0   000000000  25.498  25.498  25.589   0.996               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************     4.1     1.5     2.6     2.3     5.4
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1342637.2 1342637.2    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1338780.9 1338780.9     99.71      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    4755.9    4755.9     18.77      0.00
   9 SPCM     - SPECTRAL COMP.                      48     673.3     673.3      2.41      0.00
  10 SCAN2M - PHYSICS                               49   12365.7   12365.7     45.13      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4     948.4     948.4      0.28      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      46.5      46.5      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      25.6      25.6      0.01      0.00
  27 RESTART FILES - WRITING                         1    3793.7    3793.7      0.28      0.00
  28 RESTART FILES - READING                         1    3109.6    3109.6      0.23      0.00
  29 SU4FPOS IN CNT4                                 4       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    5972.2    5972.2      1.78      0.00
  31 POSDDH IN STEPO                                 1       0.9       0.9      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    6041.2    6041.2     22.05      0.00
  39 SU0YOMB                                         1     456.7     456.7      0.03      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      21.1      21.1      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     332.1     332.1      1.19      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      60.6      60.6      0.22      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     401.7     401.7      3.23      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     235.4     235.4      1.75      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      83.0      83.0      0.62      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     119.5     119.5      0.96      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      50.0      50.0      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      62.4      62.4      0.50      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      34.3      34.3      0.26      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108      54.8      54.8      0.44      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      31.4      31.4      0.23      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1342.6 CPU TIME    1339.6 VECTOR TIME     1339.6


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1290.7
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /boinc_data/boinc/slots/4/ICMGGh7zg+001920
Moving to projects directory: /boinc_data/boinc/slots/4/ICMSHh7zg+001920
Moving to projects directory: /boinc_data/boinc/slots/4/ICMUAh7zg+001920
Adding to the zip: /boinc_data/boinc/slots/4/NODE.001_01
Adding to the zip: /boinc_data/boinc/slots/4/ifs.stat
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001860
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001860
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMGGh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMSHh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12290006/ICMUAh7zg+001860
Zipping up the final file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0_r2117891975_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
01:33:49 (865236): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
16 Jun 2024 06:32:10 1496046 22443595 oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0 1,727,100 53,899 0.0312
16 Jun 2024 01:49:18 1496046 22443595 oifs_43r3_bl_a1an_2016092300_20_1018_12290006_0 1,209,600 37,323 0.0309


©2024 climateprediction.net