climateprediction.net home page
Task 22442013

Task 22442013

Name oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0
Workunit 12288458
Created 12 Jun 2024, 20:10:10 UTC
Sent 13 Jun 2024, 16:56:02 UTC
Report deadline 12 Aug 2024, 16:56:02 UTC
Received 14 Jun 2024, 20:19:33 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537687
Run time 16 hours 32 min 43 sec
CPU time 15 hours 31 min 13 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 7.20 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,006.65 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,377.30 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:852450:852450] [20240614:140746:1718374066:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:852450:852450] [20240614:140746:1718374066:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:852450:852450] [20240614:140746:1718374066:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:852450:852450] [20240614:140746:1718374066:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:852450:852450] [20240614:140746:1718374066:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:08:21 STEP 1440 H= 360:00 +CPU= 30.126
  14:08:51 STEP 1441 H= 360:15 +CPU= 29.519
  14:09:21 STEP 1442 H= 360:30 +CPU= 29.635
  14:09:51 STEP 1443 H= 360:45 +CPU= 29.923
  14:10:21 STEP 1444 H= 361:00 +CPU= 29.929
  14:10:52 STEP 1445 H= 361:15 +CPU= 29.950
  14:11:22 STEP 1446 H= 361:30 +CPU= 29.761
  14:11:52 STEP 1447 H= 361:45 +CPU= 29.442
  14:12:23 STEP 1448 H= 362:00 +CPU= 30.540
  14:12:52 STEP 1449 H= 362:15 +CPU= 28.956
  14:13:21 STEP 1450 H= 362:30 +CPU= 29.368
  14:13:50 STEP 1451 H= 362:45 +CPU= 29.108
  14:14:26 STEP 1452 H= 363:00 +CPU= 36.001
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001452
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001452
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001452
  14:14:57 STEP 1453 H= 363:15 +CPU= 29.632
  14:15:28 STEP 1454 H= 363:30 +CPU= 30.420
  14:15:59 STEP 1455 H= 363:45 +CPU= 29.952
  14:16:29 STEP 1456 H= 364:00 +CPU= 30.144
  14:16:59 STEP 1457 H= 364:15 +CPU= 29.803
  14:17:30 STEP 1458 H= 364:30 +CPU= 29.885
  14:18:00 STEP 1459 H= 364:45 +CPU= 29.316
  14:18:31 STEP 1460 H= 365:00 +CPU= 30.704
  14:19:01 STEP 1461 H= 365:15 +CPU= 29.377
  14:19:31 STEP 1462 H= 365:30 +CPU= 29.221
  14:20:01 STEP 1463 H= 365:45 +CPU= 29.599
  14:20:40 STEP 1464 H= 366:00 +CPU= 37.875
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001464
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001464
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001464
  14:21:09 STEP 1465 H= 366:15 +CPU= 29.592
  14:21:39 STEP 1466 H= 366:30 +CPU= 29.512
  14:22:09 STEP 1467 H= 366:45 +CPU= 29.430
  14:22:39 STEP 1468 H= 367:00 +CPU= 29.852
  14:23:09 STEP 1469 H= 367:15 +CPU= 29.337
  14:23:38 STEP 1470 H= 367:30 +CPU= 28.857
  14:24:08 STEP 1471 H= 367:45 +CPU= 29.634
  14:24:38 STEP 1472 H= 368:00 +CPU= 30.490
  14:25:08 STEP 1473 H= 368:15 +CPU= 29.363
  14:25:39 STEP 1474 H= 368:30 +CPU= 29.464
  14:26:08 STEP 1475 H= 368:45 +CPU= 28.993
  14:26:43 STEP 1476 H= 369:00 +CPU= 35.477
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001476
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001476
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001476
  14:27:13 STEP 1477 H= 369:15 +CPU= 29.090
  14:27:42 STEP 1478 H= 369:30 +CPU= 29.101
  14:28:11 STEP 1479 H= 369:45 +CPU= 28.864
  14:28:40 STEP 1480 H= 370:00 +CPU= 29.561
  14:29:09 STEP 1481 H= 370:15 +CPU= 29.116
  14:29:40 STEP 1482 H= 370:30 +CPU= 30.106
  14:30:09 STEP 1483 H= 370:45 +CPU= 29.108
  14:30:39 STEP 1484 H= 371:00 +CPU= 29.539
  14:31:09 STEP 1485 H= 371:15 +CPU= 29.385
  14:31:40 STEP 1486 H= 371:30 +CPU= 29.798
  14:32:10 STEP 1487 H= 371:45 +CPU= 29.616
  14:32:50 STEP 1488 H= 372:00 +CPU= 37.853
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001488
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001488
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001488
  14:33:20 STEP 1489 H= 372:15 +CPU= 29.559
  14:33:49 STEP 1490 H= 372:30 +CPU= 29.691
  14:34:20 STEP 1491 H= 372:45 +CPU= 29.770
  14:34:51 STEP 1492 H= 373:00 +CPU= 31.013
  14:35:21 STEP 1493 H= 373:15 +CPU= 29.753
  14:35:51 STEP 1494 H= 373:30 +CPU= 29.797
  14:36:21 STEP 1495 H= 373:45 +CPU= 29.371
  14:36:51 STEP 1496 H= 374:00 +CPU= 30.508
  14:37:21 STEP 1497 H= 374:15 +CPU= 29.506
  14:37:51 STEP 1498 H= 374:30 +CPU= 30.108
  14:38:21 STEP 1499 H= 374:45 +CPU= 29.881
  14:38:59 STEP 1500 H= 375:00 +CPU= 36.828
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001500
  14:39:29 STEP 1501 H= 375:15 +CPU= 29.692
  14:40:00 STEP 1502 H= 375:30 +CPU= 30.110
  14:40:30 STEP 1503 H= 375:45 +CPU= 29.748
  14:41:01 STEP 1504 H= 376:00 +CPU= 30.796
  14:41:31 STEP 1505 H= 376:15 +CPU= 29.178
  14:42:01 STEP 1506 H= 376:30 +CPU= 29.464
  14:42:31 STEP 1507 H= 376:45 +CPU= 29.074
  14:43:01 STEP 1508 H= 377:00 +CPU= 30.081
  14:43:31 STEP 1509 H= 377:15 +CPU= 29.137
  14:44:00 STEP 1510 H= 377:30 +CPU= 29.438
  14:44:30 STEP 1511 H= 377:45 +CPU= 29.862
  14:45:07 STEP 1512 H= 378:00 +CPU= 36.262
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001512
  14:45:36 STEP 1513 H= 378:15 +CPU= 29.255
  14:46:06 STEP 1514 H= 378:30 +CPU= 29.176
  14:46:35 STEP 1515 H= 378:45 +CPU= 29.201
  14:47:05 STEP 1516 H= 379:00 +CPU= 29.430
  14:47:36 STEP 1517 H= 379:15 +CPU= 29.572
  14:48:05 STEP 1518 H= 379:30 +CPU= 29.039
  14:48:35 STEP 1519 H= 379:45 +CPU= 29.159
  14:49:05 STEP 1520 H= 380:00 +CPU= 29.883
  14:49:34 STEP 1521 H= 380:15 +CPU= 29.447
  14:50:04 STEP 1522 H= 380:30 +CPU= 29.429
  14:50:34 STEP 1523 H= 380:45 +CPU= 29.290
  14:51:10 STEP 1524 H= 381:00 +CPU= 35.304
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001524
  14:51:39 STEP 1525 H= 381:15 +CPU= 29.397
  14:52:09 STEP 1526 H= 381:30 +CPU= 29.110
  14:52:39 STEP 1527 H= 381:45 +CPU= 29.041
  14:53:10 STEP 1528 H= 382:00 +CPU= 30.550
  14:53:40 STEP 1529 H= 382:15 +CPU= 29.340
  14:54:10 STEP 1530 H= 382:30 +CPU= 29.154
  14:54:39 STEP 1531 H= 382:45 +CPU= 28.449
  14:55:10 STEP 1532 H= 383:00 +CPU= 29.343
  14:55:41 STEP 1533 H= 383:15 +CPU= 29.260
  14:56:11 STEP 1534 H= 383:30 +CPU= 29.657
  14:56:40 STEP 1535 H= 383:45 +CPU= 28.427
  14:57:20 STEP 1536 H= 384:00 +CPU= 38.941
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001536
  14:57:50 STEP 1537 H= 384:15 +CPU= 29.792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001536
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0_r1890634521_15.zip
Uploading the intermediate file: upload_file_15.zip
  14:58:20 STEP 1538 H= 384:30 +CPU= 28.952
  14:58:49 STEP 1539 H= 384:45 +CPU= 29.539
  14:59:20 STEP 1540 H= 385:00 +CPU= 30.609
  14:59:51 STEP 1541 H= 385:15 +CPU= 29.763
  15:00:21 STEP 1542 H= 385:30 +CPU= 29.958
  15:00:51 STEP 1543 H= 385:45 +CPU= 29.719
  15:01:22 STEP 1544 H= 386:00 +CPU= 30.607
  15:01:52 STEP 1545 H= 386:15 +CPU= 29.593
  15:02:22 STEP 1546 H= 386:30 +CPU= 30.142
  15:02:52 STEP 1547 H= 386:45 +CPU= 29.260
  15:03:30 STEP 1548 H= 387:00 +CPU= 37.381
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001548
  15:03:59 STEP 1549 H= 387:15 +CPU= 29.460
  15:04:29 STEP 1550 H= 387:30 +CPU= 29.514
  15:04:58 STEP 1551 H= 387:45 +CPU= 28.707
  15:05:28 STEP 1552 H= 388:00 +CPU= 30.037
  15:05:58 STEP 1553 H= 388:15 +CPU= 29.070
  15:06:28 STEP 1554 H= 388:30 +CPU= 29.452
  15:06:59 STEP 1555 H= 388:45 +CPU= 29.389
  15:07:30 STEP 1556 H= 389:00 +CPU= 29.827
  15:08:00 STEP 1557 H= 389:15 +CPU= 29.426
  15:08:29 STEP 1558 H= 389:30 +CPU= 28.823
  15:09:00 STEP 1559 H= 389:45 +CPU= 29.524
  15:09:37 STEP 1560 H= 390:00 +CPU= 35.569
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001560
  15:10:06 STEP 1561 H= 390:15 +CPU= 28.790
  15:10:36 STEP 1562 H= 390:30 +CPU= 29.543
  15:11:05 STEP 1563 H= 390:45 +CPU= 29.598
  15:11:35 STEP 1564 H= 391:00 +CPU= 29.430
  15:12:04 STEP 1565 H= 391:15 +CPU= 28.775
  15:12:34 STEP 1566 H= 391:30 +CPU= 29.564
  15:13:03 STEP 1567 H= 391:45 +CPU= 29.121
  15:13:34 STEP 1568 H= 392:00 +CPU= 30.123
  15:14:03 STEP 1569 H= 392:15 +CPU= 29.023
  15:14:33 STEP 1570 H= 392:30 +CPU= 29.315
  15:15:02 STEP 1571 H= 392:45 +CPU= 28.869
  15:15:38 STEP 1572 H= 393:00 +CPU= 35.652
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001572
  15:16:02 STEP 1573 H= 393:15 +CPU= 24.254
  15:16:25 STEP 1574 H= 393:30 +CPU= 22.913
  15:16:46 STEP 1575 H= 393:45 +CPU= 21.383
  15:17:14 STEP 1576 H= 394:00 +CPU= 27.917
  15:17:38 STEP 1577 H= 394:15 +CPU= 24.129
  15:18:02 STEP 1578 H= 394:30 +CPU= 23.778
  15:18:27 STEP 1579 H= 394:45 +CPU= 24.548
  15:18:53 STEP 1580 H= 395:00 +CPU= 26.121
  15:19:23 STEP 1581 H= 395:15 +CPU= 29.375
  15:19:54 STEP 1582 H= 395:30 +CPU= 29.833
  15:20:24 STEP 1583 H= 395:45 +CPU= 29.030
  15:21:05 STEP 1584 H= 396:00 +CPU= 39.973
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001584
  15:21:34 STEP 1585 H= 396:15 +CPU= 29.383
  15:22:05 STEP 1586 H= 396:30 +CPU= 29.476
  15:22:34 STEP 1587 H= 396:45 +CPU= 29.224
  15:23:05 STEP 1588 H= 397:00 +CPU= 29.864
  15:23:35 STEP 1589 H= 397:15 +CPU= 29.478
  15:24:05 STEP 1590 H= 397:30 +CPU= 29.145
  15:24:34 STEP 1591 H= 397:45 +CPU= 28.898
  15:25:04 STEP 1592 H= 398:00 +CPU= 29.894
  15:25:35 STEP 1593 H= 398:15 +CPU= 29.719
  15:26:04 STEP 1594 H= 398:30 +CPU= 28.842
  15:26:34 STEP 1595 H= 398:45 +CPU= 29.374
  15:27:09 STEP 1596 H= 399:00 +CPU= 35.106
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001596
  15:27:39 STEP 1597 H= 399:15 +CPU= 29.939
  15:28:09 STEP 1598 H= 399:30 +CPU= 29.566
  15:28:39 STEP 1599 H= 399:45 +CPU= 29.307
  15:29:09 STEP 1600 H= 400:00 +CPU= 29.968
  15:29:39 STEP 1601 H= 400:15 +CPU= 29.925
  15:30:09 STEP 1602 H= 400:30 +CPU= 29.566
  15:30:39 STEP 1603 H= 400:45 +CPU= 29.760
  15:31:09 STEP 1604 H= 401:00 +CPU= 29.952
  15:31:39 STEP 1605 H= 401:15 +CPU= 30.149
  15:32:09 STEP 1606 H= 401:30 +CPU= 29.604
  15:32:39 STEP 1607 H= 401:45 +CPU= 29.341
  15:33:15 STEP 1608 H= 402:00 +CPU= 36.390
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001608
  15:33:46 STEP 1609 H= 402:15 +CPU= 30.088
  15:34:16 STEP 1610 H= 402:30 +CPU= 29.810
  15:34:45 STEP 1611 H= 402:45 +CPU= 28.907
  15:35:16 STEP 1612 H= 403:00 +CPU= 30.619
  15:35:45 STEP 1613 H= 403:15 +CPU= 29.309
  15:36:15 STEP 1614 H= 403:30 +CPU= 29.514
  15:36:44 STEP 1615 H= 403:45 +CPU= 28.935
  15:37:15 STEP 1616 H= 404:00 +CPU= 30.333
  15:37:45 STEP 1617 H= 404:15 +CPU= 29.353
  15:38:14 STEP 1618 H= 404:30 +CPU= 29.434
  15:38:43 STEP 1619 H= 404:45 +CPU= 29.276
  15:39:20 STEP 1620 H= 405:00 +CPU= 36.125
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001620
  15:39:50 STEP 1621 H= 405:15 +CPU= 29.704
  15:40:20 STEP 1622 H= 405:30 +CPU= 29.145
  15:40:48 STEP 1623 H= 405:45 +CPU= 28.037
  15:41:19 STEP 1624 H= 406:00 +CPU= 30.045
  15:41:50 STEP 1625 H= 406:15 +CPU= 28.661
  15:42:19 STEP 1626 H= 406:30 +CPU= 28.984
  15:42:50 STEP 1627 H= 406:45 +CPU= 30.090
  15:43:21 STEP 1628 H= 407:00 +CPU= 29.527
  15:43:51 STEP 1629 H= 407:15 +CPU= 30.028
  15:44:20 STEP 1630 H= 407:30 +CPU= 29.453
  15:44:50 STEP 1631 H= 407:45 +CPU= 29.644
  15:45:31 STEP 1632 H= 408:00 +CPU= 39.343
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001632
  15:46:01 STEP 1633 H= 408:15 +CPU= 29.709
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001632
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0_r1890634521_16.zip
Uploading the intermediate file: upload_file_16.zip
  15:46:31 STEP 1634 H= 408:30 +CPU= 29.343
  15:47:01 STEP 1635 H= 408:45 +CPU= 29.255
  15:47:32 STEP 1636 H= 409:00 +CPU= 30.801
  15:48:02 STEP 1637 H= 409:15 +CPU= 29.895
  15:48:32 STEP 1638 H= 409:30 +CPU= 29.234
  15:49:02 STEP 1639 H= 409:45 +CPU= 29.459
  15:49:33 STEP 1640 H= 410:00 +CPU= 30.241
  15:50:02 STEP 1641 H= 410:15 +CPU= 29.377
  15:50:32 STEP 1642 H= 410:30 +CPU= 29.698
  15:51:02 STEP 1643 H= 410:45 +CPU= 29.427
  15:51:40 STEP 1644 H= 411:00 +CPU= 37.044
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001644
  15:52:10 STEP 1645 H= 411:15 +CPU= 29.459
  15:52:40 STEP 1646 H= 411:30 +CPU= 28.938
  15:53:09 STEP 1647 H= 411:45 +CPU= 29.068
  15:53:42 STEP 1648 H= 412:00 +CPU= 31.080
  15:54:12 STEP 1649 H= 412:15 +CPU= 29.367
  15:54:42 STEP 1650 H= 412:30 +CPU= 29.224
  15:55:12 STEP 1651 H= 412:45 +CPU= 29.453
  15:55:42 STEP 1652 H= 413:00 +CPU= 30.188
  15:56:13 STEP 1653 H= 413:15 +CPU= 29.878
  15:56:42 STEP 1654 H= 413:30 +CPU= 28.848
  15:57:12 STEP 1655 H= 413:45 +CPU= 29.622
  15:57:48 STEP 1656 H= 414:00 +CPU= 36.341
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001656
  15:58:18 STEP 1657 H= 414:15 +CPU= 29.354
  15:58:48 STEP 1658 H= 414:30 +CPU= 29.611
  15:59:17 STEP 1659 H= 414:45 +CPU= 29.578
  15:59:47 STEP 1660 H= 415:00 +CPU= 30.024
  16:00:16 STEP 1661 H= 415:15 +CPU= 28.968
  16:00:46 STEP 1662 H= 415:30 +CPU= 29.627
  16:01:16 STEP 1663 H= 415:45 +CPU= 29.615
  16:01:46 STEP 1664 H= 416:00 +CPU= 30.380
  16:02:16 STEP 1665 H= 416:15 +CPU= 29.253
  16:02:45 STEP 1666 H= 416:30 +CPU= 29.138
  16:03:15 STEP 1667 H= 416:45 +CPU= 29.614
  16:03:51 STEP 1668 H= 417:00 +CPU= 35.754
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001668
  16:04:21 STEP 1669 H= 417:15 +CPU= 29.555
  16:04:50 STEP 1670 H= 417:30 +CPU= 28.980
  16:05:20 STEP 1671 H= 417:45 +CPU= 28.452
  16:05:50 STEP 1672 H= 418:00 +CPU= 29.464
  16:06:20 STEP 1673 H= 418:15 +CPU= 29.192
  16:06:49 STEP 1674 H= 418:30 +CPU= 28.851
  16:07:19 STEP 1675 H= 418:45 +CPU= 29.202
  16:07:49 STEP 1676 H= 419:00 +CPU= 28.775
  16:08:20 STEP 1677 H= 419:15 +CPU= 29.582
  16:08:50 STEP 1678 H= 419:30 +CPU= 28.848
  16:09:20 STEP 1679 H= 419:45 +CPU= 29.071
  16:10:00 STEP 1680 H= 420:00 +CPU= 39.504
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001680
  16:10:30 STEP 1681 H= 420:15 +CPU= 29.598
  16:10:59 STEP 1682 H= 420:30 +CPU= 29.336
  16:11:30 STEP 1683 H= 420:45 +CPU= 29.882
  16:12:00 STEP 1684 H= 421:00 +CPU= 30.321
  16:12:29 STEP 1685 H= 421:15 +CPU= 29.066
  16:13:00 STEP 1686 H= 421:30 +CPU= 29.816
  16:13:30 STEP 1687 H= 421:45 +CPU= 29.580
  16:14:00 STEP 1688 H= 422:00 +CPU= 30.283
  16:14:30 STEP 1689 H= 422:15 +CPU= 29.653
  16:14:59 STEP 1690 H= 422:30 +CPU= 29.627
  16:15:30 STEP 1691 H= 422:45 +CPU= 29.737
  16:16:06 STEP 1692 H= 423:00 +CPU= 36.831
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001692
  16:16:37 STEP 1693 H= 423:15 +CPU= 29.906
  16:17:07 STEP 1694 H= 423:30 +CPU= 29.531
  16:17:37 STEP 1695 H= 423:45 +CPU= 29.639
  16:18:07 STEP 1696 H= 424:00 +CPU= 30.459
  16:18:37 STEP 1697 H= 424:15 +CPU= 29.158
  16:19:07 STEP 1698 H= 424:30 +CPU= 30.030
  16:19:37 STEP 1699 H= 424:45 +CPU= 29.919
  16:20:08 STEP 1700 H= 425:00 +CPU= 30.527
  16:20:38 STEP 1701 H= 425:15 +CPU= 29.402
  16:21:07 STEP 1702 H= 425:30 +CPU= 29.489
  16:21:36 STEP 1703 H= 425:45 +CPU= 29.349
  16:22:13 STEP 1704 H= 426:00 +CPU= 36.000
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001704
  16:22:42 STEP 1705 H= 426:15 +CPU= 29.306
  16:23:11 STEP 1706 H= 426:30 +CPU= 29.345
  16:23:42 STEP 1707 H= 426:45 +CPU= 29.540
  16:24:11 STEP 1708 H= 427:00 +CPU= 29.589
  16:24:41 STEP 1709 H= 427:15 +CPU= 28.991
  16:25:10 STEP 1710 H= 427:30 +CPU= 29.036
  16:25:39 STEP 1711 H= 427:45 +CPU= 29.582
  16:26:09 STEP 1712 H= 428:00 +CPU= 29.807
  16:26:39 STEP 1713 H= 428:15 +CPU= 29.117
  16:27:08 STEP 1714 H= 428:30 +CPU= 29.339
  16:27:37 STEP 1715 H= 428:45 +CPU= 29.401
  16:28:14 STEP 1716 H= 429:00 +CPU= 36.565
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001716
  16:28:44 STEP 1717 H= 429:15 +CPU= 29.595
  16:29:14 STEP 1718 H= 429:30 +CPU= 29.372
  16:29:45 STEP 1719 H= 429:45 +CPU= 29.852
  16:30:15 STEP 1720 H= 430:00 +CPU= 29.620
  16:30:45 STEP 1721 H= 430:15 +CPU= 28.830
  16:31:15 STEP 1722 H= 430:30 +CPU= 29.566
  16:31:45 STEP 1723 H= 430:45 +CPU= 29.621
  16:32:15 STEP 1724 H= 431:00 +CPU= 30.263
  16:32:46 STEP 1725 H= 431:15 +CPU= 29.847
  16:33:15 STEP 1726 H= 431:30 +CPU= 29.639
  16:33:45 STEP 1727 H= 431:45 +CPU= 28.893
  16:34:25 STEP 1728 H= 432:00 +CPU= 39.467
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001728
  16:34:55 STEP 1729 H= 432:15 +CPU= 29.426
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001728
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0_r1890634521_17.zip
Uploading the intermediate file: upload_file_17.zip
  16:35:24 STEP 1730 H= 432:30 +CPU= 28.920
  16:35:54 STEP 1731 H= 432:45 +CPU= 29.315
  16:36:25 STEP 1732 H= 433:00 +CPU= 30.207
  16:36:55 STEP 1733 H= 433:15 +CPU= 29.830
  16:37:25 STEP 1734 H= 433:30 +CPU= 29.357
  16:37:54 STEP 1735 H= 433:45 +CPU= 29.453
  16:38:24 STEP 1736 H= 434:00 +CPU= 29.192
  16:38:53 STEP 1737 H= 434:15 +CPU= 29.400
  16:39:22 STEP 1738 H= 434:30 +CPU= 29.192
  16:39:52 STEP 1739 H= 434:45 +CPU= 29.219
  16:40:28 STEP 1740 H= 435:00 +CPU= 36.222
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001740
  16:40:58 STEP 1741 H= 435:15 +CPU= 29.539
  16:41:27 STEP 1742 H= 435:30 +CPU= 29.005
  16:41:57 STEP 1743 H= 435:45 +CPU= 29.491
  16:42:27 STEP 1744 H= 436:00 +CPU= 30.062
  16:42:56 STEP 1745 H= 436:15 +CPU= 29.236
  16:43:26 STEP 1746 H= 436:30 +CPU= 29.397
  16:43:55 STEP 1747 H= 436:45 +CPU= 29.180
  16:44:25 STEP 1748 H= 437:00 +CPU= 29.788
  16:44:54 STEP 1749 H= 437:15 +CPU= 29.271
  16:45:24 STEP 1750 H= 437:30 +CPU= 29.628
  16:45:53 STEP 1751 H= 437:45 +CPU= 29.358
  16:46:30 STEP 1752 H= 438:00 +CPU= 36.266
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001752
  16:47:00 STEP 1753 H= 438:15 +CPU= 30.011
  16:47:30 STEP 1754 H= 438:30 +CPU= 29.747
  16:47:59 STEP 1755 H= 438:45 +CPU= 29.360
  16:48:29 STEP 1756 H= 439:00 +CPU= 29.499
  16:48:58 STEP 1757 H= 439:15 +CPU= 29.727
  16:49:28 STEP 1758 H= 439:30 +CPU= 29.676
  16:49:58 STEP 1759 H= 439:45 +CPU= 29.204
  16:50:28 STEP 1760 H= 440:00 +CPU= 30.542
  16:50:58 STEP 1761 H= 440:15 +CPU= 29.634
  16:51:29 STEP 1762 H= 440:30 +CPU= 30.039
  16:51:58 STEP 1763 H= 440:45 +CPU= 29.354
  16:52:36 STEP 1764 H= 441:00 +CPU= 35.621
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001764
  16:53:06 STEP 1765 H= 441:15 +CPU= 29.255
  16:53:36 STEP 1766 H= 441:30 +CPU= 29.750
  16:54:05 STEP 1767 H= 441:45 +CPU= 28.635
  16:54:35 STEP 1768 H= 442:00 +CPU= 29.876
  16:55:05 STEP 1769 H= 442:15 +CPU= 28.479
  16:55:34 STEP 1770 H= 442:30 +CPU= 28.936
  16:56:04 STEP 1771 H= 442:45 +CPU= 29.532
  16:56:34 STEP 1772 H= 443:00 +CPU= 29.928
  16:57:04 STEP 1773 H= 443:15 +CPU= 29.385
  16:57:34 STEP 1774 H= 443:30 +CPU= 29.450
  16:58:03 STEP 1775 H= 443:45 +CPU= 28.938
  16:58:43 STEP 1776 H= 444:00 +CPU= 39.383
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001776
  16:59:12 STEP 1777 H= 444:15 +CPU= 29.504
  16:59:42 STEP 1778 H= 444:30 +CPU= 29.351
  17:00:11 STEP 1779 H= 444:45 +CPU= 29.698
  17:00:42 STEP 1780 H= 445:00 +CPU= 30.419
  17:01:12 STEP 1781 H= 445:15 +CPU= 29.565
  17:01:41 STEP 1782 H= 445:30 +CPU= 29.289
  17:02:10 STEP 1783 H= 445:45 +CPU= 29.515
  17:02:41 STEP 1784 H= 446:00 +CPU= 30.125
  17:03:11 STEP 1785 H= 446:15 +CPU= 29.809
  17:03:40 STEP 1786 H= 446:30 +CPU= 28.998
  17:04:09 STEP 1787 H= 446:45 +CPU= 29.233
  17:04:45 STEP 1788 H= 447:00 +CPU= 35.662
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001788
  17:05:15 STEP 1789 H= 447:15 +CPU= 30.017
  17:05:45 STEP 1790 H= 447:30 +CPU= 29.567
  17:06:14 STEP 1791 H= 447:45 +CPU= 29.286
  17:06:45 STEP 1792 H= 448:00 +CPU= 30.121
  17:07:15 STEP 1793 H= 448:15 +CPU= 29.628
  17:07:45 STEP 1794 H= 448:30 +CPU= 29.900
  17:08:14 STEP 1795 H= 448:45 +CPU= 29.287
  17:08:45 STEP 1796 H= 449:00 +CPU= 30.848
  17:09:16 STEP 1797 H= 449:15 +CPU= 29.659
  17:09:45 STEP 1798 H= 449:30 +CPU= 29.154
  17:10:16 STEP 1799 H= 449:45 +CPU= 29.807
  17:10:53 STEP 1800 H= 450:00 +CPU= 37.215
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001800
  17:11:22 STEP 1801 H= 450:15 +CPU= 29.339
  17:11:52 STEP 1802 H= 450:30 +CPU= 29.161
  17:12:21 STEP 1803 H= 450:45 +CPU= 29.128
  17:12:52 STEP 1804 H= 451:00 +CPU= 30.562
  17:13:22 STEP 1805 H= 451:15 +CPU= 29.851
  17:13:52 STEP 1806 H= 451:30 +CPU= 30.134
  17:14:22 STEP 1807 H= 451:45 +CPU= 29.736
  17:14:53 STEP 1808 H= 452:00 +CPU= 30.499
  17:15:24 STEP 1809 H= 452:15 +CPU= 30.314
  17:15:53 STEP 1810 H= 452:30 +CPU= 29.134
  17:16:19 STEP 1811 H= 452:45 +CPU= 25.890
  17:16:51 STEP 1812 H= 453:00 +CPU= 32.186
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001812
  17:17:18 STEP 1813 H= 453:15 +CPU= 26.824
  17:17:45 STEP 1814 H= 453:30 +CPU= 26.669
  17:18:12 STEP 1815 H= 453:45 +CPU= 27.647
  17:18:41 STEP 1816 H= 454:00 +CPU= 28.350
  17:19:08 STEP 1817 H= 454:15 +CPU= 27.002
  17:19:35 STEP 1818 H= 454:30 +CPU= 27.221
  17:20:04 STEP 1819 H= 454:45 +CPU= 28.904
  17:20:34 STEP 1820 H= 455:00 +CPU= 29.759
  17:21:04 STEP 1821 H= 455:15 +CPU= 29.617
  17:21:33 STEP 1822 H= 455:30 +CPU= 28.957
  17:22:03 STEP 1823 H= 455:45 +CPU= 29.270
  17:22:43 STEP 1824 H= 456:00 +CPU= 39.136
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001824
  17:23:13 STEP 1825 H= 456:15 +CPU= 28.938
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001824
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0_r1890634521_18.zip
Uploading the intermediate file: upload_file_18.zip
  17:23:43 STEP 1826 H= 456:30 +CPU= 28.537
  17:24:12 STEP 1827 H= 456:45 +CPU= 29.438
  17:24:42 STEP 1828 H= 457:00 +CPU= 29.798
  17:25:12 STEP 1829 H= 457:15 +CPU= 29.542
  17:25:41 STEP 1830 H= 457:30 +CPU= 29.139
  17:26:10 STEP 1831 H= 457:45 +CPU= 28.996
  17:26:40 STEP 1832 H= 458:00 +CPU= 29.520
  17:27:10 STEP 1833 H= 458:15 +CPU= 29.293
  17:27:40 STEP 1834 H= 458:30 +CPU= 27.788
  17:28:10 STEP 1835 H= 458:45 +CPU= 29.017
  17:28:48 STEP 1836 H= 459:00 +CPU= 36.434
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001836
  17:29:18 STEP 1837 H= 459:15 +CPU= 29.163
  17:29:48 STEP 1838 H= 459:30 +CPU= 29.476
  17:30:18 STEP 1839 H= 459:45 +CPU= 29.358
  17:30:48 STEP 1840 H= 460:00 +CPU= 29.734
  17:31:17 STEP 1841 H= 460:15 +CPU= 28.513
  17:31:46 STEP 1842 H= 460:30 +CPU= 29.296
  17:32:17 STEP 1843 H= 460:45 +CPU= 29.791
  17:32:48 STEP 1844 H= 461:00 +CPU= 30.257
  17:33:17 STEP 1845 H= 461:15 +CPU= 28.767
  17:33:47 STEP 1846 H= 461:30 +CPU= 29.258
  17:34:16 STEP 1847 H= 461:45 +CPU= 29.200
  17:34:53 STEP 1848 H= 462:00 +CPU= 36.695
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001848
  17:35:22 STEP 1849 H= 462:15 +CPU= 29.291
  17:35:53 STEP 1850 H= 462:30 +CPU= 29.610
  17:36:23 STEP 1851 H= 462:45 +CPU= 29.217
  17:36:53 STEP 1852 H= 463:00 +CPU= 30.072
  17:37:22 STEP 1853 H= 463:15 +CPU= 29.162
  17:37:51 STEP 1854 H= 463:30 +CPU= 28.640
  17:38:21 STEP 1855 H= 463:45 +CPU= 29.731
  17:38:51 STEP 1856 H= 464:00 +CPU= 29.692
  17:39:21 STEP 1857 H= 464:15 +CPU= 29.223
  17:39:51 STEP 1858 H= 464:30 +CPU= 27.947
  17:40:21 STEP 1859 H= 464:45 +CPU= 29.655
  17:40:57 STEP 1860 H= 465:00 +CPU= 35.821
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001860
  17:41:27 STEP 1861 H= 465:15 +CPU= 29.165
  17:41:57 STEP 1862 H= 465:30 +CPU= 28.085
  17:42:28 STEP 1863 H= 465:45 +CPU= 29.270
  17:42:59 STEP 1864 H= 466:00 +CPU= 30.228
  17:43:29 STEP 1865 H= 466:15 +CPU= 29.524
  17:44:00 STEP 1866 H= 466:30 +CPU= 29.993
  17:44:30 STEP 1867 H= 466:45 +CPU= 29.048
  17:45:00 STEP 1868 H= 467:00 +CPU= 29.516
  17:45:31 STEP 1869 H= 467:15 +CPU= 29.653
  17:46:01 STEP 1870 H= 467:30 +CPU= 29.870
  17:46:29 STEP 1871 H= 467:45 +CPU= 27.222
  17:47:03 STEP 1872 H= 468:00 +CPU= 34.638
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001872
  17:47:31 STEP 1873 H= 468:15 +CPU= 28.132
  17:47:56 STEP 1874 H= 468:30 +CPU= 24.806
  17:48:25 STEP 1875 H= 468:45 +CPU= 28.269
  17:48:53 STEP 1876 H= 469:00 +CPU= 28.490
  17:49:19 STEP 1877 H= 469:15 +CPU= 25.458
  17:49:47 STEP 1878 H= 469:30 +CPU= 28.658
  17:50:16 STEP 1879 H= 469:45 +CPU= 28.410
  17:50:47 STEP 1880 H= 470:00 +CPU= 30.317
  17:51:16 STEP 1881 H= 470:15 +CPU= 29.300
  17:51:47 STEP 1882 H= 470:30 +CPU= 29.682
  17:52:17 STEP 1883 H= 470:45 +CPU= 29.565
  17:52:52 STEP 1884 H= 471:00 +CPU= 35.237
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001884
  17:53:23 STEP 1885 H= 471:15 +CPU= 29.433
  17:53:52 STEP 1886 H= 471:30 +CPU= 29.340
  17:54:22 STEP 1887 H= 471:45 +CPU= 29.554
  17:54:53 STEP 1888 H= 472:00 +CPU= 30.090
  17:55:22 STEP 1889 H= 472:15 +CPU= 29.258
  17:55:52 STEP 1890 H= 472:30 +CPU= 29.082
  17:56:21 STEP 1891 H= 472:45 +CPU= 28.814
  17:56:51 STEP 1892 H= 473:00 +CPU= 29.805
  17:57:21 STEP 1893 H= 473:15 +CPU= 29.319
  17:57:50 STEP 1894 H= 473:30 +CPU= 29.597
  17:58:20 STEP 1895 H= 473:45 +CPU= 29.520
  17:58:57 STEP 1896 H= 474:00 +CPU= 36.111
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001896
  17:59:27 STEP 1897 H= 474:15 +CPU= 29.398
  17:59:57 STEP 1898 H= 474:30 +CPU= 29.325
  18:00:19 STEP 1899 H= 474:45 +CPU= 21.653
  18:00:41 STEP 1900 H= 475:00 +CPU= 22.501
  18:01:04 STEP 1901 H= 475:15 +CPU= 23.135
  18:01:31 STEP 1902 H= 475:30 +CPU= 26.821
  18:01:58 STEP 1903 H= 475:45 +CPU= 26.160
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a03n
(argv4) batchid: 1018
(argv5) wuid: 12288458
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d1ab9818117a2d7a01d99b3bbd8a54e6 to: /var/lib/boinc-client/slots/8/oifs_43r3_bl_a03n_2016092300_20_1018_12288458.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_bl_a03n_2016092300_20_1018_12288458.zip
ic_ancil_file: ic_ancil_12288458
ifsdata_file: ifsdata_12288458
climate_data_file: clim_data_12288458
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/8/ic_ancil_12288458.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12288458.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3c1088f64d20123e804782f4dcfe577a to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12288458.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12288458.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12288458.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12288458.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12288458.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12288458.xml
-- Model is restarting --
Adjusting last_iter, 1904, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12288458.xml
last_cpu_time: 54504
upload_file_number: 19
last_iter: 1873
last_upload: 1642500
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0_r1890634521
The child process has been launched with process id: 860046
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:860046:860046] [20240614:190016:1718391616:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:00:47 STEP 1872 H= 468:00 +CPU= 26.445
  19:01:11 STEP 1873 H= 468:15 +CPU= 24.720
  19:01:35 STEP 1874 H= 468:30 +CPU= 23.674
  19:01:59 STEP 1875 H= 468:45 +CPU= 24.245
  19:02:27 STEP 1876 H= 469:00 +CPU= 27.270
  19:02:51 STEP 1877 H= 469:15 +CPU= 23.854
  19:03:20 STEP 1878 H= 469:30 +CPU= 28.357
  19:03:50 STEP 1879 H= 469:45 +CPU= 29.477
  19:04:20 STEP 1880 H= 470:00 +CPU= 29.982
  19:04:50 STEP 1881 H= 470:15 +CPU= 29.340
  19:05:19 STEP 1882 H= 470:30 +CPU= 29.297
  19:05:45 STEP 1883 H= 470:45 +CPU= 26.027
  19:06:11 STEP 1884 H= 471:00 +CPU= 25.408
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001884
  19:06:32 STEP 1885 H= 471:15 +CPU= 21.470
  19:06:58 STEP 1886 H= 471:30 +CPU= 25.768
  19:07:23 STEP 1887 H= 471:45 +CPU= 24.230
  19:07:48 STEP 1888 H= 472:00 +CPU= 25.484
  19:08:15 STEP 1889 H= 472:15 +CPU= 26.691
  19:08:40 STEP 1890 H= 472:30 +CPU= 25.272
  19:09:09 STEP 1891 H= 472:45 +CPU= 28.913
  19:09:39 STEP 1892 H= 473:00 +CPU= 29.711
  19:10:09 STEP 1893 H= 473:15 +CPU= 29.724
  19:10:39 STEP 1894 H= 473:30 +CPU= 29.629
  19:11:08 STEP 1895 H= 473:45 +CPU= 29.346
  19:11:44 STEP 1896 H= 474:00 +CPU= 35.598
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001896
  19:12:15 STEP 1897 H= 474:15 +CPU= 29.114
  19:12:45 STEP 1898 H= 474:30 +CPU= 29.439
  19:13:14 STEP 1899 H= 474:45 +CPU= 28.644
  19:13:46 STEP 1900 H= 475:00 +CPU= 29.367
  19:14:15 STEP 1901 H= 475:15 +CPU= 29.337
  19:14:45 STEP 1902 H= 475:30 +CPU= 29.175
  19:15:15 STEP 1903 H= 475:45 +CPU= 29.300
  19:15:45 STEP 1904 H= 476:00 +CPU= 29.802
  19:16:15 STEP 1905 H= 476:15 +CPU= 29.334
  19:16:44 STEP 1906 H= 476:30 +CPU= 28.916
  19:17:08 STEP 1907 H= 476:45 +CPU= 24.640
  19:17:36 STEP 1908 H= 477:00 +CPU= 27.959
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001908
  19:18:00 STEP 1909 H= 477:15 +CPU= 23.904
  19:18:25 STEP 1910 H= 477:30 +CPU= 24.288
  19:18:50 STEP 1911 H= 477:45 +CPU= 25.073
  19:19:17 STEP 1912 H= 478:00 +CPU= 26.904
  19:19:42 STEP 1913 H= 478:15 +CPU= 25.398
  19:20:10 STEP 1914 H= 478:30 +CPU= 27.451
  19:20:41 STEP 1915 H= 478:45 +CPU= 29.873
  19:21:11 STEP 1916 H= 479:00 +CPU= 30.450
  19:21:42 STEP 1917 H= 479:15 +CPU= 30.022
  19:22:12 STEP 1918 H= 479:30 +CPU= 30.089
  19:22:42 STEP 1919 H= 479:45 +CPU= 29.721
  19:23:19 STEP 1920 H= 480:00 +CPU= 36.989
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1224434521E-21      I       0.1077322922E-05      I
ICloud fractionI      0.0000000000E+00      I       0.2782999324E-01      I
I  Relat. Hum. I      0.2066617693E+00      I       0.3076542534E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3412137657E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    311343412 BYTES IN      7 RECORDS TRANSFERRED IN    0.0010 SECONDS311343.4 Mbytes/s, TOTAL TIME=    0.0158( 6.3%)
 19:23:19 STEP 1920 H= 480:00 +CPU= 36.989
 END CNT3
 NSTEP =  1920 CNT0   000000000  27.946  27.946  28.456   0.982               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************     4.0     1.8     2.3     2.6     4.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1383311.1 1383311.1    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1379150.0 1379150.0     99.70      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    5145.3    5145.3     19.71      0.00
   9 SPCM     - SPECTRAL COMP.                      48     849.7     849.7      2.95      0.00
  10 SCAN2M - PHYSICS                               49   12558.4   12558.4     44.48      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4     819.3     819.3      0.24      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      88.4      88.4      0.02      0.00
  17 GRIDFPOS IN CNT4                                4      30.5      30.5      0.01      0.00
  27 RESTART FILES - WRITING                         1    3268.9    3268.9      0.24      0.00
  28 RESTART FILES - READING                         1    3626.3    3626.3      0.26      0.00
  29 SU4FPOS IN CNT4                                 4       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    5012.4    5012.4      1.45      0.00
  31 POSDDH IN STEPO                                 1      11.5      11.5      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    5959.7    5959.7     21.11      0.00
  39 SU0YOMB                                         1     286.2     286.2      0.02      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      29.2      29.2      0.10      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     508.9     508.9      1.77      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      63.4      63.4      0.22      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     361.4     361.4      2.82      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     219.2     219.2      1.58      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      70.4      70.4      0.51      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     105.8     105.8      0.83      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      40.7      40.7      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      53.7      53.7      0.42      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      30.9      30.9      0.22      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108     112.8     112.8      0.88      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      55.5      55.5      0.40      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1383.3 CPU TIME    1369.2 VECTOR TIME     1369.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1252.9
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/8/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/8/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288458/ICMUAh7zg+001848
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0_r1890634521_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
19:26:48 (860037): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Jun 2024 20:20:50 1537687 22442013 oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0 1,727,100 55,873 0.0324
14 Jun 2024 14:04:29 1537687 22442013 oifs_43r3_bl_a03n_2016092300_20_1018_12288458_0 1,296,900 39,356 0.0303


©2024 climateprediction.net