climateprediction.net home page
Task 22444858

Task 22444858

Name oifs_43r3_bl_a29g_2016092300_20_1018_12291259_0
Workunit 12291259
Created 13 Jun 2024, 0:17:04 UTC
Sent 14 Jun 2024, 5:20:10 UTC
Report deadline 13 Aug 2024, 5:20:10 UTC
Received 16 Jun 2024, 9:34:08 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1496046
Run time 15 hours 31 min 26 sec
CPU time 15 hours 31 min 26 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 5.56 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,017.80 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,389.51 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
K_PROFILE_LIMIT=-10.000
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:865257:865257] [20240616:010756:1718518076:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:08:26 STEP 1536 H= 384:00 +CPU= 25.903
  01:08:52 STEP 1537 H= 384:15 +CPU= 25.901
  01:09:20 STEP 1538 H= 384:30 +CPU= 27.567
  01:09:47 STEP 1539 H= 384:45 +CPU= 27.278
  01:10:15 STEP 1540 H= 385:00 +CPU= 28.004
  01:10:41 STEP 1541 H= 385:15 +CPU= 26.375
  01:11:09 STEP 1542 H= 385:30 +CPU= 27.261
  01:11:36 STEP 1543 H= 385:45 +CPU= 27.334
  01:12:06 STEP 1544 H= 386:00 +CPU= 29.765
  01:12:34 STEP 1545 H= 386:15 +CPU= 28.124
  01:13:01 STEP 1546 H= 386:30 +CPU= 26.515
  01:13:29 STEP 1547 H= 386:45 +CPU= 28.437
  01:14:03 STEP 1548 H= 387:00 +CPU= 34.212
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001548
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001548
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001548
  01:14:31 STEP 1549 H= 387:15 +CPU= 27.363
  01:14:59 STEP 1550 H= 387:30 +CPU= 27.931
  01:15:28 STEP 1551 H= 387:45 +CPU= 28.665
  01:15:56 STEP 1552 H= 388:00 +CPU= 28.567
  01:16:24 STEP 1553 H= 388:15 +CPU= 27.472
  01:16:52 STEP 1554 H= 388:30 +CPU= 27.634
  01:17:19 STEP 1555 H= 388:45 +CPU= 27.679
  01:17:48 STEP 1556 H= 389:00 +CPU= 28.453
  01:18:16 STEP 1557 H= 389:15 +CPU= 27.837
  01:18:43 STEP 1558 H= 389:30 +CPU= 27.626
  01:19:12 STEP 1559 H= 389:45 +CPU= 28.174
  01:19:46 STEP 1560 H= 390:00 +CPU= 34.355
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001560
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001560
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001560
  01:20:15 STEP 1561 H= 390:15 +CPU= 28.331
  01:20:43 STEP 1562 H= 390:30 +CPU= 27.918
  01:21:11 STEP 1563 H= 390:45 +CPU= 28.146
  01:21:40 STEP 1564 H= 391:00 +CPU= 29.297
  01:22:07 STEP 1565 H= 391:15 +CPU= 27.010
  01:22:35 STEP 1566 H= 391:30 +CPU= 27.463
  01:23:02 STEP 1567 H= 391:45 +CPU= 27.323
  01:23:30 STEP 1568 H= 392:00 +CPU= 27.948
  01:23:59 STEP 1569 H= 392:15 +CPU= 28.366
  01:24:26 STEP 1570 H= 392:30 +CPU= 27.555
  01:24:55 STEP 1571 H= 392:45 +CPU= 28.282
  01:25:28 STEP 1572 H= 393:00 +CPU= 33.619
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001572
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001572
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001572
  01:25:58 STEP 1573 H= 393:15 +CPU= 29.352
  01:26:26 STEP 1574 H= 393:30 +CPU= 28.248
  01:26:54 STEP 1575 H= 393:45 +CPU= 27.805
  01:27:23 STEP 1576 H= 394:00 +CPU= 28.665
  01:27:50 STEP 1577 H= 394:15 +CPU= 27.170
  01:28:18 STEP 1578 H= 394:30 +CPU= 27.714
  01:28:46 STEP 1579 H= 394:45 +CPU= 28.273
  01:29:15 STEP 1580 H= 395:00 +CPU= 29.217
  01:29:44 STEP 1581 H= 395:15 +CPU= 28.250
  01:30:11 STEP 1582 H= 395:30 +CPU= 27.507
  01:30:38 STEP 1583 H= 395:45 +CPU= 26.794
  01:31:14 STEP 1584 H= 396:00 +CPU= 36.078
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001584
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001584
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001584
  01:31:39 STEP 1585 H= 396:15 +CPU= 24.540
  01:32:05 STEP 1586 H= 396:30 +CPU= 25.889
  01:32:30 STEP 1587 H= 396:45 +CPU= 25.339
  01:32:58 STEP 1588 H= 397:00 +CPU= 27.470
  01:33:23 STEP 1589 H= 397:15 +CPU= 25.653
  01:33:51 STEP 1590 H= 397:30 +CPU= 27.501
  01:34:18 STEP 1591 H= 397:45 +CPU= 27.252
  01:34:45 STEP 1592 H= 398:00 +CPU= 27.026
  01:35:12 STEP 1593 H= 398:15 +CPU= 26.445
  01:35:38 STEP 1594 H= 398:30 +CPU= 26.366
  01:36:05 STEP 1595 H= 398:45 +CPU= 26.613
  01:36:39 STEP 1596 H= 399:00 +CPU= 33.885
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001596
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001596
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001596
  01:37:05 STEP 1597 H= 399:15 +CPU= 25.968
  01:37:33 STEP 1598 H= 399:30 +CPU= 28.336
  01:38:01 STEP 1599 H= 399:45 +CPU= 27.548
  01:38:30 STEP 1600 H= 400:00 +CPU= 28.579
  01:38:57 STEP 1601 H= 400:15 +CPU= 26.745
  01:39:23 STEP 1602 H= 400:30 +CPU= 26.184
  01:39:49 STEP 1603 H= 400:45 +CPU= 26.173
  01:40:17 STEP 1604 H= 401:00 +CPU= 27.999
  01:40:44 STEP 1605 H= 401:15 +CPU= 27.006
  01:41:12 STEP 1606 H= 401:30 +CPU= 27.912
  01:41:40 STEP 1607 H= 401:45 +CPU= 28.147
  01:42:14 STEP 1608 H= 402:00 +CPU= 33.971
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001608
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001608
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001608
  01:42:41 STEP 1609 H= 402:15 +CPU= 26.954
  01:43:08 STEP 1610 H= 402:30 +CPU= 26.965
  01:43:35 STEP 1611 H= 402:45 +CPU= 26.431
  01:44:01 STEP 1612 H= 403:00 +CPU= 26.323
  01:44:28 STEP 1613 H= 403:15 +CPU= 26.842
  01:44:55 STEP 1614 H= 403:30 +CPU= 26.293
  01:45:23 STEP 1615 H= 403:45 +CPU= 27.930
  01:45:50 STEP 1616 H= 404:00 +CPU= 27.413
  01:46:17 STEP 1617 H= 404:15 +CPU= 27.142
  01:46:45 STEP 1618 H= 404:30 +CPU= 27.561
  01:47:12 STEP 1619 H= 404:45 +CPU= 27.290
  01:47:47 STEP 1620 H= 405:00 +CPU= 35.075
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001620
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001620
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001620
  01:48:14 STEP 1621 H= 405:15 +CPU= 26.604
  01:48:39 STEP 1622 H= 405:30 +CPU= 25.305
  01:49:07 STEP 1623 H= 405:45 +CPU= 27.606
  01:49:35 STEP 1624 H= 406:00 +CPU= 27.545
  01:50:02 STEP 1625 H= 406:15 +CPU= 27.451
  01:50:29 STEP 1626 H= 406:30 +CPU= 27.135
  01:50:56 STEP 1627 H= 406:45 +CPU= 26.659
  01:51:24 STEP 1628 H= 407:00 +CPU= 27.626
  01:51:50 STEP 1629 H= 407:15 +CPU= 26.220
  01:52:16 STEP 1630 H= 407:30 +CPU= 26.109
  01:52:43 STEP 1631 H= 407:45 +CPU= 26.601
  01:53:20 STEP 1632 H= 408:00 +CPU= 37.131
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001632
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001632
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001620
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001620
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001620
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a29g_2016092300_20_1018_12291259_0_r883522508_16.zip
Uploading the intermediate file: upload_file_16.zip
  01:53:47 STEP 1633 H= 408:15 +CPU= 27.070
  01:54:13 STEP 1634 H= 408:30 +CPU= 25.138
  01:54:40 STEP 1635 H= 408:45 +CPU= 27.236
  01:55:07 STEP 1636 H= 409:00 +CPU= 27.383
  01:55:33 STEP 1637 H= 409:15 +CPU= 26.035
  01:56:01 STEP 1638 H= 409:30 +CPU= 27.929
  01:56:29 STEP 1639 H= 409:45 +CPU= 27.546
  01:56:57 STEP 1640 H= 410:00 +CPU= 27.726
  01:57:23 STEP 1641 H= 410:15 +CPU= 26.339
  01:57:50 STEP 1642 H= 410:30 +CPU= 26.854
  01:58:17 STEP 1643 H= 410:45 +CPU= 26.444
  01:58:50 STEP 1644 H= 411:00 +CPU= 33.204
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001644
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001644
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001644
  01:59:18 STEP 1645 H= 411:15 +CPU= 27.515
  01:59:43 STEP 1646 H= 411:30 +CPU= 25.818
  02:00:10 STEP 1647 H= 411:45 +CPU= 26.870
  02:00:38 STEP 1648 H= 412:00 +CPU= 27.190
  02:01:04 STEP 1649 H= 412:15 +CPU= 26.380
  02:01:32 STEP 1650 H= 412:30 +CPU= 27.438
  02:02:00 STEP 1651 H= 412:45 +CPU= 28.351
  02:02:29 STEP 1652 H= 413:00 +CPU= 28.871
  02:02:57 STEP 1653 H= 413:15 +CPU= 27.584
  02:03:25 STEP 1654 H= 413:30 +CPU= 28.423
  02:03:51 STEP 1655 H= 413:45 +CPU= 26.268
  02:04:25 STEP 1656 H= 414:00 +CPU= 33.311
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001656
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001656
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001656
  02:04:52 STEP 1657 H= 414:15 +CPU= 26.725
  02:05:20 STEP 1658 H= 414:30 +CPU= 28.005
  02:05:48 STEP 1659 H= 414:45 +CPU= 28.041
  02:06:15 STEP 1660 H= 415:00 +CPU= 27.055
  02:06:42 STEP 1661 H= 415:15 +CPU= 26.634
  02:07:09 STEP 1662 H= 415:30 +CPU= 27.828
  02:07:37 STEP 1663 H= 415:45 +CPU= 27.085
  02:08:05 STEP 1664 H= 416:00 +CPU= 28.277
  02:08:32 STEP 1665 H= 416:15 +CPU= 27.073
  02:08:59 STEP 1666 H= 416:30 +CPU= 26.553
  02:09:25 STEP 1667 H= 416:45 +CPU= 26.078
  02:09:58 STEP 1668 H= 417:00 +CPU= 33.603
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001668
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001668
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001668
  02:10:26 STEP 1669 H= 417:15 +CPU= 27.577
  02:10:54 STEP 1670 H= 417:30 +CPU= 27.451
  02:11:20 STEP 1671 H= 417:45 +CPU= 26.630
  02:11:47 STEP 1672 H= 418:00 +CPU= 26.249
  02:12:13 STEP 1673 H= 418:15 +CPU= 26.626
  02:12:40 STEP 1674 H= 418:30 +CPU= 26.677
  02:13:09 STEP 1675 H= 418:45 +CPU= 28.465
  02:13:36 STEP 1676 H= 419:00 +CPU= 27.805
  02:14:03 STEP 1677 H= 419:15 +CPU= 26.555
  02:14:30 STEP 1678 H= 419:30 +CPU= 26.704
  02:14:56 STEP 1679 H= 419:45 +CPU= 26.108
  02:15:32 STEP 1680 H= 420:00 +CPU= 36.068
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001680
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001680
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001680
  02:15:58 STEP 1681 H= 420:15 +CPU= 25.653
  02:16:25 STEP 1682 H= 420:30 +CPU= 26.756
  02:16:52 STEP 1683 H= 420:45 +CPU= 26.900
  02:17:19 STEP 1684 H= 421:00 +CPU= 27.463
  02:17:47 STEP 1685 H= 421:15 +CPU= 27.308
  02:18:14 STEP 1686 H= 421:30 +CPU= 27.506
  02:18:41 STEP 1687 H= 421:45 +CPU= 26.472
  02:19:08 STEP 1688 H= 422:00 +CPU= 26.924
  02:19:34 STEP 1689 H= 422:15 +CPU= 26.485
  02:20:01 STEP 1690 H= 422:30 +CPU= 26.648
  02:20:28 STEP 1691 H= 422:45 +CPU= 26.623
  02:21:01 STEP 1692 H= 423:00 +CPU= 33.249
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001692
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001692
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001692
  02:21:29 STEP 1693 H= 423:15 +CPU= 27.469
  02:21:55 STEP 1694 H= 423:30 +CPU= 26.765
  02:22:22 STEP 1695 H= 423:45 +CPU= 26.305
  02:22:49 STEP 1696 H= 424:00 +CPU= 27.539
  02:23:16 STEP 1697 H= 424:15 +CPU= 26.738
  02:23:43 STEP 1698 H= 424:30 +CPU= 26.391
  02:24:09 STEP 1699 H= 424:45 +CPU= 26.186
  02:24:36 STEP 1700 H= 425:00 +CPU= 27.096
  02:25:04 STEP 1701 H= 425:15 +CPU= 27.573
  02:25:31 STEP 1702 H= 425:30 +CPU= 26.741
  02:25:57 STEP 1703 H= 425:45 +CPU= 26.629
  02:26:31 STEP 1704 H= 426:00 +CPU= 33.734
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001704
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001704
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001704
  02:26:58 STEP 1705 H= 426:15 +CPU= 26.562
  02:27:23 STEP 1706 H= 426:30 +CPU= 25.731
  02:27:50 STEP 1707 H= 426:45 +CPU= 26.888
  02:28:18 STEP 1708 H= 427:00 +CPU= 28.040
  02:28:46 STEP 1709 H= 427:15 +CPU= 27.292
  02:29:14 STEP 1710 H= 427:30 +CPU= 27.946
  02:29:41 STEP 1711 H= 427:45 +CPU= 26.735
  02:30:08 STEP 1712 H= 428:00 +CPU= 27.203
  02:30:35 STEP 1713 H= 428:15 +CPU= 27.248
  02:31:03 STEP 1714 H= 428:30 +CPU= 27.765
  02:31:31 STEP 1715 H= 428:45 +CPU= 28.122
  02:32:05 STEP 1716 H= 429:00 +CPU= 33.398
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001716
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001716
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001716
  02:32:32 STEP 1717 H= 429:15 +CPU= 27.318
  02:33:00 STEP 1718 H= 429:30 +CPU= 27.735
  02:33:27 STEP 1719 H= 429:45 +CPU= 27.039
  02:33:55 STEP 1720 H= 430:00 +CPU= 27.509
  02:34:22 STEP 1721 H= 430:15 +CPU= 27.145
  02:34:47 STEP 1722 H= 430:30 +CPU= 25.427
  02:35:14 STEP 1723 H= 430:45 +CPU= 26.608
  02:35:41 STEP 1724 H= 431:00 +CPU= 27.486
  02:36:07 STEP 1725 H= 431:15 +CPU= 25.267
  02:36:32 STEP 1726 H= 431:30 +CPU= 25.037
  02:37:00 STEP 1727 H= 431:45 +CPU= 27.778
  02:37:37 STEP 1728 H= 432:00 +CPU= 37.715
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001728
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001728
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001716
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001716
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001716
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a29g_2016092300_20_1018_12291259_0_r883522508_17.zip
Uploading the intermediate file: upload_file_17.zip
  02:38:05 STEP 1729 H= 432:15 +CPU= 27.156
  02:38:32 STEP 1730 H= 432:30 +CPU= 27.633
  02:39:00 STEP 1731 H= 432:45 +CPU= 27.361
  02:39:28 STEP 1732 H= 433:00 +CPU= 28.109
  02:39:54 STEP 1733 H= 433:15 +CPU= 26.248
  02:40:21 STEP 1734 H= 433:30 +CPU= 26.931
  02:40:48 STEP 1735 H= 433:45 +CPU= 26.716
  02:41:15 STEP 1736 H= 434:00 +CPU= 27.109
  02:41:43 STEP 1737 H= 434:15 +CPU= 27.654
  02:42:10 STEP 1738 H= 434:30 +CPU= 27.143
  02:42:37 STEP 1739 H= 434:45 +CPU= 26.655
  02:43:10 STEP 1740 H= 435:00 +CPU= 32.674
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001740
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001740
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001740
  02:43:35 STEP 1741 H= 435:15 +CPU= 25.718
  02:44:02 STEP 1742 H= 435:30 +CPU= 26.424
  02:44:29 STEP 1743 H= 435:45 +CPU= 26.790
  02:44:57 STEP 1744 H= 436:00 +CPU= 27.939
  02:45:23 STEP 1745 H= 436:15 +CPU= 26.758
  02:45:51 STEP 1746 H= 436:30 +CPU= 27.038
  02:46:18 STEP 1747 H= 436:45 +CPU= 27.419
  02:46:48 STEP 1748 H= 437:00 +CPU= 29.589
  02:47:15 STEP 1749 H= 437:15 +CPU= 27.514
  02:47:43 STEP 1750 H= 437:30 +CPU= 27.415
  02:48:09 STEP 1751 H= 437:45 +CPU= 26.556
  02:48:45 STEP 1752 H= 438:00 +CPU= 35.113
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001752
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001752
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001752
  02:49:11 STEP 1753 H= 438:15 +CPU= 26.390
  02:49:39 STEP 1754 H= 438:30 +CPU= 28.361
  02:50:07 STEP 1755 H= 438:45 +CPU= 27.914
  02:50:35 STEP 1756 H= 439:00 +CPU= 27.108
  02:51:02 STEP 1757 H= 439:15 +CPU= 27.304
  02:51:31 STEP 1758 H= 439:30 +CPU= 28.554
  02:51:58 STEP 1759 H= 439:45 +CPU= 27.278
  02:52:27 STEP 1760 H= 440:00 +CPU= 28.808
  02:52:55 STEP 1761 H= 440:15 +CPU= 28.480
  02:53:23 STEP 1762 H= 440:30 +CPU= 27.632
  02:53:50 STEP 1763 H= 440:45 +CPU= 27.232
  02:54:24 STEP 1764 H= 441:00 +CPU= 33.563
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001764
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001764
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001764
  02:54:50 STEP 1765 H= 441:15 +CPU= 26.010
  02:55:18 STEP 1766 H= 441:30 +CPU= 28.102
  02:55:43 STEP 1767 H= 441:45 +CPU= 25.161
  02:56:12 STEP 1768 H= 442:00 +CPU= 28.216
  02:56:38 STEP 1769 H= 442:15 +CPU= 26.164
  02:57:07 STEP 1770 H= 442:30 +CPU= 29.121
  02:57:35 STEP 1771 H= 442:45 +CPU= 28.330
  02:58:03 STEP 1772 H= 443:00 +CPU= 27.934
  02:58:30 STEP 1773 H= 443:15 +CPU= 26.156
  02:58:57 STEP 1774 H= 443:30 +CPU= 26.927
  02:59:23 STEP 1775 H= 443:45 +CPU= 26.291
  02:59:59 STEP 1776 H= 444:00 +CPU= 36.055
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001776
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001776
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001776
  03:00:25 STEP 1777 H= 444:15 +CPU= 25.798
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.131972046349518E-06 0.000000000000000E+00 0.579444666303657E-03
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.121026269057032E-06 0.000000000000000E+00 0.444094434700986E-03
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.141095571980028E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.236999132516828E-05 0.000000000000000E+00 0.807938972143875E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.118228322810070E-06 0.000000000000000E+00 0.360464097465774E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.141206871883829E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.502580251878827E-06 0.000000000000000E+00 0.195881258684477E-02
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.708129073950324E-07 0.000000000000000E+00 0.299548228543690E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.190677862023145E-05 0.000000000000000E+00 0.318396519530233E-02
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.200212667296594E-06 0.000000000000000E+00 0.366102024298094E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.511272674359836E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1776 STEPO   F00000000
 CREATE RESTART FILE ./srf00181200.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00181200.0001 MODE=w
 IO-STREAM CLOSED - ./srf00181200.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    2.3885 SECONDS******** Mbytes/s, TOTAL TIME=    3.0482(78.4%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00180000.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1776 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001776 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001776
 MPI-TASK:   1 -    421022916 BYTES IN      6 RECORDS TRANSFERRED IN    0.0007 SECONDS601461.3 Mbytes/s, TOTAL TIME=    0.0106( 6.6%)
 NSTEP =  1776 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001776 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001776
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0107 SECONDS 66112.1 Mbytes/s, TOTAL TIME=    0.1911( 5.6%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001776 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001776
 MPI-TASK:   1 -   6736366656 BYTES IN    125 RECORDS TRANSFERRED IN    0.0062 SECONDS******** Mbytes/s, TOTAL TIME=    0.1525( 4.1%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001776 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001776
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0031( 6.5%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001776 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001776
 MPI-TASK:   1 -    412443187 BYTES IN      4 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0050( 6.0%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1776 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1776 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001776 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001776
 MPI-TASK:   1 -    421022916 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS701704.9 Mbytes/s, TOTAL TIME=    0.0107( 5.6%)
 02:59:59 STEP 1776 H= 444:00 +CPU= 36.055
 NSTEP =  1777 STEPO   0AAA00AAA
 03:00:25 STEP 1777 H= 444:15 +CPU= 25.798
 NSTEP =  1778 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 02:59:27 0AA000000 STEPO     1776     3.628    3.628    3.641    111:14    111:30 0.26882979228400E-05       2GB       0MB
 02:59:28 FULLPOS-B DYNFPOS   1776     1.644    1.644    1.651    111:16    111:32 0.26882979228400E-05       2GB       0MB
 02:59:31 FULLPOS-V DYNFPOS   1776     2.883    2.883    2.888    111:19    111:34 0.26882979228400E-05       2GB       0MB
 02:59:32 FULLPOS-T DYNFPOS   1776     0.474    0.474    0.474    111:19    111:35 0.26882979228400E-05       2GB       0MB
 02:59:32 0AAA00AAA STEPO     1776     0.462    0.462    0.462    111:20    111:35 0.26882979228400E-05       2GB       0MB
 02:59:59 0AAA00AAA STEPO     1777    26.817   26.817   26.863    111:47    112:02 0.26882979228400E-05       2GB       0MB
 03:00:25 0AAA00AAA STEPO     1778    25.798   25.798   25.833    112:12    112:28 0.26882979228400E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1776",
 CTIME   = "00181200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1764, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /boinc_data/boinc/slots/3/progress_file_12291259.xml

<running_values>
  <last_cpu_time>51990.600000</last_cpu_time>
  <upload_file_number>18</upload_file_number>
  <last_iter>1778</last_iter>
  <last_upload>1555200</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a29g
(argv4) batchid: 1018
(argv5) wuid: 12291259
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /boinc_data/boinc/slots/3
Project directory is: /boinc_data/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259
..mkdir for temp folder for results failed
Copying: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /boinc_data/boinc/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /boinc_data/boinc/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ad0490f04dc6ae64f10896ed1f61bbf4 to: /boinc_data/boinc/slots/3/oifs_43r3_bl_a29g_2016092300_20_1018_12291259.zip
Unzipping the namelist zip file: /boinc_data/boinc/slots/3/oifs_43r3_bl_a29g_2016092300_20_1018_12291259.zip
ic_ancil_file: ic_ancil_12291259
ifsdata_file: ifsdata_12291259
climate_data_file: clim_data_12291259
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /boinc_data/boinc/slots/3/ic_ancil_12291259.zip
Unzipping the IC ancils zip file: /boinc_data/boinc/slots/3/ic_ancil_12291259.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e290feb532a78dce9ccdc9c05426be9b to: /boinc_data/boinc/slots/3/ifsdata/ifsdata_12291259.zip
Unzipping the ifsdata_zip file: /boinc_data/boinc/slots/3/ifsdata/ifsdata_12291259.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /boinc_data/boinc/slots/3/159l_2/clim_data_12291259.zip
Unzipping the climate data zip file: /boinc_data/boinc/slots/3/159l_2/clim_data_12291259.zip
Checking for progress XML file: /boinc_data/boinc/slots/3/progress_file_12291259.xml
Opened progress file ok : /boinc_data/boinc/slots/3/progress_file_12291259.xml
-- Model is restarting --
Adjusting last_iter, 1778, to previous model restart step.
Creating progress file: /boinc_data/boinc/slots/3/progress_file_12291259.xml
last_cpu_time: 51990
upload_file_number: 18
last_iter: 1777
last_upload: 1555200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a29g_2016092300_20_1018_12291259_0_r883522508
The child process has been launched with process id: 871887
Executing the command: /boinc_data/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871887:871887] [20240616:030809:1718525289:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:08:39 STEP 1776 H= 444:00 +CPU= 25.830
  03:09:04 STEP 1777 H= 444:15 +CPU= 25.225
  03:09:29 STEP 1778 H= 444:30 +CPU= 24.765
  03:09:53 STEP 1779 H= 444:45 +CPU= 24.395
  03:10:17 STEP 1780 H= 445:00 +CPU= 23.822
  03:10:37 STEP 1781 H= 445:15 +CPU= 19.577
  03:11:00 STEP 1782 H= 445:30 +CPU= 22.890
  03:11:27 STEP 1783 H= 445:45 +CPU= 26.894
  03:11:55 STEP 1784 H= 446:00 +CPU= 27.916
  03:12:21 STEP 1785 H= 446:15 +CPU= 26.318
  03:12:47 STEP 1786 H= 446:30 +CPU= 26.335
  03:13:15 STEP 1787 H= 446:45 +CPU= 26.996
  03:13:48 STEP 1788 H= 447:00 +CPU= 33.272
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001788
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001788
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001788
  03:14:15 STEP 1789 H= 447:15 +CPU= 27.373
  03:14:43 STEP 1790 H= 447:30 +CPU= 27.348
  03:15:10 STEP 1791 H= 447:45 +CPU= 26.974
  03:15:37 STEP 1792 H= 448:00 +CPU= 27.188
  03:16:04 STEP 1793 H= 448:15 +CPU= 27.155
  03:16:30 STEP 1794 H= 448:30 +CPU= 26.141
  03:16:58 STEP 1795 H= 448:45 +CPU= 28.164
  03:17:25 STEP 1796 H= 449:00 +CPU= 26.672
  03:17:53 STEP 1797 H= 449:15 +CPU= 27.540
  03:18:20 STEP 1798 H= 449:30 +CPU= 26.945
  03:18:47 STEP 1799 H= 449:45 +CPU= 26.716
  03:19:20 STEP 1800 H= 450:00 +CPU= 33.392
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001800
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001800
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001800
  03:19:48 STEP 1801 H= 450:15 +CPU= 27.452
  03:20:15 STEP 1802 H= 450:30 +CPU= 27.248
  03:20:42 STEP 1803 H= 450:45 +CPU= 27.294
  03:21:09 STEP 1804 H= 451:00 +CPU= 26.625
  03:21:35 STEP 1805 H= 451:15 +CPU= 26.044
  03:22:02 STEP 1806 H= 451:30 +CPU= 27.282
  03:22:29 STEP 1807 H= 451:45 +CPU= 26.881
  03:22:57 STEP 1808 H= 452:00 +CPU= 27.331
  03:23:23 STEP 1809 H= 452:15 +CPU= 26.272
  03:23:47 STEP 1810 H= 452:30 +CPU= 24.065
  03:24:06 STEP 1811 H= 452:45 +CPU= 19.329
  03:24:34 STEP 1812 H= 453:00 +CPU= 27.386
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001812
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001812
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001812
  03:25:01 STEP 1813 H= 453:15 +CPU= 27.362
  03:25:27 STEP 1814 H= 453:30 +CPU= 26.121
  03:25:55 STEP 1815 H= 453:45 +CPU= 27.322
  03:26:23 STEP 1816 H= 454:00 +CPU= 27.807
  03:26:49 STEP 1817 H= 454:15 +CPU= 26.715
  03:27:15 STEP 1818 H= 454:30 +CPU= 26.091
  03:27:42 STEP 1819 H= 454:45 +CPU= 26.619
  03:28:09 STEP 1820 H= 455:00 +CPU= 27.129
  03:28:36 STEP 1821 H= 455:15 +CPU= 26.520
  03:29:02 STEP 1822 H= 455:30 +CPU= 26.005
  03:29:29 STEP 1823 H= 455:45 +CPU= 27.266
  03:30:06 STEP 1824 H= 456:00 +CPU= 36.655
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001824
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001824
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001812
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001812
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001812
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a29g_2016092300_20_1018_12291259_0_r883522508_18.zip
Uploading the intermediate file: upload_file_18.zip
  03:30:33 STEP 1825 H= 456:15 +CPU= 26.644
  03:30:59 STEP 1826 H= 456:30 +CPU= 26.278
  03:31:25 STEP 1827 H= 456:45 +CPU= 26.163
  03:31:51 STEP 1828 H= 457:00 +CPU= 25.772
  03:32:17 STEP 1829 H= 457:15 +CPU= 25.623
  03:32:44 STEP 1830 H= 457:30 +CPU= 27.087
  03:33:11 STEP 1831 H= 457:45 +CPU= 26.804
  03:33:39 STEP 1832 H= 458:00 +CPU= 28.122
  03:34:05 STEP 1833 H= 458:15 +CPU= 25.994
  03:34:32 STEP 1834 H= 458:30 +CPU= 27.410
  03:34:59 STEP 1835 H= 458:45 +CPU= 26.027
  03:35:31 STEP 1836 H= 459:00 +CPU= 32.445
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001836
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001836
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001836
  03:35:57 STEP 1837 H= 459:15 +CPU= 26.030
  03:36:24 STEP 1838 H= 459:30 +CPU= 26.771
  03:36:50 STEP 1839 H= 459:45 +CPU= 26.296
  03:37:18 STEP 1840 H= 460:00 +CPU= 27.649
  03:37:44 STEP 1841 H= 460:15 +CPU= 26.363
  03:38:11 STEP 1842 H= 460:30 +CPU= 26.944
  03:38:37 STEP 1843 H= 460:45 +CPU= 26.005
  03:39:06 STEP 1844 H= 461:00 +CPU= 28.135
  03:39:32 STEP 1845 H= 461:15 +CPU= 25.917
  03:39:58 STEP 1846 H= 461:30 +CPU= 26.131
  03:40:23 STEP 1847 H= 461:45 +CPU= 25.575
  03:40:58 STEP 1848 H= 462:00 +CPU= 34.170
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001848
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001848
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001848
  03:41:24 STEP 1849 H= 462:15 +CPU= 26.444
  03:41:51 STEP 1850 H= 462:30 +CPU= 26.679
  03:42:18 STEP 1851 H= 462:45 +CPU= 27.354
  03:42:39 STEP 1852 H= 463:00 +CPU= 20.815
  03:42:58 STEP 1853 H= 463:15 +CPU= 19.264
  03:43:25 STEP 1854 H= 463:30 +CPU= 27.065
  03:43:51 STEP 1855 H= 463:45 +CPU= 25.223
  03:44:16 STEP 1856 H= 464:00 +CPU= 25.617
  03:44:42 STEP 1857 H= 464:15 +CPU= 25.581
  03:45:08 STEP 1858 H= 464:30 +CPU= 25.824
  03:45:35 STEP 1859 H= 464:45 +CPU= 26.731
  03:46:09 STEP 1860 H= 465:00 +CPU= 33.834
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001860
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001860
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001860
  03:46:34 STEP 1861 H= 465:15 +CPU= 25.783
  03:47:00 STEP 1862 H= 465:30 +CPU= 25.180
  03:47:26 STEP 1863 H= 465:45 +CPU= 26.053
  03:47:53 STEP 1864 H= 466:00 +CPU= 27.196
  03:48:19 STEP 1865 H= 466:15 +CPU= 25.664
  03:48:46 STEP 1866 H= 466:30 +CPU= 26.937
  03:49:14 STEP 1867 H= 466:45 +CPU= 28.074
  03:49:41 STEP 1868 H= 467:00 +CPU= 27.530
  03:50:08 STEP 1869 H= 467:15 +CPU= 26.378
  03:50:34 STEP 1870 H= 467:30 +CPU= 26.548
  03:51:00 STEP 1871 H= 467:45 +CPU= 25.879
  03:51:36 STEP 1872 H= 468:00 +CPU= 35.534
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001872
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001872
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001872
  03:52:02 STEP 1873 H= 468:15 +CPU= 26.280
  03:52:28 STEP 1874 H= 468:30 +CPU= 25.520
  03:52:55 STEP 1875 H= 468:45 +CPU= 26.881
  03:53:23 STEP 1876 H= 469:00 +CPU= 28.480
  03:53:51 STEP 1877 H= 469:15 +CPU= 27.755
  03:54:17 STEP 1878 H= 469:30 +CPU= 26.361
  03:54:44 STEP 1879 H= 469:45 +CPU= 26.690
  03:55:12 STEP 1880 H= 470:00 +CPU= 27.981
  03:55:38 STEP 1881 H= 470:15 +CPU= 25.550
  03:56:04 STEP 1882 H= 470:30 +CPU= 26.325
  03:56:30 STEP 1883 H= 470:45 +CPU= 26.092
  03:57:03 STEP 1884 H= 471:00 +CPU= 32.589
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001884
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001884
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001884
  03:57:30 STEP 1885 H= 471:15 +CPU= 26.647
  03:57:57 STEP 1886 H= 471:30 +CPU= 26.803
  03:58:22 STEP 1887 H= 471:45 +CPU= 25.940
  03:58:49 STEP 1888 H= 472:00 +CPU= 26.463
  03:59:16 STEP 1889 H= 472:15 +CPU= 27.201
  03:59:44 STEP 1890 H= 472:30 +CPU= 27.898
  04:00:05 STEP 1891 H= 472:45 +CPU= 20.947
  04:00:26 STEP 1892 H= 473:00 +CPU= 20.788
  04:00:51 STEP 1893 H= 473:15 +CPU= 24.914
  04:01:17 STEP 1894 H= 473:30 +CPU= 26.057
  04:01:43 STEP 1895 H= 473:45 +CPU= 26.331
  04:02:16 STEP 1896 H= 474:00 +CPU= 32.909
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001896
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001896
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001896
  04:02:43 STEP 1897 H= 474:15 +CPU= 26.203
  04:03:09 STEP 1898 H= 474:30 +CPU= 26.436
  04:03:36 STEP 1899 H= 474:45 +CPU= 26.539
  04:04:02 STEP 1900 H= 475:00 +CPU= 26.737
  04:04:29 STEP 1901 H= 475:15 +CPU= 26.456
  04:04:55 STEP 1902 H= 475:30 +CPU= 26.113
  04:05:22 STEP 1903 H= 475:45 +CPU= 26.451
  04:05:49 STEP 1904 H= 476:00 +CPU= 26.950
  04:06:15 STEP 1905 H= 476:15 +CPU= 26.337
  04:06:42 STEP 1906 H= 476:30 +CPU= 27.069
  04:07:10 STEP 1907 H= 476:45 +CPU= 27.523
  04:07:43 STEP 1908 H= 477:00 +CPU= 32.890
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001908
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001908
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001908
  04:08:09 STEP 1909 H= 477:15 +CPU= 26.825
  04:08:35 STEP 1910 H= 477:30 +CPU= 25.891
  04:09:03 STEP 1911 H= 477:45 +CPU= 27.590
  04:09:30 STEP 1912 H= 478:00 +CPU= 27.021
  04:09:57 STEP 1913 H= 478:15 +CPU= 26.536
  04:10:24 STEP 1914 H= 478:30 +CPU= 26.765
  04:10:50 STEP 1915 H= 478:45 +CPU= 26.628
  04:11:18 STEP 1916 H= 479:00 +CPU= 27.416
  04:11:45 STEP 1917 H= 479:15 +CPU= 26.875
  04:12:11 STEP 1918 H= 479:30 +CPU= 26.566
  04:12:38 STEP 1919 H= 479:45 +CPU= 26.741
  04:13:13 STEP 1920 H= 480:00 +CPU= 34.473
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.1216204823E-20      I      -0.1737238207E-05      I
ICloud fractionI      0.0000000000E+00      I       0.2101407430E-01      I
I  Relat. Hum. I      0.1104940311E+00      I       0.3819766876E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.7089452031E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    416369860 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS693949.8 Mbytes/s, TOTAL TIME=    0.0117( 5.1%)
 04:13:13 STEP 1920 H= 480:00 +CPU= 34.473
 END CNT3
 NSTEP =  1920 CNT0   000000000  25.180  25.180  25.215   0.999               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         4290/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     78    0       46      1044068     4290 ************     8.9     1.5     7.4     2.2    14.9
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 3903221.8 3903221.8    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 3899529.9 3899529.9     99.91      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  157    4366.3    4366.3     17.56      0.00
   9 SPCM     - SPECTRAL COMP.                     144     592.4     592.4      2.19      0.00
  10 SCAN2M - PHYSICS                              145   12972.3   12972.3     48.19      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 12     874.0     874.0      0.27      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  7      40.0      40.0      0.01      0.00
  17 GRIDFPOS IN CNT4                               12      24.8      24.8      0.01      0.00
  27 RESTART FILES - WRITING                         3    3495.9    3495.9      0.27      0.00
  28 RESTART FILES - READING                         1    2941.3    2941.3      0.08      0.00
  29 SU4FPOS IN CNT4                                12       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                12    5593.5    5593.5      1.72      0.00
  31 POSDDH IN STEPO                                 1       1.1       1.1      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    145    5756.7    5756.7     21.39      0.00
  39 SU0YOMB                                         1     448.4     448.4      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    145      19.7      19.7      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              144     287.0     287.0      1.06      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              144      50.6      50.6      0.19      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  7       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      316     359.8     359.8      2.91      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       300     223.6     223.6      1.72      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        300      82.3      82.3      0.63      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       316     107.5     107.5      0.87      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      50.8      50.8      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            316      58.0      58.0      0.47      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            300      31.7      31.7      0.24      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            316      46.4      46.4      0.38      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            300      26.9      26.9      0.21      0.00
 400 GSTATS                                      17763       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 16473       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     3903.2 CPU TIME    3896.3 VECTOR TIME     3896.3


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      443.1
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /boinc_data/boinc/slots/3/ICMGGh7zg+001920
Moving to projects directory: /boinc_data/boinc/slots/3/ICMSHh7zg+001920
Moving to projects directory: /boinc_data/boinc/slots/3/ICMUAh7zg+001920
Adding to the zip: /boinc_data/boinc/slots/3/NODE.001_01
Adding to the zip: /boinc_data/boinc/slots/3/ifs.stat
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001860
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001860
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMGGh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMSHh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12291259/ICMUAh7zg+001860
Zipping up the final file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a29g_2016092300_20_1018_12291259_0_r883522508_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
04:16:47 (871870): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
16 Jun 2024 09:35:21 1496046 22444858 oifs_43r3_bl_a29g_2016092300_20_1018_12291259_0 1,727,100 55,886 0.0324
16 Jun 2024 02:50:11 1496046 22444858 oifs_43r3_bl_a29g_2016092300_20_1018_12291259_0 1,036,800 33,673 0.0325


©2024 climateprediction.net