climateprediction.net home page
Task 22444030

Task 22444030

Name oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0
Workunit 12290437
Created 12 Jun 2024, 23:00:52 UTC
Sent 13 Jun 2024, 16:36:32 UTC
Report deadline 12 Aug 2024, 16:36:32 UTC
Received 17 Jun 2024, 16:08:02 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1489681
Run time 12 hours 4 min 34 sec
CPU time 1 hours 23 min 27 sec
Validate state Invalid
Credit 2,635.55
Device peak FLOPS 8.68 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,330.73 MB
Peak swap size 5,981.12 MB
Peak disk usage 1,557.91 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:31:41 STEP 1680 H= 420:00 +CPU= 20.764
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001632
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001632
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001632
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001644
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001644
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001644
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001656
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001656
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001656
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001668
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001668
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001668
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_0.zip
Uploading the intermediate file: upload_file_0.zip
  15:32:00 STEP 1681 H= 420:15 +CPU= 18.652
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:32:21 STEP 1682 H= 420:30 +CPU= 18.751
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:32:42 STEP 1683 H= 420:45 +CPU= 18.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:33:03 STEP 1684 H= 421:00 +CPU= 19.199
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:33:24 STEP 1685 H= 421:15 +CPU= 18.525
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:33:44 STEP 1686 H= 421:30 +CPU= 18.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:34:05 STEP 1687 H= 421:45 +CPU= 18.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:34:26 STEP 1688 H= 422:00 +CPU= 19.192
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:34:46 STEP 1689 H= 422:15 +CPU= 18.395
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:35:07 STEP 1690 H= 422:30 +CPU= 18.460
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:35:27 STEP 1691 H= 422:45 +CPU= 18.107
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:35:56 STEP 1692 H= 423:00 +CPU= 25.318
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001692
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001692
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001692
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:36:17 STEP 1693 H= 423:15 +CPU= 18.877
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:36:38 STEP 1694 H= 423:30 +CPU= 18.828
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:37:01 STEP 1695 H= 423:45 +CPU= 18.776
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:37:23 STEP 1696 H= 424:00 +CPU= 20.184
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:37:45 STEP 1697 H= 424:15 +CPU= 20.077
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:38:07 STEP 1698 H= 424:30 +CPU= 20.302
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:38:30 STEP 1699 H= 424:45 +CPU= 20.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:38:53 STEP 1700 H= 425:00 +CPU= 20.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:39:15 STEP 1701 H= 425:15 +CPU= 20.148
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:39:36 STEP 1702 H= 425:30 +CPU= 19.655
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:39:58 STEP 1703 H= 425:45 +CPU= 19.131
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:40:27 STEP 1704 H= 426:00 +CPU= 25.172
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001704
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001704
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001704
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:40:48 STEP 1705 H= 426:15 +CPU= 19.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:41:10 STEP 1706 H= 426:30 +CPU= 19.553
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:41:32 STEP 1707 H= 426:45 +CPU= 20.192
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:41:55 STEP 1708 H= 427:00 +CPU= 20.816
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:42:17 STEP 1709 H= 427:15 +CPU= 19.939
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:42:41 STEP 1710 H= 427:30 +CPU= 19.972
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:43:03 STEP 1711 H= 427:45 +CPU= 19.816
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:43:26 STEP 1712 H= 428:00 +CPU= 20.776
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:43:48 STEP 1713 H= 428:15 +CPU= 20.010
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:44:09 STEP 1714 H= 428:30 +CPU= 18.948
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:44:30 STEP 1715 H= 428:45 +CPU= 18.981
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:44:57 STEP 1716 H= 429:00 +CPU= 24.976
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001716
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001716
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001716
  15:45:19 STEP 1717 H= 429:15 +CPU= 19.742
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:45:40 STEP 1718 H= 429:30 +CPU= 19.773
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:46:02 STEP 1719 H= 429:45 +CPU= 19.900
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:46:25 STEP 1720 H= 430:00 +CPU= 20.909
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:46:47 STEP 1721 H= 430:15 +CPU= 20.023
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:47:09 STEP 1722 H= 430:30 +CPU= 20.176
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:47:32 STEP 1723 H= 430:45 +CPU= 20.272
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:47:54 STEP 1724 H= 431:00 +CPU= 20.568
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:48:16 STEP 1725 H= 431:15 +CPU= 19.347
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:48:37 STEP 1726 H= 431:30 +CPU= 18.813
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:49:00 STEP 1727 H= 431:45 +CPU= 19.049
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:49:30 STEP 1728 H= 432:00 +CPU= 28.210
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001728
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001728
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001728
  15:49:52 STEP 1729 H= 432:15 +CPU= 19.870
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:50:13 STEP 1730 H= 432:30 +CPU= 19.497
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:50:35 STEP 1731 H= 432:45 +CPU= 19.423
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:51:00 STEP 1732 H= 433:00 +CPU= 20.819
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:51:22 STEP 1733 H= 433:15 +CPU= 20.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:51:46 STEP 1734 H= 433:30 +CPU= 21.087
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:52:08 STEP 1735 H= 433:45 +CPU= 20.679
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:52:31 STEP 1736 H= 434:00 +CPU= 20.547
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:52:52 STEP 1737 H= 434:15 +CPU= 19.475
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:53:14 STEP 1738 H= 434:30 +CPU= 19.122
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:53:35 STEP 1739 H= 434:45 +CPU= 19.605
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:54:06 STEP 1740 H= 435:00 +CPU= 26.392
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001740
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001740
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001740
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:54:29 STEP 1741 H= 435:15 +CPU= 20.976
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:54:52 STEP 1742 H= 435:30 +CPU= 21.053
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:55:15 STEP 1743 H= 435:45 +CPU= 20.789
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:55:40 STEP 1744 H= 436:00 +CPU= 21.389
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:56:03 STEP 1745 H= 436:15 +CPU= 20.931
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:56:26 STEP 1746 H= 436:30 +CPU= 20.874
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:56:49 STEP 1747 H= 436:45 +CPU= 20.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:57:13 STEP 1748 H= 437:00 +CPU= 21.773
  15:57:33 STEP 1749 H= 437:15 +CPU= 20.538
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:57:55 STEP 1750 H= 437:30 +CPU= 19.813
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:58:17 STEP 1751 H= 437:45 +CPU= 19.603
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:58:45 STEP 1752 H= 438:00 +CPU= 25.968
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001752
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001752
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001752
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:59:07 STEP 1753 H= 438:15 +CPU= 20.251
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:59:30 STEP 1754 H= 438:30 +CPU= 20.666
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:59:53 STEP 1755 H= 438:45 +CPU= 20.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:00:16 STEP 1756 H= 439:00 +CPU= 21.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:00:41 STEP 1757 H= 439:15 +CPU= 20.534
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:01:03 STEP 1758 H= 439:30 +CPU= 20.133
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:01:25 STEP 1759 H= 439:45 +CPU= 20.040
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:01:49 STEP 1760 H= 440:00 +CPU= 21.665
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:02:11 STEP 1761 H= 440:15 +CPU= 20.411
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:02:33 STEP 1762 H= 440:30 +CPU= 20.181
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:02:55 STEP 1763 H= 440:45 +CPU= 20.228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:03:26 STEP 1764 H= 441:00 +CPU= 26.086
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001764
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001764
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001764
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:03:48 STEP 1765 H= 441:15 +CPU= 20.157
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:04:10 STEP 1766 H= 441:30 +CPU= 20.575
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:04:33 STEP 1767 H= 441:45 +CPU= 20.696
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:04:56 STEP 1768 H= 442:00 +CPU= 21.144
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:05:21 STEP 1769 H= 442:15 +CPU= 20.279
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:05:43 STEP 1770 H= 442:30 +CPU= 20.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:06:05 STEP 1771 H= 442:45 +CPU= 20.376
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:06:28 STEP 1772 H= 443:00 +CPU= 21.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:06:51 STEP 1773 H= 443:15 +CPU= 20.574
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:07:13 STEP 1774 H= 443:30 +CPU= 20.171
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:07:35 STEP 1775 H= 443:45 +CPU= 20.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:08:08 STEP 1776 H= 444:00 +CPU= 28.490
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001776
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001776
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001776
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001680
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001680
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001680
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001692
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001692
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001692
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001704
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001704
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001704
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001716
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001716
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001716
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001728
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001728
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001728
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001740
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001740
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001740
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001752
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001752
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001752
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001764
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001764
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001764
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_1.zip
Uploading the intermediate file: upload_file_1.zip
  16:08:29 STEP 1777 H= 444:15 +CPU= 19.700
  16:08:50 STEP 1778 H= 444:30 +CPU= 20.461
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:09:13 STEP 1779 H= 444:45 +CPU= 20.778
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:09:36 STEP 1780 H= 445:00 +CPU= 21.587
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:10:01 STEP 1781 H= 445:15 +CPU= 20.700
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:10:24 STEP 1782 H= 445:30 +CPU= 20.689
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:10:46 STEP 1783 H= 445:45 +CPU= 20.711
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:11:10 STEP 1784 H= 446:00 +CPU= 21.501
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:11:32 STEP 1785 H= 446:15 +CPU= 20.469
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:11:55 STEP 1786 H= 446:30 +CPU= 20.251
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:12:19 STEP 1787 H= 446:45 +CPU= 19.916
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:12:48 STEP 1788 H= 447:00 +CPU= 27.402
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001788
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001788
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001788
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:13:11 STEP 1789 H= 447:15 +CPU= 20.790
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:13:34 STEP 1790 H= 447:30 +CPU= 20.752
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:13:57 STEP 1791 H= 447:45 +CPU= 20.735
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:22 STEP 1792 H= 448:00 +CPU= 21.427
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:45 STEP 1793 H= 448:15 +CPU= 20.806
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:07 STEP 1794 H= 448:30 +CPU= 20.515
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:30 STEP 1795 H= 448:45 +CPU= 20.034
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:52 STEP 1796 H= 449:00 +CPU= 20.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:14 STEP 1797 H= 449:15 +CPU= 19.807
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:36 STEP 1798 H= 449:30 +CPU= 20.433
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:01 STEP 1799 H= 449:45 +CPU= 20.789
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:30 STEP 1800 H= 450:00 +CPU= 26.729
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001800
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001800
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001800
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:52 STEP 1801 H= 450:15 +CPU= 20.043
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:13 STEP 1802 H= 450:30 +CPU= 19.621
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:35 STEP 1803 H= 450:45 +CPU= 19.318
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:59 STEP 1804 H= 451:00 +CPU= 20.685
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:19:22 STEP 1805 H= 451:15 +CPU= 20.959
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:19:45 STEP 1806 H= 451:30 +CPU= 20.699
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:08 STEP 1807 H= 451:45 +CPU= 20.419
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:30 STEP 1808 H= 452:00 +CPU= 20.600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:52 STEP 1809 H= 452:15 +CPU= 19.986
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:15 STEP 1810 H= 452:30 +CPU= 20.145
  16:21:35 STEP 1811 H= 452:45 +CPU= 19.870
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:22:04 STEP 1812 H= 453:00 +CPU= 25.367
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001812
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001812
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001812
  16:22:23 STEP 1813 H= 453:15 +CPU= 19.312
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:22:46 STEP 1814 H= 453:30 +CPU= 20.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:08 STEP 1815 H= 453:45 +CPU= 20.609
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:31 STEP 1816 H= 454:00 +CPU= 20.942
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:54 STEP 1817 H= 454:15 +CPU= 20.432
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:24:16 STEP 1818 H= 454:30 +CPU= 20.132
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:24:39 STEP 1819 H= 454:45 +CPU= 19.637
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:25:01 STEP 1820 H= 455:00 +CPU= 19.884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:25:22 STEP 1821 H= 455:15 +CPU= 18.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:25:44 STEP 1822 H= 455:30 +CPU= 19.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:26:07 STEP 1823 H= 455:45 +CPU= 20.480
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:26:36 STEP 1824 H= 456:00 +CPU= 27.587
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001824
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001824
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001824
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:26:59 STEP 1825 H= 456:15 +CPU= 18.933
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:27:20 STEP 1826 H= 456:30 +CPU= 18.975
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:27:41 STEP 1827 H= 456:45 +CPU= 18.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:28:04 STEP 1828 H= 457:00 +CPU= 20.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:28:26 STEP 1829 H= 457:15 +CPU= 20.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:28:48 STEP 1830 H= 457:30 +CPU= 20.064
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:29:11 STEP 1831 H= 457:45 +CPU= 20.222
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:29:34 STEP 1832 H= 458:00 +CPU= 21.377
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:29:56 STEP 1833 H= 458:15 +CPU= 19.661
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:30:19 STEP 1834 H= 458:30 +CPU= 19.600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:30:41 STEP 1835 H= 458:45 +CPU= 19.150
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:31:08 STEP 1836 H= 459:00 +CPU= 25.641
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001836
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001836
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:31:31 STEP 1837 H= 459:15 +CPU= 20.447
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:31:53 STEP 1838 H= 459:30 +CPU= 20.604
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:32:16 STEP 1839 H= 459:45 +CPU= 20.429
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:32:41 STEP 1840 H= 460:00 +CPU= 21.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:04 STEP 1841 H= 460:15 +CPU= 20.106
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:25 STEP 1842 H= 460:30 +CPU= 19.382
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:46 STEP 1843 H= 460:45 +CPU= 18.894
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:08 STEP 1844 H= 461:00 +CPU= 20.296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:31 STEP 1845 H= 461:15 +CPU= 20.194
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:53 STEP 1846 H= 461:30 +CPU= 20.237
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:15 STEP 1847 H= 461:45 +CPU= 19.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:45 STEP 1848 H= 462:00 +CPU= 26.551
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001848
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001848
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:07 STEP 1849 H= 462:15 +CPU= 19.824
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:29 STEP 1850 H= 462:30 +CPU= 19.778
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:50 STEP 1851 H= 462:45 +CPU= 19.216
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:13 STEP 1852 H= 463:00 +CPU= 20.419
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:35 STEP 1853 H= 463:15 +CPU= 20.237
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:59 STEP 1854 H= 463:30 +CPU= 20.102
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:38:22 STEP 1855 H= 463:45 +CPU= 20.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:38:45 STEP 1856 H= 464:00 +CPU= 21.034
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:07 STEP 1857 H= 464:15 +CPU= 20.361
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:29 STEP 1858 H= 464:30 +CPU= 19.608
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:50 STEP 1859 H= 464:45 +CPU= 19.361
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:40:20 STEP 1860 H= 465:00 +CPU= 25.770
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001860
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001860
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001860
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:40:42 STEP 1861 H= 465:15 +CPU= 20.023
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:41:05 STEP 1862 H= 465:30 +CPU= 20.452
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:41:27 STEP 1863 H= 465:45 +CPU= 20.430
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:41:50 STEP 1864 H= 466:00 +CPU= 20.433
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:42:11 STEP 1865 H= 466:15 +CPU= 19.735
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:42:33 STEP 1866 H= 466:30 +CPU= 19.663
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:42:54 STEP 1867 H= 466:45 +CPU= 19.519
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:43:19 STEP 1868 H= 467:00 +CPU= 20.797
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:43:42 STEP 1869 H= 467:15 +CPU= 20.497
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:04 STEP 1870 H= 467:30 +CPU= 20.078
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:26 STEP 1871 H= 467:45 +CPU= 20.019
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:58 STEP 1872 H= 468:00 +CPU= 27.751
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001872
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001872
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001872
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001776
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001776
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001776
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001788
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001788
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001788
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001800
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001800
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001800
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001812
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001812
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001812
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001824
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001824
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001824
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001836
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001836
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001836
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001848
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001848
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001848
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001860
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001860
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001860
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_2.zip
Uploading the intermediate file: upload_file_2.zip
  16:45:18 STEP 1873 H= 468:15 +CPU= 18.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:45:39 STEP 1874 H= 468:30 +CPU= 20.068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:46:02 STEP 1875 H= 468:45 +CPU= 20.634
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:46:25 STEP 1876 H= 469:00 +CPU= 21.254
  16:46:46 STEP 1877 H= 469:15 +CPU= 20.628
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:09 STEP 1878 H= 469:30 +CPU= 20.424
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:31 STEP 1879 H= 469:45 +CPU= 20.002
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:53 STEP 1880 H= 470:00 +CPU= 20.297
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:48:15 STEP 1881 H= 470:15 +CPU= 19.594
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:48:39 STEP 1882 H= 470:30 +CPU= 20.044
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:49:01 STEP 1883 H= 470:45 +CPU= 20.478
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:49:30 STEP 1884 H= 471:00 +CPU= 26.441
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001884
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001884
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001884
  16:49:53 STEP 1885 H= 471:15 +CPU= 19.244
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:14 STEP 1886 H= 471:30 +CPU= 19.395
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:35 STEP 1887 H= 471:45 +CPU= 19.120
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:59 STEP 1888 H= 472:00 +CPU= 19.756
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:51:21 STEP 1889 H= 472:15 +CPU= 19.367
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:51:43 STEP 1890 H= 472:30 +CPU= 20.125
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:52:05 STEP 1891 H= 472:45 +CPU= 20.580
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:52:29 STEP 1892 H= 473:00 +CPU= 21.104
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:52:51 STEP 1893 H= 473:15 +CPU= 20.427
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:53:14 STEP 1894 H= 473:30 +CPU= 20.471
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:53:36 STEP 1895 H= 473:45 +CPU= 19.971
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:54:06 STEP 1896 H= 474:00 +CPU= 26.138
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001896
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001896
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001896
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:54:28 STEP 1897 H= 474:15 +CPU= 19.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:54:49 STEP 1898 H= 474:30 +CPU= 19.783
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:11 STEP 1899 H= 474:45 +CPU= 19.666
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:34 STEP 1900 H= 475:00 +CPU= 21.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:59 STEP 1901 H= 475:15 +CPU= 20.820
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:56:22 STEP 1902 H= 475:30 +CPU= 20.674
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:56:44 STEP 1903 H= 475:45 +CPU= 19.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:57:06 STEP 1904 H= 476:00 +CPU= 20.218
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:57:28 STEP 1905 H= 476:15 +CPU= 20.182
  16:57:49 STEP 1906 H= 476:30 +CPU= 20.681
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:58:12 STEP 1907 H= 476:45 +CPU= 20.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:58:43 STEP 1908 H= 477:00 +CPU= 26.992
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001908
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001908
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001908
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:59:07 STEP 1909 H= 477:15 +CPU= 21.941
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:59:31 STEP 1910 H= 477:30 +CPU= 22.050
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:59:55 STEP 1911 H= 477:45 +CPU= 22.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:00:22 STEP 1912 H= 478:00 +CPU= 22.339
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:00:45 STEP 1913 H= 478:15 +CPU= 21.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:01:10 STEP 1914 H= 478:30 +CPU= 22.330
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:01:35 STEP 1915 H= 478:45 +CPU= 23.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:02:04 STEP 1916 H= 479:00 +CPU= 24.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:02:30 STEP 1917 H= 479:15 +CPU= 23.685
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:02:53 STEP 1918 H= 479:30 +CPU= 21.625
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:03:19 STEP 1919 H= 479:45 +CPU= 21.597
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:03:50 STEP 1920 H= 480:00 +CPU= 29.136
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.2858338309E-21      I       0.2173645409E-05      I
ICloud fractionI      0.0000000000E+00      I       0.1518838675E-01      I
I  Relat. Hum. I      0.2190151138E+00      I       0.2719843779E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.2714301623E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    319004980 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS531675.0 Mbytes/s, TOTAL TIME=    0.0100( 6.0%)
 17:03:50 STEP 1920 H= 480:00 +CPU= 29.136
 END CNT3
 NSTEP =  1920 CNT0   000000000  21.652  21.652  23.675   0.915               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         7150/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    128    0       46      1044068     7150 ************    12.2     1.5    10.7     2.3    21.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 5557850.8 5557850.8    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 5554189.2 5554189.2     99.93      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  261    3774.2    3774.2     17.72      0.00
   9 SPCM     - SPECTRAL COMP.                     240     430.6     430.6      1.86      0.00
  10 SCAN2M - PHYSICS                              241   10981.7   10981.7     47.62      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 20     709.1     709.1      0.26      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 11      24.4      24.4      0.00      0.00
  17 GRIDFPOS IN CNT4                               20      21.1      21.1      0.01      0.00
  27 RESTART FILES - WRITING                         5    2835.0    2835.0      0.26      0.00
  28 RESTART FILES - READING                         1    3000.3    3000.3      0.05      0.00
  29 SU4FPOS IN CNT4                                20       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                20    5538.8    5538.8      1.99      0.00
  31 POSDDH IN STEPO                                 2       1.1       1.1      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    241    5298.9    5298.9     22.98      0.00
  39 SU0YOMB                                         1     293.6     293.6      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    241      11.9      11.9      0.05      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              240      72.2      72.2      0.31      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              240      38.7      38.7      0.17      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 11       0.1       0.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      524     290.3     290.3      2.74      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       500     197.9     197.9      1.78      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        500      59.0      59.0      0.53      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       524      93.1      93.1      0.88      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      38.0      38.0      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            524      40.2      40.2      0.38      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            500      20.7      20.7      0.19      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            524      43.7      43.7      0.41      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            500      26.1      26.1      0.23      0.00
 400 GSTATS                                      28864       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 26802       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     5557.9 CPU TIME    5007.1 VECTOR TIME     5007.1


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      311.1
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001920
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001920
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001920
Adding to the zip: /mnt/nvme/scratch1/boinc/slots/0/NODE.001_01
Adding to the zip: /mnt/nvme/scratch1/boinc/slots/0/ifs.stat
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001872
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001872
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001872
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001884
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001884
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001884
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001896
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001896
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001896
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001908
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001908
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001908
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001920
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001920
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001920
Zipping up the final file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_3.zip
Uploading the final file: upload_file_3.zip
Uploading trickle at timestep: 1727100
17:07:16 (341736): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
17 Jun 2024 15:06:33 1489681 22444030 oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0 1,727,100 5,007 0.0029
16 Jun 2024 15:44:21 1489681 22444030 oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0 864,000 20,756 0.0240


©2024 cpdn.org