Name | oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0 |
Workunit | 12290437 |
Created | 12 Jun 2024, 23:00:52 UTC |
Sent | 13 Jun 2024, 16:36:32 UTC |
Report deadline | 12 Aug 2024, 16:36:32 UTC |
Received | 17 Jun 2024, 16:08:02 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 0 (0x00000000) |
Computer ID | 1489681 |
Run time | 12 hours 4 min 34 sec |
CPU time | 1 hours 23 min 27 sec |
Validate state | Invalid |
Credit | 2,635.55 |
Device peak FLOPS | 8.68 GFLOPS |
Application version | OpenIFS 43r3 Baroclinic Lifecycle v1.13 x86_64-pc-linux-gnu |
Peak working set size | 5,330.73 MB |
Peak swap size | 5,981.12 MB |
Peak disk usage | 1,557.91 MB |
Stderr | <core_client_version>7.20.2</core_client_version> <![CDATA[ <stderr_txt> 341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:341806:341806] [20240617:153112:1718631072:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:31:41 STEP 1680 H= 420:00 +CPU= 20.764 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001632 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001632 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001632 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001644 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001644 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001644 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001656 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001656 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001656 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001668 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001668 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001668 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_0.zip Uploading the intermediate file: upload_file_0.zip 15:32:00 STEP 1681 H= 420:15 +CPU= 18.652 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:32:21 STEP 1682 H= 420:30 +CPU= 18.751 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:32:42 STEP 1683 H= 420:45 +CPU= 18.488 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:33:03 STEP 1684 H= 421:00 +CPU= 19.199 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:33:24 STEP 1685 H= 421:15 +CPU= 18.525 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:33:44 STEP 1686 H= 421:30 +CPU= 18.437 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:34:05 STEP 1687 H= 421:45 +CPU= 18.498 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:34:26 STEP 1688 H= 422:00 +CPU= 19.192 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:34:46 STEP 1689 H= 422:15 +CPU= 18.395 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:35:07 STEP 1690 H= 422:30 +CPU= 18.460 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:35:27 STEP 1691 H= 422:45 +CPU= 18.107 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:35:56 STEP 1692 H= 423:00 +CPU= 25.318 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001692 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001692 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001692 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:36:17 STEP 1693 H= 423:15 +CPU= 18.877 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:36:38 STEP 1694 H= 423:30 +CPU= 18.828 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:37:01 STEP 1695 H= 423:45 +CPU= 18.776 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:37:23 STEP 1696 H= 424:00 +CPU= 20.184 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:37:45 STEP 1697 H= 424:15 +CPU= 20.077 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:38:07 STEP 1698 H= 424:30 +CPU= 20.302 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:38:30 STEP 1699 H= 424:45 +CPU= 20.255 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:38:53 STEP 1700 H= 425:00 +CPU= 20.973 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:39:15 STEP 1701 H= 425:15 +CPU= 20.148 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:39:36 STEP 1702 H= 425:30 +CPU= 19.655 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:39:58 STEP 1703 H= 425:45 +CPU= 19.131 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:40:27 STEP 1704 H= 426:00 +CPU= 25.172 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001704 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001704 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001704 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:40:48 STEP 1705 H= 426:15 +CPU= 19.392 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:41:10 STEP 1706 H= 426:30 +CPU= 19.553 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:41:32 STEP 1707 H= 426:45 +CPU= 20.192 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:41:55 STEP 1708 H= 427:00 +CPU= 20.816 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:42:17 STEP 1709 H= 427:15 +CPU= 19.939 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:42:41 STEP 1710 H= 427:30 +CPU= 19.972 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:43:03 STEP 1711 H= 427:45 +CPU= 19.816 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:43:26 STEP 1712 H= 428:00 +CPU= 20.776 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:43:48 STEP 1713 H= 428:15 +CPU= 20.010 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:44:09 STEP 1714 H= 428:30 +CPU= 18.948 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:44:30 STEP 1715 H= 428:45 +CPU= 18.981 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:44:57 STEP 1716 H= 429:00 +CPU= 24.976 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001716 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001716 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001716 15:45:19 STEP 1717 H= 429:15 +CPU= 19.742 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:45:40 STEP 1718 H= 429:30 +CPU= 19.773 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:46:02 STEP 1719 H= 429:45 +CPU= 19.900 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:46:25 STEP 1720 H= 430:00 +CPU= 20.909 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:46:47 STEP 1721 H= 430:15 +CPU= 20.023 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:47:09 STEP 1722 H= 430:30 +CPU= 20.176 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:47:32 STEP 1723 H= 430:45 +CPU= 20.272 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:47:54 STEP 1724 H= 431:00 +CPU= 20.568 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:48:16 STEP 1725 H= 431:15 +CPU= 19.347 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:48:37 STEP 1726 H= 431:30 +CPU= 18.813 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:49:00 STEP 1727 H= 431:45 +CPU= 19.049 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:49:30 STEP 1728 H= 432:00 +CPU= 28.210 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001728 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001728 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001728 15:49:52 STEP 1729 H= 432:15 +CPU= 19.870 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:50:13 STEP 1730 H= 432:30 +CPU= 19.497 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:50:35 STEP 1731 H= 432:45 +CPU= 19.423 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:51:00 STEP 1732 H= 433:00 +CPU= 20.819 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:51:22 STEP 1733 H= 433:15 +CPU= 20.498 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:51:46 STEP 1734 H= 433:30 +CPU= 21.087 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:52:08 STEP 1735 H= 433:45 +CPU= 20.679 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:52:31 STEP 1736 H= 434:00 +CPU= 20.547 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:52:52 STEP 1737 H= 434:15 +CPU= 19.475 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:53:14 STEP 1738 H= 434:30 +CPU= 19.122 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:53:35 STEP 1739 H= 434:45 +CPU= 19.605 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:54:06 STEP 1740 H= 435:00 +CPU= 26.392 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001740 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001740 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001740 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:54:29 STEP 1741 H= 435:15 +CPU= 20.976 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:54:52 STEP 1742 H= 435:30 +CPU= 21.053 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:55:15 STEP 1743 H= 435:45 +CPU= 20.789 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:55:40 STEP 1744 H= 436:00 +CPU= 21.389 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:56:03 STEP 1745 H= 436:15 +CPU= 20.931 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:56:26 STEP 1746 H= 436:30 +CPU= 20.874 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:56:49 STEP 1747 H= 436:45 +CPU= 20.763 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:57:13 STEP 1748 H= 437:00 +CPU= 21.773 15:57:33 STEP 1749 H= 437:15 +CPU= 20.538 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:57:55 STEP 1750 H= 437:30 +CPU= 19.813 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:58:17 STEP 1751 H= 437:45 +CPU= 19.603 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:58:45 STEP 1752 H= 438:00 +CPU= 25.968 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001752 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001752 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001752 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:59:07 STEP 1753 H= 438:15 +CPU= 20.251 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:59:30 STEP 1754 H= 438:30 +CPU= 20.666 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:59:53 STEP 1755 H= 438:45 +CPU= 20.774 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:00:16 STEP 1756 H= 439:00 +CPU= 21.436 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:00:41 STEP 1757 H= 439:15 +CPU= 20.534 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:01:03 STEP 1758 H= 439:30 +CPU= 20.133 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:01:25 STEP 1759 H= 439:45 +CPU= 20.040 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:01:49 STEP 1760 H= 440:00 +CPU= 21.665 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:02:11 STEP 1761 H= 440:15 +CPU= 20.411 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:02:33 STEP 1762 H= 440:30 +CPU= 20.181 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:02:55 STEP 1763 H= 440:45 +CPU= 20.228 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:03:26 STEP 1764 H= 441:00 +CPU= 26.086 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001764 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001764 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001764 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:03:48 STEP 1765 H= 441:15 +CPU= 20.157 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:04:10 STEP 1766 H= 441:30 +CPU= 20.575 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:04:33 STEP 1767 H= 441:45 +CPU= 20.696 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:04:56 STEP 1768 H= 442:00 +CPU= 21.144 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:05:21 STEP 1769 H= 442:15 +CPU= 20.279 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:05:43 STEP 1770 H= 442:30 +CPU= 20.004 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:06:05 STEP 1771 H= 442:45 +CPU= 20.376 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:06:28 STEP 1772 H= 443:00 +CPU= 21.442 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:06:51 STEP 1773 H= 443:15 +CPU= 20.574 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:07:13 STEP 1774 H= 443:30 +CPU= 20.171 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:07:35 STEP 1775 H= 443:45 +CPU= 20.140 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:08:08 STEP 1776 H= 444:00 +CPU= 28.490 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001776 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001776 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001776 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001680 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001680 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001680 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001692 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001692 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001692 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001704 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001704 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001704 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001716 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001716 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001716 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001728 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001728 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001728 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001740 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001740 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001740 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001752 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001752 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001752 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001764 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001764 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001764 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_1.zip Uploading the intermediate file: upload_file_1.zip 16:08:29 STEP 1777 H= 444:15 +CPU= 19.700 16:08:50 STEP 1778 H= 444:30 +CPU= 20.461 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:09:13 STEP 1779 H= 444:45 +CPU= 20.778 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:09:36 STEP 1780 H= 445:00 +CPU= 21.587 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:10:01 STEP 1781 H= 445:15 +CPU= 20.700 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:10:24 STEP 1782 H= 445:30 +CPU= 20.689 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:10:46 STEP 1783 H= 445:45 +CPU= 20.711 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:11:10 STEP 1784 H= 446:00 +CPU= 21.501 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:11:32 STEP 1785 H= 446:15 +CPU= 20.469 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:11:55 STEP 1786 H= 446:30 +CPU= 20.251 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:12:19 STEP 1787 H= 446:45 +CPU= 19.916 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:12:48 STEP 1788 H= 447:00 +CPU= 27.402 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001788 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001788 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001788 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:13:11 STEP 1789 H= 447:15 +CPU= 20.790 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:13:34 STEP 1790 H= 447:30 +CPU= 20.752 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:13:57 STEP 1791 H= 447:45 +CPU= 20.735 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:14:22 STEP 1792 H= 448:00 +CPU= 21.427 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:14:45 STEP 1793 H= 448:15 +CPU= 20.806 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:15:07 STEP 1794 H= 448:30 +CPU= 20.515 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:15:30 STEP 1795 H= 448:45 +CPU= 20.034 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:15:52 STEP 1796 H= 449:00 +CPU= 20.335 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:16:14 STEP 1797 H= 449:15 +CPU= 19.807 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:16:36 STEP 1798 H= 449:30 +CPU= 20.433 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:17:01 STEP 1799 H= 449:45 +CPU= 20.789 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:17:30 STEP 1800 H= 450:00 +CPU= 26.729 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001800 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001800 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001800 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:17:52 STEP 1801 H= 450:15 +CPU= 20.043 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:18:13 STEP 1802 H= 450:30 +CPU= 19.621 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:18:35 STEP 1803 H= 450:45 +CPU= 19.318 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:18:59 STEP 1804 H= 451:00 +CPU= 20.685 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:19:22 STEP 1805 H= 451:15 +CPU= 20.959 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:19:45 STEP 1806 H= 451:30 +CPU= 20.699 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:20:08 STEP 1807 H= 451:45 +CPU= 20.419 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:20:30 STEP 1808 H= 452:00 +CPU= 20.600 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:20:52 STEP 1809 H= 452:15 +CPU= 19.986 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:21:15 STEP 1810 H= 452:30 +CPU= 20.145 16:21:35 STEP 1811 H= 452:45 +CPU= 19.870 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:22:04 STEP 1812 H= 453:00 +CPU= 25.367 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001812 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001812 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001812 16:22:23 STEP 1813 H= 453:15 +CPU= 19.312 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:22:46 STEP 1814 H= 453:30 +CPU= 20.250 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:23:08 STEP 1815 H= 453:45 +CPU= 20.609 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:23:31 STEP 1816 H= 454:00 +CPU= 20.942 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:23:54 STEP 1817 H= 454:15 +CPU= 20.432 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:24:16 STEP 1818 H= 454:30 +CPU= 20.132 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:24:39 STEP 1819 H= 454:45 +CPU= 19.637 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:25:01 STEP 1820 H= 455:00 +CPU= 19.884 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:25:22 STEP 1821 H= 455:15 +CPU= 18.973 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:25:44 STEP 1822 H= 455:30 +CPU= 19.759 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:26:07 STEP 1823 H= 455:45 +CPU= 20.480 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:26:36 STEP 1824 H= 456:00 +CPU= 27.587 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001824 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001824 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001824 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:26:59 STEP 1825 H= 456:15 +CPU= 18.933 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:27:20 STEP 1826 H= 456:30 +CPU= 18.975 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:27:41 STEP 1827 H= 456:45 +CPU= 18.973 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:28:04 STEP 1828 H= 457:00 +CPU= 20.479 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:28:26 STEP 1829 H= 457:15 +CPU= 20.392 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:28:48 STEP 1830 H= 457:30 +CPU= 20.064 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:29:11 STEP 1831 H= 457:45 +CPU= 20.222 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:29:34 STEP 1832 H= 458:00 +CPU= 21.377 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:29:56 STEP 1833 H= 458:15 +CPU= 19.661 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:30:19 STEP 1834 H= 458:30 +CPU= 19.600 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:30:41 STEP 1835 H= 458:45 +CPU= 19.150 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:31:08 STEP 1836 H= 459:00 +CPU= 25.641 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001836 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001836 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001836 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:31:31 STEP 1837 H= 459:15 +CPU= 20.447 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:31:53 STEP 1838 H= 459:30 +CPU= 20.604 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:32:16 STEP 1839 H= 459:45 +CPU= 20.429 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:32:41 STEP 1840 H= 460:00 +CPU= 21.401 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:33:04 STEP 1841 H= 460:15 +CPU= 20.106 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:33:25 STEP 1842 H= 460:30 +CPU= 19.382 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:33:46 STEP 1843 H= 460:45 +CPU= 18.894 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:34:08 STEP 1844 H= 461:00 +CPU= 20.296 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:34:31 STEP 1845 H= 461:15 +CPU= 20.194 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:34:53 STEP 1846 H= 461:30 +CPU= 20.237 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:35:15 STEP 1847 H= 461:45 +CPU= 19.989 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:35:45 STEP 1848 H= 462:00 +CPU= 26.551 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001848 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001848 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001848 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:36:07 STEP 1849 H= 462:15 +CPU= 19.824 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:36:29 STEP 1850 H= 462:30 +CPU= 19.778 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:36:50 STEP 1851 H= 462:45 +CPU= 19.216 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:37:13 STEP 1852 H= 463:00 +CPU= 20.419 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:37:35 STEP 1853 H= 463:15 +CPU= 20.237 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:37:59 STEP 1854 H= 463:30 +CPU= 20.102 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:38:22 STEP 1855 H= 463:45 +CPU= 20.437 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:38:45 STEP 1856 H= 464:00 +CPU= 21.034 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:39:07 STEP 1857 H= 464:15 +CPU= 20.361 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:39:29 STEP 1858 H= 464:30 +CPU= 19.608 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:39:50 STEP 1859 H= 464:45 +CPU= 19.361 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:40:20 STEP 1860 H= 465:00 +CPU= 25.770 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001860 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001860 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001860 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:40:42 STEP 1861 H= 465:15 +CPU= 20.023 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:41:05 STEP 1862 H= 465:30 +CPU= 20.452 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:41:27 STEP 1863 H= 465:45 +CPU= 20.430 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:41:50 STEP 1864 H= 466:00 +CPU= 20.433 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:42:11 STEP 1865 H= 466:15 +CPU= 19.735 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:42:33 STEP 1866 H= 466:30 +CPU= 19.663 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:42:54 STEP 1867 H= 466:45 +CPU= 19.519 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:43:19 STEP 1868 H= 467:00 +CPU= 20.797 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:43:42 STEP 1869 H= 467:15 +CPU= 20.497 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:44:04 STEP 1870 H= 467:30 +CPU= 20.078 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:44:26 STEP 1871 H= 467:45 +CPU= 20.019 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:44:58 STEP 1872 H= 468:00 +CPU= 27.751 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001872 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001872 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001872 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001776 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001776 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001776 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001788 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001788 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001788 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001800 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001800 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001800 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001812 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001812 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001812 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001824 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001824 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001824 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001836 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001836 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001836 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001848 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001848 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001848 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001860 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001860 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001860 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_2.zip Uploading the intermediate file: upload_file_2.zip 16:45:18 STEP 1873 H= 468:15 +CPU= 18.987 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:45:39 STEP 1874 H= 468:30 +CPU= 20.068 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:46:02 STEP 1875 H= 468:45 +CPU= 20.634 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:46:25 STEP 1876 H= 469:00 +CPU= 21.254 16:46:46 STEP 1877 H= 469:15 +CPU= 20.628 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:47:09 STEP 1878 H= 469:30 +CPU= 20.424 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:47:31 STEP 1879 H= 469:45 +CPU= 20.002 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:47:53 STEP 1880 H= 470:00 +CPU= 20.297 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:48:15 STEP 1881 H= 470:15 +CPU= 19.594 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:48:39 STEP 1882 H= 470:30 +CPU= 20.044 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:49:01 STEP 1883 H= 470:45 +CPU= 20.478 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:49:30 STEP 1884 H= 471:00 +CPU= 26.441 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001884 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001884 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001884 16:49:53 STEP 1885 H= 471:15 +CPU= 19.244 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:50:14 STEP 1886 H= 471:30 +CPU= 19.395 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:50:35 STEP 1887 H= 471:45 +CPU= 19.120 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:50:59 STEP 1888 H= 472:00 +CPU= 19.756 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:51:21 STEP 1889 H= 472:15 +CPU= 19.367 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:51:43 STEP 1890 H= 472:30 +CPU= 20.125 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:52:05 STEP 1891 H= 472:45 +CPU= 20.580 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:52:29 STEP 1892 H= 473:00 +CPU= 21.104 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:52:51 STEP 1893 H= 473:15 +CPU= 20.427 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:53:14 STEP 1894 H= 473:30 +CPU= 20.471 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:53:36 STEP 1895 H= 473:45 +CPU= 19.971 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:54:06 STEP 1896 H= 474:00 +CPU= 26.138 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001896 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001896 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001896 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:54:28 STEP 1897 H= 474:15 +CPU= 19.829 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:54:49 STEP 1898 H= 474:30 +CPU= 19.783 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:55:11 STEP 1899 H= 474:45 +CPU= 19.666 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:55:34 STEP 1900 H= 475:00 +CPU= 21.205 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:55:59 STEP 1901 H= 475:15 +CPU= 20.820 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:56:22 STEP 1902 H= 475:30 +CPU= 20.674 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:56:44 STEP 1903 H= 475:45 +CPU= 19.829 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:57:06 STEP 1904 H= 476:00 +CPU= 20.218 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:57:28 STEP 1905 H= 476:15 +CPU= 20.182 16:57:49 STEP 1906 H= 476:30 +CPU= 20.681 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:58:12 STEP 1907 H= 476:45 +CPU= 20.905 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:58:43 STEP 1908 H= 477:00 +CPU= 26.992 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001908 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001908 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001908 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:59:07 STEP 1909 H= 477:15 +CPU= 21.941 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:59:31 STEP 1910 H= 477:30 +CPU= 22.050 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:59:55 STEP 1911 H= 477:45 +CPU= 22.178 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:00:22 STEP 1912 H= 478:00 +CPU= 22.339 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:00:45 STEP 1913 H= 478:15 +CPU= 21.491 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:01:10 STEP 1914 H= 478:30 +CPU= 22.330 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:01:35 STEP 1915 H= 478:45 +CPU= 23.402 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:02:04 STEP 1916 H= 479:00 +CPU= 24.486 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:02:30 STEP 1917 H= 479:15 +CPU= 23.685 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:02:53 STEP 1918 H= 479:30 +CPU= 21.625 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:03:19 STEP 1919 H= 479:45 +CPU= 21.597 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:03:50 STEP 1920 H= 480:00 +CPU= 29.136 ..The child process terminated with status: 0 >>> Printing last 70 lines from file: NODE.001_01 DDH------------------------------------------------------------------------ I I I TIME = 480.000H DDH * Diagnostic Var., AREA GLOBAL I I I I-------------------------------------------------------------------------I I VARIABLE I Initial value I Current value I I-------------------------------------------------------------------------I I OMEGA (P/S) I -0.2858338309E-21 I 0.2173645409E-05 I ICloud fractionI 0.0000000000E+00 I 0.1518838675E-01 I I Relat. Hum. I 0.2190151138E+00 I 0.2719843779E+00 I I PASS 01 I 0.0000000000E+00 I 0.2714301623E+03 I DDH------------------------------------------------------------------------ MAXGPFV : MAX. VALUE = 0.000000000000000E+000 MAXGPFV : MAX. VALUE = 0.000000000000000E+000 NSTEP = 1920 SCAN2M_HPOS P IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH = ICMGGh7zg+001920 MODE=a IO-STREAM CLOSED - ICMGGh7zg+001920 MPI-TASK: 1 - 319004980 BYTES IN 7 RECORDS TRANSFERRED IN 0.0006 SECONDS531675.0 Mbytes/s, TOTAL TIME= 0.0100( 6.0%) 17:03:50 STEP 1920 H= 480:00 +CPU= 29.136 END CNT3 NSTEP = 1920 CNT0 000000000 21.652 21.652 23.675 0.915 0 0 0 0 0 IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ 46/ 1069125708 WRITTEN 7150/************ -TASK-OPENED-OPEN-RECS IN -KBYTE IN -RECS OUT-KBYTE OUT-WALL -WALL IN-WALL OU-TOT IN -TOT OUT 1 128 0 46 1044068 7150 ************ 12.2 1.5 10.7 2.3 21.0 ===-=== START OF TIMING STATISTICS ===-=== STATS FOR ALL TASKS NUM ROUTINE CALLS MEAN(ms) MAX(ms) FRAC(%) UNBAL(%) 0 CNT0 - COMPLETE EXECUTION 1 5557850.8 5557850.8 100.00 0.00 1 CNT4 - FORWARD INTEGRATION 1 5554189.2 5554189.2 99.93 0.00 8 SCAN2M - GRID-POINT DYNAMICS 261 3774.2 3774.2 17.72 0.00 9 SPCM - SPECTRAL COMP. 240 430.6 430.6 1.86 0.00 10 SCAN2M - PHYSICS 241 10981.7 10981.7 47.62 0.00 11 IOPACK - OUTPUT P.P. RESULTS 20 709.1 709.1 0.26 0.00 12 SPNORM - SPECTRAL NORM COMP. 11 24.4 24.4 0.00 0.00 17 GRIDFPOS IN CNT4 20 21.1 21.1 0.01 0.00 27 RESTART FILES - WRITING 5 2835.0 2835.0 0.26 0.00 28 RESTART FILES - READING 1 3000.3 3000.3 0.05 0.00 29 SU4FPOS IN CNT4 20 0.0 0.0 0.00 0.00 30 DYNFPOS IN CNT4 20 5538.8 5538.8 1.99 0.00 31 POSDDH IN STEPO 2 1.1 1.1 0.00 0.00 37 CPGLAG - SL COMPUTATIONS 241 5298.9 5298.9 22.98 0.00 39 SU0YOMB 1 293.6 293.6 0.01 0.00 51 SCAN2M - SL COMM. PART 1 241 11.9 11.9 0.05 0.00 54 SPCM - M TO S/S TO M TRANSP. 240 72.2 72.2 0.31 0.00 55 SPCIMPF - S TO M/M TO S TRANSP. 240 38.7 38.7 0.17 0.00 56 SPNORM - SPECTRAL NORM COMM. 11 0.1 0.1 0.00 0.00 102 LTINV_CTL - INVERSE LEGENDRE TRANSFORM 524 290.3 290.3 2.74 0.00 103 LTDIR_CTL - DIRECT LEGENDRE TRANSFORM 500 197.9 197.9 1.78 0.00 106 FTDIR_CTL - DIRECT FOURIER TRANSFORM 500 59.0 59.0 0.53 0.00 107 FTINV_CTL - INVERSE FOURIER TRANSFORM 524 93.1 93.1 0.88 0.00 140 SULEG - COMP. OF LEGENDRE POL. 1 38.0 38.0 0.00 0.00 152 LTINV_CTL - M TO L TRANSPOSITION 524 40.2 40.2 0.38 0.00 153 LTDIR_CTL - L TO M TRANSPOSITION 500 20.7 20.7 0.19 0.00 157 FTINV_CTL - L TO G TRANSPOSITION 524 43.7 43.7 0.41 0.00 158 FTDIR_CTL - G TO L TRANSPOSITION 500 26.1 26.1 0.23 0.00 400 GSTATS 28864 0.0 0.0 0.00 0.00 401 GSTATS HOOK 26802 0.0 0.0 0.00 0.00 TOTAL MEASURED IMBALANCE = 0.0 SECONDS, 0.0 PERCENT TOTAL WALLCLOCK TIME 5557.9 CPU TIME 5007.1 VECTOR TIME 5007.1 ===-=== END OF TIMING STATISTICS ===-=== FORECAST DAYS PER DAY 311.1 *** END CNT0 *** ------------------------------------------------ Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMGGh7zg+001920 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMSHh7zg+001920 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/0/ICMUAh7zg+001920 Adding to the zip: /mnt/nvme/scratch1/boinc/slots/0/NODE.001_01 Adding to the zip: /mnt/nvme/scratch1/boinc/slots/0/ifs.stat Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001872 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001872 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001872 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001884 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001884 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001884 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001896 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001896 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001896 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001908 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001908 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001908 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMGGh7zg+001920 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMSHh7zg+001920 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_12290437/ICMUAh7zg+001920 Zipping up the final file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_3.zip Uploading the final file: upload_file_3.zip Uploading trickle at timestep: 1727100 17:07:16 (341736): called boinc_finish(0) </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_17.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_18.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0_r487374498_19.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
17 Jun 2024 15:06:33 | 1489681 | 22444030 | oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0 | 1,727,100 | 5,007 | 0.0029 |
16 Jun 2024 15:44:21 | 1489681 | 22444030 | oifs_43r3_bl_a1mm_2016092300_20_1018_12290437_0 | 864,000 | 20,756 | 0.0240 |
©2024 cpdn.org