climateprediction.net home page
Task 22448819

Task 22448819

Name oifs_43r3_bl_a3tz_2016092300_20_1018_12293294_1
Workunit 12293294
Created 14 Jun 2024, 12:06:21 UTC
Sent 14 Jun 2024, 12:06:42 UTC
Report deadline 13 Aug 2024, 12:06:42 UTC
Received 15 Jun 2024, 18:43:39 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1550182
Run time 21 hours 17 min 33 sec
CPU time 21 hours 17 min 33 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 1.83 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,023.33 MB
Peak swap size 5,981.07 MB
Peak disk usage 2,023.60 MB
Stderr
<core_client_version>8.1.0</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:penguin:1:1:303:303] [20240615:045734:1718441854:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:58:46 STEP  816 H= 204:00 +CPU= 57.971
  04:59:42 STEP  817 H= 204:15 +CPU= 52.590
  05:00:36 STEP  818 H= 204:30 +CPU= 51.027
  05:01:29 STEP  819 H= 204:45 +CPU= 50.932
  05:02:23 STEP  820 H= 205:00 +CPU= 51.263
  05:03:17 STEP  821 H= 205:15 +CPU= 51.873
  05:04:14 STEP  822 H= 205:30 +CPU= 54.246
  05:05:10 STEP  823 H= 205:45 +CPU= 53.873
  05:06:05 STEP  824 H= 206:00 +CPU= 52.395
  05:06:59 STEP  825 H= 206:15 +CPU= 51.868
  05:07:51 STEP  826 H= 206:30 +CPU= 49.473
  05:08:47 STEP  827 H= 206:45 +CPU= 52.939
  05:09:52 STEP  828 H= 207:00 +CPU= 61.747
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000828
  05:10:46 STEP  829 H= 207:15 +CPU= 51.989
  05:11:39 STEP  830 H= 207:30 +CPU= 50.760
  05:12:32 STEP  831 H= 207:45 +CPU= 51.164
  05:13:23 STEP  832 H= 208:00 +CPU= 49.418
  05:14:14 STEP  833 H= 208:15 +CPU= 49.448
  05:15:08 STEP  834 H= 208:30 +CPU= 52.444
  05:16:03 STEP  835 H= 208:45 +CPU= 53.472
  05:16:54 STEP  836 H= 209:00 +CPU= 49.511
  05:17:45 STEP  837 H= 209:15 +CPU= 48.982
  05:18:35 STEP  838 H= 209:30 +CPU= 49.387
  05:19:26 STEP  839 H= 209:45 +CPU= 49.577
  05:20:28 STEP  840 H= 210:00 +CPU= 60.037
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000840
  05:21:21 STEP  841 H= 210:15 +CPU= 51.360
  05:22:12 STEP  842 H= 210:30 +CPU= 49.342
  05:23:01 STEP  843 H= 210:45 +CPU= 47.443
  05:23:50 STEP  844 H= 211:00 +CPU= 48.481
  05:24:41 STEP  845 H= 211:15 +CPU= 48.756
  05:25:31 STEP  846 H= 211:30 +CPU= 48.352
  05:26:25 STEP  847 H= 211:45 +CPU= 51.525
  05:27:17 STEP  848 H= 212:00 +CPU= 50.275
  05:28:05 STEP  849 H= 212:15 +CPU= 47.310
  05:28:55 STEP  850 H= 212:30 +CPU= 48.230
  05:29:44 STEP  851 H= 212:45 +CPU= 47.253
  05:30:44 STEP  852 H= 213:00 +CPU= 58.257
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000852
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000852
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000852
  05:31:35 STEP  853 H= 213:15 +CPU= 50.063
  05:32:25 STEP  854 H= 213:30 +CPU= 48.700
  05:33:16 STEP  855 H= 213:45 +CPU= 49.125
  05:34:05 STEP  856 H= 214:00 +CPU= 47.015
  05:34:57 STEP  857 H= 214:15 +CPU= 49.235
  05:35:45 STEP  858 H= 214:30 +CPU= 45.813
  05:36:35 STEP  859 H= 214:45 +CPU= 47.267
  05:37:25 STEP  860 H= 215:00 +CPU= 48.692
  05:38:14 STEP  861 H= 215:15 +CPU= 46.689
  05:39:03 STEP  862 H= 215:30 +CPU= 46.802
  05:39:51 STEP  863 H= 215:45 +CPU= 46.673
  05:40:57 STEP  864 H= 216:00 +CPU= 62.993
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000864
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000864
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000852
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tz_2016092300_20_1018_12293294_1_r155626358_8.zip
Uploading the intermediate file: upload_file_8.zip
  05:41:50 STEP  865 H= 216:15 +CPU= 48.842
  05:42:39 STEP  866 H= 216:30 +CPU= 47.126
  05:43:29 STEP  867 H= 216:45 +CPU= 48.187
  05:44:18 STEP  868 H= 217:00 +CPU= 47.162
  05:45:11 STEP  869 H= 217:15 +CPU= 51.459
  05:46:04 STEP  870 H= 217:30 +CPU= 51.398
  05:46:53 STEP  871 H= 217:45 +CPU= 47.785
  05:47:44 STEP  872 H= 218:00 +CPU= 48.420
  05:48:34 STEP  873 H= 218:15 +CPU= 48.709
  05:49:25 STEP  874 H= 218:30 +CPU= 48.549
  05:50:14 STEP  875 H= 218:45 +CPU= 47.787
  05:51:18 STEP  876 H= 219:00 +CPU= 61.013
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000876
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000876
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000876
  05:52:10 STEP  877 H= 219:15 +CPU= 50.050
  05:53:01 STEP  878 H= 219:30 +CPU= 48.480
  05:53:52 STEP  879 H= 219:45 +CPU= 49.297
  05:54:47 STEP  880 H= 220:00 +CPU= 52.307
  05:55:41 STEP  881 H= 220:15 +CPU= 51.866
  05:56:37 STEP  882 H= 220:30 +CPU= 52.651
  05:57:33 STEP  883 H= 220:45 +CPU= 53.053
  05:58:26 STEP  884 H= 221:00 +CPU= 50.969
  05:59:24 STEP  885 H= 221:15 +CPU= 54.953
  06:00:19 STEP  886 H= 221:30 +CPU= 52.108
  06:01:13 STEP  887 H= 221:45 +CPU= 51.514
  06:02:21 STEP  888 H= 222:00 +CPU= 64.244
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000888
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000888
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000888
  06:03:17 STEP  889 H= 222:15 +CPU= 53.626
  06:04:13 STEP  890 H= 222:30 +CPU= 53.866
  06:05:08 STEP  891 H= 222:45 +CPU= 52.372
  06:06:03 STEP  892 H= 223:00 +CPU= 52.292
  06:06:59 STEP  893 H= 223:15 +CPU= 51.840
  06:07:54 STEP  894 H= 223:30 +CPU= 52.422
  06:08:52 STEP  895 H= 223:45 +CPU= 55.140
  06:09:45 STEP  896 H= 224:00 +CPU= 50.907
  06:10:38 STEP  897 H= 224:15 +CPU= 50.378
  06:11:31 STEP  898 H= 224:30 +CPU= 50.613
  06:12:27 STEP  899 H= 224:45 +CPU= 52.392
  06:13:40 STEP  900 H= 225:00 +CPU= 69.463
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000900
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000900
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000900
  06:14:38 STEP  901 H= 225:15 +CPU= 55.533
  06:15:35 STEP  902 H= 225:30 +CPU= 53.887
  06:16:27 STEP  903 H= 225:45 +CPU= 49.933
  06:17:23 STEP  904 H= 226:00 +CPU= 53.336
  06:18:19 STEP  905 H= 226:15 +CPU= 54.247
  06:19:15 STEP  906 H= 226:30 +CPU= 52.776
  06:20:10 STEP  907 H= 226:45 +CPU= 52.631
  06:21:04 STEP  908 H= 227:00 +CPU= 52.308
  06:21:59 STEP  909 H= 227:15 +CPU= 53.077
  06:22:53 STEP  910 H= 227:30 +CPU= 51.102
  06:23:47 STEP  911 H= 227:45 +CPU= 51.051
  06:25:05 STEP  912 H= 228:00 +CPU= 75.082
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000912
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000912
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000912
  06:26:02 STEP  913 H= 228:15 +CPU= 53.137
  06:26:58 STEP  914 H= 228:30 +CPU= 53.773
  06:27:54 STEP  915 H= 228:45 +CPU= 51.899
  06:28:50 STEP  916 H= 229:00 +CPU= 53.115
  06:29:43 STEP  917 H= 229:15 +CPU= 51.115
  06:30:40 STEP  918 H= 229:30 +CPU= 54.001
  06:31:36 STEP  919 H= 229:45 +CPU= 53.701
  06:32:31 STEP  920 H= 230:00 +CPU= 52.402
  06:33:26 STEP  921 H= 230:15 +CPU= 52.385
  06:34:20 STEP  922 H= 230:30 +CPU= 50.903
  06:35:16 STEP  923 H= 230:45 +CPU= 52.726
  06:36:22 STEP  924 H= 231:00 +CPU= 62.001
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000924
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000924
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000924
  06:37:15 STEP  925 H= 231:15 +CPU= 50.342
  06:38:09 STEP  926 H= 231:30 +CPU= 51.407
  06:39:00 STEP  927 H= 231:45 +CPU= 48.358
  06:39:53 STEP  928 H= 232:00 +CPU= 50.552
  06:40:47 STEP  929 H= 232:15 +CPU= 50.839
  06:41:42 STEP  930 H= 232:30 +CPU= 51.980
  06:42:36 STEP  931 H= 232:45 +CPU= 51.786
  06:43:32 STEP  932 H= 233:00 +CPU= 53.106
  06:44:25 STEP  933 H= 233:15 +CPU= 51.347
  06:45:18 STEP  934 H= 233:30 +CPU= 50.480
  06:46:10 STEP  935 H= 233:45 +CPU= 49.650
  06:47:19 STEP  936 H= 234:00 +CPU= 64.539
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000936
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000936
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000936
  06:48:22 STEP  937 H= 234:15 +CPU= 61.713
  06:49:21 STEP  938 H= 234:30 +CPU= 55.735
  06:50:16 STEP  939 H= 234:45 +CPU= 51.293
  06:51:13 STEP  940 H= 235:00 +CPU= 54.960
  06:52:10 STEP  941 H= 235:15 +CPU= 52.546
  06:53:08 STEP  942 H= 235:30 +CPU= 54.139
  06:54:07 STEP  943 H= 235:45 +CPU= 55.685
  06:55:07 STEP  944 H= 236:00 +CPU= 56.629
  06:56:02 STEP  945 H= 236:15 +CPU= 52.818
  06:56:56 STEP  946 H= 236:30 +CPU= 51.500
  06:57:52 STEP  947 H= 236:45 +CPU= 52.052
  06:58:59 STEP  948 H= 237:00 +CPU= 63.503
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000948
  06:59:59 STEP  949 H= 237:15 +CPU= 57.450
  07:00:53 STEP  950 H= 237:30 +CPU= 51.210
  07:01:44 STEP  951 H= 237:45 +CPU= 49.279
  07:02:41 STEP  952 H= 238:00 +CPU= 54.148
  07:03:39 STEP  953 H= 238:15 +CPU= 54.981
  07:04:32 STEP  954 H= 238:30 +CPU= 49.777
  07:05:26 STEP  955 H= 238:45 +CPU= 52.600
  07:06:21 STEP  956 H= 239:00 +CPU= 52.225
  07:07:16 STEP  957 H= 239:15 +CPU= 52.249
  07:08:09 STEP  958 H= 239:30 +CPU= 50.753
  07:09:03 STEP  959 H= 239:45 +CPU= 50.522
  07:10:19 STEP  960 H= 240:00 +CPU= 72.513
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tz_2016092300_20_1018_12293294_1_r155626358_9.zip
Uploading the intermediate file: upload_file_9.zip
  07:11:11 STEP  961 H= 240:15 +CPU= 48.998
  07:12:13 STEP  962 H= 240:30 +CPU= 58.351
  07:13:14 STEP  963 H= 240:45 +CPU= 57.445
  07:14:13 STEP  964 H= 241:00 +CPU= 56.144
  07:15:12 STEP  965 H= 241:15 +CPU= 56.870
  07:16:11 STEP  966 H= 241:30 +CPU= 56.092
  07:17:08 STEP  967 H= 241:45 +CPU= 54.347
  07:18:04 STEP  968 H= 242:00 +CPU= 53.918
  07:18:58 STEP  969 H= 242:15 +CPU= 51.201
  07:19:54 STEP  970 H= 242:30 +CPU= 52.703
  07:20:47 STEP  971 H= 242:45 +CPU= 51.038
  07:21:53 STEP  972 H= 243:00 +CPU= 62.745
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000972
  07:22:50 STEP  973 H= 243:15 +CPU= 54.531
  07:23:47 STEP  974 H= 243:30 +CPU= 54.344
  07:24:41 STEP  975 H= 243:45 +CPU= 51.735
  07:25:38 STEP  976 H= 244:00 +CPU= 53.559
  07:26:33 STEP  977 H= 244:15 +CPU= 53.170
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:54:06 STEP  978 H= 244:30 +CPU= 56.383
  07:55:07 STEP  979 H= 244:45 +CPU= 56.693
  07:56:11 STEP  980 H= 245:00 +CPU= 61.368
  07:57:17 STEP  981 H= 245:15 +CPU= 61.997
  07:58:25 STEP  982 H= 245:30 +CPU= 64.899
  07:59:27 STEP  983 H= 245:45 +CPU= 58.828
  08:00:38 STEP  984 H= 246:00 +CPU= 67.414
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000984
  08:01:37 STEP  985 H= 246:15 +CPU= 55.600
  08:02:37 STEP  986 H= 246:30 +CPU= 56.921
  08:03:38 STEP  987 H= 246:45 +CPU= 57.861
  08:04:39 STEP  988 H= 247:00 +CPU= 58.810
  08:05:38 STEP  989 H= 247:15 +CPU= 56.279
  08:06:37 STEP  990 H= 247:30 +CPU= 55.700
  08:07:55 STEP  991 H= 247:45 +CPU= 73.157
  08:08:59 STEP  992 H= 248:00 +CPU= 59.903
  08:09:59 STEP  993 H= 248:15 +CPU= 57.109
  08:11:00 STEP  994 H= 248:30 +CPU= 58.196
  08:12:00 STEP  995 H= 248:45 +CPU= 57.028
  08:13:19 STEP  996 H= 249:00 +CPU= 76.256
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+000996
  08:14:20 STEP  997 H= 249:15 +CPU= 58.525
  08:15:21 STEP  998 H= 249:30 +CPU= 57.667
  08:16:21 STEP  999 H= 249:45 +CPU= 57.833
  08:17:25 STEP 1000 H= 250:00 +CPU= 60.142
  08:18:23 STEP 1001 H= 250:15 +CPU= 54.978
  08:19:24 STEP 1002 H= 250:30 +CPU= 59.426
  08:20:23 STEP 1003 H= 250:45 +CPU= 55.256
  08:21:21 STEP 1004 H= 251:00 +CPU= 55.299
  08:22:18 STEP 1005 H= 251:15 +CPU= 54.140
  08:23:14 STEP 1006 H= 251:30 +CPU= 54.156
  08:24:13 STEP 1007 H= 251:45 +CPU= 55.292
  08:25:30 STEP 1008 H= 252:00 +CPU= 73.941
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001008
  08:26:41 STEP 1009 H= 252:15 +CPU= 65.886
  08:27:49 STEP 1010 H= 252:30 +CPU= 65.716
  08:28:51 STEP 1011 H= 252:45 +CPU= 59.200
  08:29:50 STEP 1012 H= 253:00 +CPU= 56.743
  08:30:53 STEP 1013 H= 253:15 +CPU= 59.393
  08:31:51 STEP 1014 H= 253:30 +CPU= 55.747
  08:32:50 STEP 1015 H= 253:45 +CPU= 55.697
  08:33:53 STEP 1016 H= 254:00 +CPU= 58.105
  08:34:53 STEP 1017 H= 254:15 +CPU= 56.676
  08:35:53 STEP 1018 H= 254:30 +CPU= 56.940
  08:36:46 STEP 1019 H= 254:45 +CPU= 50.412
  08:38:04 STEP 1020 H= 255:00 +CPU= 73.827
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001020
  08:39:11 STEP 1021 H= 255:15 +CPU= 62.344
  08:40:11 STEP 1022 H= 255:30 +CPU= 57.703
  08:41:12 STEP 1023 H= 255:45 +CPU= 57.971
  08:42:17 STEP 1024 H= 256:00 +CPU= 60.968
  08:43:30 STEP 1025 H= 256:15 +CPU= 68.893
  08:44:39 STEP 1026 H= 256:30 +CPU= 64.864
  08:45:55 STEP 1027 H= 256:45 +CPU= 71.324
  08:47:13 STEP 1028 H= 257:00 +CPU= 72.158
  08:48:29 STEP 1029 H= 257:15 +CPU= 71.881
  08:49:38 STEP 1030 H= 257:30 +CPU= 65.151
  08:50:41 STEP 1031 H= 257:45 +CPU= 60.451
  08:52:03 STEP 1032 H= 258:00 +CPU= 75.520
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001032
  08:53:06 STEP 1033 H= 258:15 +CPU= 59.029
  08:54:09 STEP 1034 H= 258:30 +CPU= 59.692
  08:55:10 STEP 1035 H= 258:45 +CPU= 58.081
  08:56:13 STEP 1036 H= 259:00 +CPU= 59.820
  08:57:14 STEP 1037 H= 259:15 +CPU= 57.899
  08:58:14 STEP 1038 H= 259:30 +CPU= 57.537
  08:59:13 STEP 1039 H= 259:45 +CPU= 56.550
  09:00:14 STEP 1040 H= 260:00 +CPU= 58.695
  09:01:15 STEP 1041 H= 260:15 +CPU= 57.476
  09:02:15 STEP 1042 H= 260:30 +CPU= 56.505
  09:03:13 STEP 1043 H= 260:45 +CPU= 54.952
  09:04:28 STEP 1044 H= 261:00 +CPU= 69.682
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001044
  09:05:31 STEP 1045 H= 261:15 +CPU= 58.688
  09:06:31 STEP 1046 H= 261:30 +CPU= 56.425
  09:07:30 STEP 1047 H= 261:45 +CPU= 54.870
  09:08:25 STEP 1048 H= 262:00 +CPU= 52.262
  09:09:25 STEP 1049 H= 262:15 +CPU= 56.061
  09:10:38 STEP 1050 H= 262:30 +CPU= 69.544
  09:11:43 STEP 1051 H= 262:45 +CPU= 60.828
  09:12:56 STEP 1052 H= 263:00 +CPU= 69.198
  09:14:01 STEP 1053 H= 263:15 +CPU= 62.240
  09:15:03 STEP 1054 H= 263:30 +CPU= 59.169
  09:16:09 STEP 1055 H= 263:45 +CPU= 61.878
  09:17:31 STEP 1056 H= 264:00 +CPU= 77.393
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tz_2016092300_20_1018_12293294_1_r155626358_10.zip
Uploading the intermediate file: upload_file_10.zip
  09:18:35 STEP 1057 H= 264:15 +CPU= 57.811
  09:19:37 STEP 1058 H= 264:30 +CPU= 57.372
  09:20:37 STEP 1059 H= 264:45 +CPU= 57.708
  09:21:37 STEP 1060 H= 265:00 +CPU= 56.704
  09:22:31 STEP 1061 H= 265:15 +CPU= 51.177
  09:23:28 STEP 1062 H= 265:30 +CPU= 53.551
  09:24:26 STEP 1063 H= 265:45 +CPU= 54.735
  09:25:24 STEP 1064 H= 266:00 +CPU= 55.276
  09:26:21 STEP 1065 H= 266:15 +CPU= 54.185
  09:27:19 STEP 1066 H= 266:30 +CPU= 55.028
  09:28:17 STEP 1067 H= 266:45 +CPU= 54.622
  09:29:36 STEP 1068 H= 267:00 +CPU= 73.984
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001068
  09:30:40 STEP 1069 H= 267:15 +CPU= 60.301
  09:31:44 STEP 1070 H= 267:30 +CPU= 59.922
  09:32:54 STEP 1071 H= 267:45 +CPU= 66.074
  09:34:06 STEP 1072 H= 268:00 +CPU= 68.401
  09:35:10 STEP 1073 H= 268:15 +CPU= 61.270
  09:36:09 STEP 1074 H= 268:30 +CPU= 55.617
  09:37:08 STEP 1075 H= 268:45 +CPU= 56.404
  09:38:10 STEP 1076 H= 269:00 +CPU= 59.097
  09:39:13 STEP 1077 H= 269:15 +CPU= 58.322
  09:40:23 STEP 1078 H= 269:30 +CPU= 66.145
  09:41:24 STEP 1079 H= 269:45 +CPU= 58.180
  09:42:39 STEP 1080 H= 270:00 +CPU= 71.757
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001080
  09:43:37 STEP 1081 H= 270:15 +CPU= 55.228
  09:44:34 STEP 1082 H= 270:30 +CPU= 54.075
  09:45:37 STEP 1083 H= 270:45 +CPU= 59.863
  09:46:42 STEP 1084 H= 271:00 +CPU= 61.729
  09:47:47 STEP 1085 H= 271:15 +CPU= 62.563
  09:48:52 STEP 1086 H= 271:30 +CPU= 61.458
  09:49:58 STEP 1087 H= 271:45 +CPU= 63.612
  09:51:07 STEP 1088 H= 272:00 +CPU= 65.014
  12:00:39 STEP 1089 H= 272:15 +CPU=*******
  12:01:21 STEP 1090 H= 272:30 +CPU= 39.357
  12:02:12 STEP 1091 H= 272:45 +CPU= 48.424
  12:03:21 STEP 1092 H= 273:00 +CPU= 65.218
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001092
  12:04:14 STEP 1093 H= 273:15 +CPU= 49.950
  12:05:15 STEP 1094 H= 273:30 +CPU= 57.422
  12:06:22 STEP 1095 H= 273:45 +CPU= 62.463
  12:07:33 STEP 1096 H= 274:00 +CPU= 66.898
  12:08:49 STEP 1097 H= 274:15 +CPU= 71.299
  12:09:56 STEP 1098 H= 274:30 +CPU= 64.817
  12:11:01 STEP 1099 H= 274:45 +CPU= 61.845
  12:12:08 STEP 1100 H= 275:00 +CPU= 63.127
  12:13:12 STEP 1101 H= 275:15 +CPU= 61.471
  12:14:18 STEP 1102 H= 275:30 +CPU= 62.321
  12:15:23 STEP 1103 H= 275:45 +CPU= 61.513
  12:16:52 STEP 1104 H= 276:00 +CPU= 81.212
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001104
  12:18:04 STEP 1105 H= 276:15 +CPU= 67.349
  12:19:13 STEP 1106 H= 276:30 +CPU= 65.517
  12:20:20 STEP 1107 H= 276:45 +CPU= 62.016
  12:21:29 STEP 1108 H= 277:00 +CPU= 64.742
  12:22:42 STEP 1109 H= 277:15 +CPU= 68.543
  12:23:44 STEP 1110 H= 277:30 +CPU= 58.277
  12:24:43 STEP 1111 H= 277:45 +CPU= 56.803
  12:25:44 STEP 1112 H= 278:00 +CPU= 56.965
  12:26:44 STEP 1113 H= 278:15 +CPU= 57.198
  12:27:43 STEP 1114 H= 278:30 +CPU= 56.047
  12:28:42 STEP 1115 H= 278:45 +CPU= 56.031
  12:29:53 STEP 1116 H= 279:00 +CPU= 67.059
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001116
  12:30:46 STEP 1117 H= 279:15 +CPU= 50.799
  12:31:40 STEP 1118 H= 279:30 +CPU= 52.011
  12:32:35 STEP 1119 H= 279:45 +CPU= 51.708
  12:33:30 STEP 1120 H= 280:00 +CPU= 52.649
  12:34:23 STEP 1121 H= 280:15 +CPU= 50.739
  12:35:17 STEP 1122 H= 280:30 +CPU= 51.256
  12:36:12 STEP 1123 H= 280:45 +CPU= 52.354
  12:37:14 STEP 1124 H= 281:00 +CPU= 57.440
  12:38:12 STEP 1125 H= 281:15 +CPU= 54.965
  12:39:08 STEP 1126 H= 281:30 +CPU= 53.493
  12:40:13 STEP 1127 H= 281:45 +CPU= 61.679
  12:41:26 STEP 1128 H= 282:00 +CPU= 69.899
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001128
  12:42:23 STEP 1129 H= 282:15 +CPU= 54.459
  12:43:22 STEP 1130 H= 282:30 +CPU= 56.778
  12:44:20 STEP 1131 H= 282:45 +CPU= 55.936
  12:45:16 STEP 1132 H= 283:00 +CPU= 54.189
  12:46:13 STEP 1133 H= 283:15 +CPU= 54.544
  12:47:10 STEP 1134 H= 283:30 +CPU= 55.214
  12:48:07 STEP 1135 H= 283:45 +CPU= 53.516
  12:49:06 STEP 1136 H= 284:00 +CPU= 56.472
  12:50:05 STEP 1137 H= 284:15 +CPU= 56.116
  12:51:01 STEP 1138 H= 284:30 +CPU= 51.707
  12:52:00 STEP 1139 H= 284:45 +CPU= 55.971
  12:53:09 STEP 1140 H= 285:00 +CPU= 66.583
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001140
  12:54:06 STEP 1141 H= 285:15 +CPU= 54.612
  12:55:03 STEP 1142 H= 285:30 +CPU= 55.231
  12:56:06 STEP 1143 H= 285:45 +CPU= 59.171
  12:57:10 STEP 1144 H= 286:00 +CPU= 61.252
  12:58:09 STEP 1145 H= 286:15 +CPU= 56.852
  12:59:03 STEP 1146 H= 286:30 +CPU= 52.169
  13:00:01 STEP 1147 H= 286:45 +CPU= 55.914
  13:00:59 STEP 1148 H= 287:00 +CPU= 55.360
  13:01:58 STEP 1149 H= 287:15 +CPU= 56.401
  13:02:57 STEP 1150 H= 287:30 +CPU= 56.455
  13:03:53 STEP 1151 H= 287:45 +CPU= 54.002
  13:05:07 STEP 1152 H= 288:00 +CPU= 70.943
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12293294/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tz_2016092300_20_1018_12293294_1_r155626358_11.zip
Uploading the intermediate file: upload_file_11.zip
  13:06:03 STEP 1153 H= 288:15 +CPU= 51.899
  13:07:01 STEP 1154 H= 288:30 +CPU= 54.823
  13:07:58 STEP 1155 H= 288:45 +CPU= 54.372
  13:08:56 STEP 1156 H= 289:00 +CPU= 55.492
  13:09:53 STEP 1157 H= 289:15 +CPU= 54.294
  13:10:48 STEP 1158 H= 289:30 +CPU= 52.335
  13:11:47 STEP 1159 H= 289:45 +CPU= 57.759
  13:12:52 STEP 1160 H= 290:00 +CPU= 62.330
  13:13:58 STEP 1161 H= 290:15 +CPU= 62.673
  13:15:06 STEP 1162 H= 290:30 +CPU= 64.038
  13:16:12 STEP 1163 H= 290:45 +CPU= 62.505
  13:17:38 STEP 1164 H= 291:00 +CPU= 83.382
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001164
  13:18:48 STEP 1165 H= 291:15 +CPU= 67.138
  13:19:55 STEP 1166 H= 291:30 +CPU= 63.933
  13:20:57 STEP 1167 H= 291:45 +CPU= 59.691
  13:22:05 STEP 1168 H= 292:00 +CPU= 65.924
  13:23:11 STEP 1169 H= 292:15 +CPU= 62.896
  13:24:16 STEP 1170 H= 292:30 +CPU= 61.341
  13:25:21 STEP 1171 H= 292:45 +CPU= 61.708
  13:26:26 STEP 1172 H= 293:00 +CPU= 61.767
  13:27:35 STEP 1173 H= 293:15 +CPU= 67.480
  13:28:39 STEP 1174 H= 293:30 +CPU= 60.945
  13:29:39 STEP 1175 H= 293:45 +CPU= 56.816
  13:31:07 STEP 1176 H= 294:00 +CPU= 83.275
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAh7zg+001176
  13:32:13 STEP 1177 H= 294:15 +CPU= 61.651
  13:33:17 STEP 1178 H= 294:30 +CPU= 61.705
  13:34:21 STEP 1179 H= 294:45 +CPU= 60.782
  13:35:20 STEP 1180 H= 295:00 +CPU= 56.901
  13:36:24 STEP 1181 H= 295:15 +CPU= 59.773
  13:37:28 STEP 1182 H= 295:30 +CPU= 61.286
  13:38:28 STEP 1183 H= 295:45 +CPU= 58.612
  13:39:27 STEP 1184 H= 296:00 +CPU= 56.801
  13:40:26 STEP 1185 H= 296:15 +CPU= 56.794
  13:41:26 STEP 1186 H= 296:30 +CPU= 56.727
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.936] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 3958MB (heap), 4795MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.940] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.941] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.942] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.942] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.942] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3958 MB (maxheap), 4795 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.943] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.943] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134207:1718473327:31472.944] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc/slots/13/oifs_43r3_model.exe' : sigcontextptr=0x7ffd65563ed0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc/slots/13/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31473.952] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc/slots/13/oifs_43r3_model.exe' : sigcontextptr=0x7ffd65563de0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc/slots/13/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31474.200] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:penguin:1:1:303:303] [20240615:134208:1718473328:31474.200] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.109495776529898E-04 0.000000000000000E+00 0.558627248305954E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.367925304563671E-05 0.000000000000000E+00 0.187215715088511E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.960021638124478E-06 0.000000000000000E+00 0.100596896958363E-01
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.237352565223709E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1176 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1176 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001176 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001176
 MPI-TASK:   1 -    315087668 BYTES IN      6 RECORDS TRANSFERRED IN    0.0004 SECONDS787719.2 Mbytes/s, TOTAL TIME=    0.0218( 1.8%)
 NSTEP =  1176 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001176 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001176
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0178 SECONDS 39741.5 Mbytes/s, TOTAL TIME=    0.7639( 2.3%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001176 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001176
 MPI-TASK:   1 -   5041402688 BYTES IN    125 RECORDS TRANSFERRED IN    0.0097 SECONDS519732.2 Mbytes/s, TOTAL TIME=    0.5385( 1.8%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001176 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001176
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0004 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0066( 6.1%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001176 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001176
 MPI-TASK:   1 -    306507939 BYTES IN      4 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0137( 2.2%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1176 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1176 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001176 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001176
 MPI-TASK:   1 -    315087668 BYTES IN      7 RECORDS TRANSFERRED IN    0.0706 SECONDS  4463.0 Mbytes/s, TOTAL TIME=    0.1000(70.6%)
 13:31:07 STEP 1176 H= 294:00 +CPU= 83.275
 NSTEP =  1177 STEPO   0AAA00AAA
 13:32:13 STEP 1177 H= 294:15 +CPU= 61.651
 NSTEP =  1178 STEPO   0AAA00AAA
 13:33:17 STEP 1178 H= 294:30 +CPU= 61.705
 NSTEP =  1179 STEPO   0AAA00AAA
 13:34:21 STEP 1179 H= 294:45 +CPU= 60.782
ISTEP=    1180 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1180 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1180 STEPO   0AAA00AAA
 13:35:20 STEP 1180 H= 295:00 +CPU= 56.901
 NSTEP =  1181 STEPO   0AAA00AAA
 13:36:24 STEP 1181 H= 295:15 +CPU= 59.773
 NSTEP =  1182 STEPO   0AAA00AAA
 13:37:28 STEP 1182 H= 295:30 +CPU= 61.286
 NSTEP =  1183 STEPO   0AAA00AAA
 13:38:28 STEP 1183 H= 295:45 +CPU= 58.612
 NSTEP =  1184 STEPO   0AAA00AAA
 13:39:27 STEP 1184 H= 296:00 +CPU= 56.801
ISTEP=    1185 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1185 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1185 STEPO   0AAA00AAA
 13:40:26 STEP 1185 H= 296:15 +CPU= 56.794
 NSTEP =  1186 STEPO   0AAA00AAA
 13:41:26 STEP 1186 H= 296:30 +CPU= 56.727
 NSTEP =  1187 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 13:35:20 0AAA00AAA STEPO     1181    56.901   56.901   59.721    473:28    517:46 0.32509876009647E-04       2GB       0MB
 13:36:24 0AAA00AAA STEPO     1182    59.774   59.774   63.172    474:28    518:49 0.32509876009647E-04       2GB       0MB
 13:37:28 0AAA00AAA STEPO     1183    61.286   61.286   64.139    475:29    519:54 0.32509876009647E-04       2GB       0MB
 13:38:28 0AAA00AAA STEPO     1184    58.612   58.612   60.642    476:28    520:54 0.32509876009647E-04       2GB       0MB
 13:39:27 0AAA00AAA STEPO     1185    56.801   56.801   58.847    477:24    521:53 0.32509876009647E-04       2GB       0MB
 13:40:26 0AAA00AAA STEPO     1186    56.794   56.794   59.156    478:21    522:52 0.32509876009647E-04       2GB       0MB
 13:41:26 0AAA00AAA STEPO     1187    56.727   56.727   59.089    479:18    523:51 0.32509876009647E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1152",
 CTIME   = "00120000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1140, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc/slots/13/progress_file_12293294.xml

<running_values>
  <last_cpu_time>76650.320000</last_cpu_time>
  <upload_file_number>12</upload_file_number>
  <last_iter>1187</last_iter>
  <last_upload>1036800</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 cpdn.org