climateprediction.net home page
Task 22438441

Task 22438441

Name oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0
Workunit 12286817
Created 7 Jun 2024, 18:03:37 UTC
Sent 8 Jun 2024, 4:14:38 UTC
Report deadline 7 Aug 2024, 4:14:38 UTC
Received 8 Jun 2024, 18:02:11 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1459164
Run time 10 hours 45 min 22 sec
CPU time 10 hours 45 min 22 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 4.86 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,534.70 MB
Peak swap size 5,980.79 MB
Peak disk usage 1,345.86 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12219:12219] [20240608:080311:1717851791:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:03:47 STEP 1008 H= 252:00 +CPU= 27.398
  08:04:08 STEP 1009 H= 252:15 +CPU= 21.083
  08:04:30 STEP 1010 H= 252:30 +CPU= 21.556
  08:04:51 STEP 1011 H= 252:45 +CPU= 21.192
  08:05:15 STEP 1012 H= 253:00 +CPU= 24.316
  08:05:41 STEP 1013 H= 253:15 +CPU= 26.160
  08:06:08 STEP 1014 H= 253:30 +CPU= 26.330
  08:06:44 STEP 1015 H= 253:45 +CPU= 32.130
  08:07:14 STEP 1016 H= 254:00 +CPU= 30.510
  08:07:39 STEP 1017 H= 254:15 +CPU= 25.036
  08:08:01 STEP 1018 H= 254:30 +CPU= 21.625
  08:08:22 STEP 1019 H= 254:45 +CPU= 21.383
  08:08:52 STEP 1020 H= 255:00 +CPU= 29.648
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001020
  08:09:16 STEP 1021 H= 255:15 +CPU= 24.506
  08:09:39 STEP 1022 H= 255:30 +CPU= 22.657
  08:10:03 STEP 1023 H= 255:45 +CPU= 23.772
  08:10:26 STEP 1024 H= 256:00 +CPU= 23.211
  08:10:48 STEP 1025 H= 256:15 +CPU= 22.026
  08:11:14 STEP 1026 H= 256:30 +CPU= 25.921
  08:11:37 STEP 1027 H= 256:45 +CPU= 22.987
  08:12:02 STEP 1028 H= 257:00 +CPU= 24.540
  08:12:25 STEP 1029 H= 257:15 +CPU= 23.522
  08:12:47 STEP 1030 H= 257:30 +CPU= 22.232
  08:13:10 STEP 1031 H= 257:45 +CPU= 22.563
  08:13:39 STEP 1032 H= 258:00 +CPU= 28.940
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001032
  08:14:09 STEP 1033 H= 258:15 +CPU= 30.350
  08:14:33 STEP 1034 H= 258:30 +CPU= 23.429
  08:14:58 STEP 1035 H= 258:45 +CPU= 25.459
  08:15:27 STEP 1036 H= 259:00 +CPU= 28.370
  08:15:54 STEP 1037 H= 259:15 +CPU= 27.822
  08:16:17 STEP 1038 H= 259:30 +CPU= 22.688
  08:16:40 STEP 1039 H= 259:45 +CPU= 22.933
  08:17:03 STEP 1040 H= 260:00 +CPU= 22.622
  08:17:26 STEP 1041 H= 260:15 +CPU= 23.793
  08:17:53 STEP 1042 H= 260:30 +CPU= 26.200
  08:18:23 STEP 1043 H= 260:45 +CPU= 30.516
  08:18:58 STEP 1044 H= 261:00 +CPU= 35.091
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001044
  08:19:23 STEP 1045 H= 261:15 +CPU= 24.765
  08:19:46 STEP 1046 H= 261:30 +CPU= 22.651
  08:20:09 STEP 1047 H= 261:45 +CPU= 22.855
  08:20:33 STEP 1048 H= 262:00 +CPU= 24.562
  08:21:00 STEP 1049 H= 262:15 +CPU= 27.076
  08:21:28 STEP 1050 H= 262:30 +CPU= 27.210
  08:21:51 STEP 1051 H= 262:45 +CPU= 22.966
  08:22:15 STEP 1052 H= 263:00 +CPU= 24.797
  08:22:38 STEP 1053 H= 263:15 +CPU= 22.512
  08:23:00 STEP 1054 H= 263:30 +CPU= 22.191
  08:23:25 STEP 1055 H= 263:45 +CPU= 24.427
  08:23:59 STEP 1056 H= 264:00 +CPU= 34.764
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_10.zip
Uploading the intermediate file: upload_file_10.zip
  08:24:26 STEP 1057 H= 264:15 +CPU= 26.660
  08:24:50 STEP 1058 H= 264:30 +CPU= 23.750
  08:25:14 STEP 1059 H= 264:45 +CPU= 24.004
  08:25:39 STEP 1060 H= 265:00 +CPU= 25.103
  08:26:07 STEP 1061 H= 265:15 +CPU= 27.691
  08:26:34 STEP 1062 H= 265:30 +CPU= 27.324
  08:27:03 STEP 1063 H= 265:45 +CPU= 28.502
  08:27:28 STEP 1064 H= 266:00 +CPU= 25.224
  08:27:51 STEP 1065 H= 266:15 +CPU= 23.481
  08:28:17 STEP 1066 H= 266:30 +CPU= 25.608
  08:28:48 STEP 1067 H= 266:45 +CPU= 31.544
  08:29:28 STEP 1068 H= 267:00 +CPU= 39.326
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001068
  08:29:56 STEP 1069 H= 267:15 +CPU= 27.890
  08:30:23 STEP 1070 H= 267:30 +CPU= 27.452
  08:30:55 STEP 1071 H= 267:45 +CPU= 32.114
  08:31:22 STEP 1072 H= 268:00 +CPU= 26.948
  08:31:48 STEP 1073 H= 268:15 +CPU= 25.592
  08:32:11 STEP 1074 H= 268:30 +CPU= 22.628
  08:32:37 STEP 1075 H= 268:45 +CPU= 26.142
  08:32:59 STEP 1076 H= 269:00 +CPU= 22.260
  08:33:21 STEP 1077 H= 269:15 +CPU= 22.178
  08:33:47 STEP 1078 H= 269:30 +CPU= 25.541
  08:34:09 STEP 1079 H= 269:45 +CPU= 22.381
  08:34:38 STEP 1080 H= 270:00 +CPU= 28.590
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001080
  08:35:02 STEP 1081 H= 270:15 +CPU= 24.263
  08:35:31 STEP 1082 H= 270:30 +CPU= 28.795
  08:36:03 STEP 1083 H= 270:45 +CPU= 32.530
  08:36:28 STEP 1084 H= 271:00 +CPU= 24.847
  08:36:51 STEP 1085 H= 271:15 +CPU= 22.979
  08:37:17 STEP 1086 H= 271:30 +CPU= 26.281
  08:37:41 STEP 1087 H= 271:45 +CPU= 23.552
  08:38:04 STEP 1088 H= 272:00 +CPU= 22.957
  08:38:29 STEP 1089 H= 272:15 +CPU= 25.226
  08:38:53 STEP 1090 H= 272:30 +CPU= 24.129
  08:39:19 STEP 1091 H= 272:45 +CPU= 25.197
  08:39:50 STEP 1092 H= 273:00 +CPU= 31.691
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001092
  08:40:16 STEP 1093 H= 273:15 +CPU= 25.815
  08:40:46 STEP 1094 H= 273:30 +CPU= 29.471
  08:41:12 STEP 1095 H= 273:45 +CPU= 26.451
  08:41:35 STEP 1096 H= 274:00 +CPU= 23.238
  08:42:01 STEP 1097 H= 274:15 +CPU= 25.953
  08:42:27 STEP 1098 H= 274:30 +CPU= 25.705
  08:42:52 STEP 1099 H= 274:45 +CPU= 24.748
  08:43:19 STEP 1100 H= 275:00 +CPU= 26.944
  08:43:41 STEP 1101 H= 275:15 +CPU= 22.174
  08:44:06 STEP 1102 H= 275:30 +CPU= 24.783
  08:44:28 STEP 1103 H= 275:45 +CPU= 22.701
  08:45:01 STEP 1104 H= 276:00 +CPU= 32.534
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001104
  08:45:24 STEP 1105 H= 276:15 +CPU= 23.441
  08:45:51 STEP 1106 H= 276:30 +CPU= 26.513
  08:46:15 STEP 1107 H= 276:45 +CPU= 23.590
  08:46:38 STEP 1108 H= 277:00 +CPU= 23.364
  08:47:02 STEP 1109 H= 277:15 +CPU= 23.952
  08:47:27 STEP 1110 H= 277:30 +CPU= 24.774
  08:47:53 STEP 1111 H= 277:45 +CPU= 26.322
  08:48:20 STEP 1112 H= 278:00 +CPU= 26.703
  08:48:44 STEP 1113 H= 278:15 +CPU= 24.463
  08:49:08 STEP 1114 H= 278:30 +CPU= 24.122
  08:49:31 STEP 1115 H= 278:45 +CPU= 22.516
  08:49:59 STEP 1116 H= 279:00 +CPU= 27.875
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001116
  08:50:24 STEP 1117 H= 279:15 +CPU= 25.337
  08:50:47 STEP 1118 H= 279:30 +CPU= 23.147
  08:51:14 STEP 1119 H= 279:45 +CPU= 26.892
  08:51:45 STEP 1120 H= 280:00 +CPU= 31.144
  08:52:10 STEP 1121 H= 280:15 +CPU= 24.921
  08:52:38 STEP 1122 H= 280:30 +CPU= 27.377
  08:53:06 STEP 1123 H= 280:45 +CPU= 27.871
  08:53:31 STEP 1124 H= 281:00 +CPU= 25.755
  08:53:59 STEP 1125 H= 281:15 +CPU= 28.020
  08:54:23 STEP 1126 H= 281:30 +CPU= 23.697
  08:54:48 STEP 1127 H= 281:45 +CPU= 24.800
  08:55:17 STEP 1128 H= 282:00 +CPU= 28.682
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001128
  08:55:42 STEP 1129 H= 282:15 +CPU= 25.003
  08:56:08 STEP 1130 H= 282:30 +CPU= 26.216
  08:56:32 STEP 1131 H= 282:45 +CPU= 23.773
  08:56:56 STEP 1132 H= 283:00 +CPU= 24.865
  08:57:22 STEP 1133 H= 283:15 +CPU= 25.140
  08:57:48 STEP 1134 H= 283:30 +CPU= 26.367
  08:58:15 STEP 1135 H= 283:45 +CPU= 26.978
  08:58:39 STEP 1136 H= 284:00 +CPU= 23.704
  08:59:01 STEP 1137 H= 284:15 +CPU= 22.036
  08:59:26 STEP 1138 H= 284:30 +CPU= 25.057
  08:59:52 STEP 1139 H= 284:45 +CPU= 26.589
  09:00:21 STEP 1140 H= 285:00 +CPU= 28.882
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001140
  09:00:45 STEP 1141 H= 285:15 +CPU= 23.896
  09:01:09 STEP 1142 H= 285:30 +CPU= 24.226
  09:01:34 STEP 1143 H= 285:45 +CPU= 24.882
  09:01:59 STEP 1144 H= 286:00 +CPU= 24.916
  09:02:25 STEP 1145 H= 286:15 +CPU= 25.421
  09:02:52 STEP 1146 H= 286:30 +CPU= 27.681
  09:03:17 STEP 1147 H= 286:45 +CPU= 24.252
  09:03:40 STEP 1148 H= 287:00 +CPU= 22.947
  09:04:03 STEP 1149 H= 287:15 +CPU= 23.366
  09:04:28 STEP 1150 H= 287:30 +CPU= 25.234
  09:04:52 STEP 1151 H= 287:45 +CPU= 24.056
  09:05:28 STEP 1152 H= 288:00 +CPU= 35.375
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_11.zip
Uploading the intermediate file: upload_file_11.zip
  09:05:52 STEP 1153 H= 288:15 +CPU= 24.508
  09:06:17 STEP 1154 H= 288:30 +CPU= 24.790
  09:06:42 STEP 1155 H= 288:45 +CPU= 24.862
  09:07:05 STEP 1156 H= 289:00 +CPU= 23.296
  09:07:30 STEP 1157 H= 289:15 +CPU= 24.469
  09:07:53 STEP 1158 H= 289:30 +CPU= 23.393
  09:08:17 STEP 1159 H= 289:45 +CPU= 23.457
  09:08:41 STEP 1160 H= 290:00 +CPU= 23.964
  09:09:04 STEP 1161 H= 290:15 +CPU= 23.045
  09:09:34 STEP 1162 H= 290:30 +CPU= 30.186
  09:10:03 STEP 1163 H= 290:45 +CPU= 29.048
  09:10:41 STEP 1164 H= 291:00 +CPU= 38.482
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001164
  09:11:12 STEP 1165 H= 291:15 +CPU= 30.985
  09:11:34 STEP 1166 H= 291:30 +CPU= 21.827
  09:12:00 STEP 1167 H= 291:45 +CPU= 26.150
  09:12:29 STEP 1168 H= 292:00 +CPU= 28.771
  09:12:51 STEP 1169 H= 292:15 +CPU= 22.205
  09:13:13 STEP 1170 H= 292:30 +CPU= 21.956
  09:13:38 STEP 1171 H= 292:45 +CPU= 24.384
  09:14:03 STEP 1172 H= 293:00 +CPU= 25.461
  09:14:31 STEP 1173 H= 293:15 +CPU= 27.536
  09:14:55 STEP 1174 H= 293:30 +CPU= 23.777
  09:15:18 STEP 1175 H= 293:45 +CPU= 23.197
  09:15:51 STEP 1176 H= 294:00 +CPU= 33.670
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001176
  09:16:21 STEP 1177 H= 294:15 +CPU= 29.382
  09:16:46 STEP 1178 H= 294:30 +CPU= 24.881
  09:17:10 STEP 1179 H= 294:45 +CPU= 24.650
  09:17:37 STEP 1180 H= 295:00 +CPU= 26.437
  09:18:06 STEP 1181 H= 295:15 +CPU= 29.299
  09:18:32 STEP 1182 H= 295:30 +CPU= 25.755
  09:18:56 STEP 1183 H= 295:45 +CPU= 23.695
  09:19:23 STEP 1184 H= 296:00 +CPU= 27.587
  09:19:47 STEP 1185 H= 296:15 +CPU= 24.154
  09:20:12 STEP 1186 H= 296:30 +CPU= 24.645
  09:20:36 STEP 1187 H= 296:45 +CPU= 24.327
  09:21:09 STEP 1188 H= 297:00 +CPU= 32.516
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001188
  09:21:35 STEP 1189 H= 297:15 +CPU= 26.060
  09:22:02 STEP 1190 H= 297:30 +CPU= 27.018
  09:22:26 STEP 1191 H= 297:45 +CPU= 23.554
  09:22:53 STEP 1192 H= 298:00 +CPU= 27.396
  09:23:16 STEP 1193 H= 298:15 +CPU= 23.424
  09:23:43 STEP 1194 H= 298:30 +CPU= 26.163
  09:24:07 STEP 1195 H= 298:45 +CPU= 24.690
  09:24:39 STEP 1196 H= 299:00 +CPU= 31.987
  09:25:10 STEP 1197 H= 299:15 +CPU= 30.874
  09:25:35 STEP 1198 H= 299:30 +CPU= 25.019
  09:26:06 STEP 1199 H= 299:45 +CPU= 30.982
  09:26:43 STEP 1200 H= 300:00 +CPU= 36.503
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001200
  09:27:07 STEP 1201 H= 300:15 +CPU= 23.971
  09:27:29 STEP 1202 H= 300:30 +CPU= 22.203
  09:27:51 STEP 1203 H= 300:45 +CPU= 22.102
  09:28:17 STEP 1204 H= 301:00 +CPU= 25.423
  09:28:41 STEP 1205 H= 301:15 +CPU= 24.509
  09:29:10 STEP 1206 H= 301:30 +CPU= 28.450
  09:29:35 STEP 1207 H= 301:45 +CPU= 24.840
  09:29:57 STEP 1208 H= 302:00 +CPU= 22.077
  09:30:18 STEP 1209 H= 302:15 +CPU= 21.321
  09:30:39 STEP 1210 H= 302:30 +CPU= 21.287
  09:31:02 STEP 1211 H= 302:45 +CPU= 22.553
  09:31:35 STEP 1212 H= 303:00 +CPU= 32.499
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001212
  09:31:58 STEP 1213 H= 303:15 +CPU= 23.228
  09:32:19 STEP 1214 H= 303:30 +CPU= 21.158
  09:32:41 STEP 1215 H= 303:45 +CPU= 21.097
  09:33:02 STEP 1216 H= 304:00 +CPU= 21.602
  09:33:28 STEP 1217 H= 304:15 +CPU= 25.785
  09:33:53 STEP 1218 H= 304:30 +CPU= 25.008
  09:34:18 STEP 1219 H= 304:45 +CPU= 24.649
  09:34:42 STEP 1220 H= 305:00 +CPU= 24.675
  09:35:04 STEP 1221 H= 305:15 +CPU= 21.864
  09:35:28 STEP 1222 H= 305:30 +CPU= 23.842
  09:35:56 STEP 1223 H= 305:45 +CPU= 28.221
  09:36:25 STEP 1224 H= 306:00 +CPU= 28.633
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001224
  09:36:51 STEP 1225 H= 306:15 +CPU= 25.443
  09:37:14 STEP 1226 H= 306:30 +CPU= 23.904
  09:37:40 STEP 1227 H= 306:45 +CPU= 25.133
  09:38:10 STEP 1228 H= 307:00 +CPU= 30.209
  09:38:40 STEP 1229 H= 307:15 +CPU= 30.267
  09:39:12 STEP 1230 H= 307:30 +CPU= 31.943
  09:39:42 STEP 1231 H= 307:45 +CPU= 29.477
  09:40:05 STEP 1232 H= 308:00 +CPU= 23.805
  09:40:28 STEP 1233 H= 308:15 +CPU= 23.127
  09:40:54 STEP 1234 H= 308:30 +CPU= 26.016
  09:41:18 STEP 1235 H= 308:45 +CPU= 23.099
  09:41:53 STEP 1236 H= 309:00 +CPU= 35.491
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001236
  09:42:21 STEP 1237 H= 309:15 +CPU= 27.612
  09:42:43 STEP 1238 H= 309:30 +CPU= 21.826
  09:43:09 STEP 1239 H= 309:45 +CPU= 26.611
  09:43:35 STEP 1240 H= 310:00 +CPU= 25.522
  09:43:57 STEP 1241 H= 310:15 +CPU= 22.356
  09:44:23 STEP 1242 H= 310:30 +CPU= 25.825
  09:44:45 STEP 1243 H= 310:45 +CPU= 21.599
  09:45:07 STEP 1244 H= 311:00 +CPU= 22.730
  09:45:29 STEP 1245 H= 311:15 +CPU= 22.110
  09:45:52 STEP 1246 H= 311:30 +CPU= 22.825
  09:46:17 STEP 1247 H= 311:45 +CPU= 24.698
  09:46:49 STEP 1248 H= 312:00 +CPU= 31.634
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:47:15 STEP 1249 H= 312:15 +CPU= 26.625
  09:47:43 STEP 1250 H= 312:30 +CPU= 27.974
  09:48:13 STEP 1251 H= 312:45 +CPU= 29.477
  09:48:42 STEP 1252 H= 313:00 +CPU= 29.462
  09:49:08 STEP 1253 H= 313:15 +CPU= 25.855
  09:49:37 STEP 1254 H= 313:30 +CPU= 28.351
  09:50:01 STEP 1255 H= 313:45 +CPU= 24.462
  09:50:28 STEP 1256 H= 314:00 +CPU= 26.664
  09:50:56 STEP 1257 H= 314:15 +CPU= 28.058
  09:51:23 STEP 1258 H= 314:30 +CPU= 26.611
  09:51:45 STEP 1259 H= 314:45 +CPU= 22.315
  09:52:12 STEP 1260 H= 315:00 +CPU= 27.277
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001260
  09:52:35 STEP 1261 H= 315:15 +CPU= 22.646
  09:52:57 STEP 1262 H= 315:30 +CPU= 22.070
  09:53:21 STEP 1263 H= 315:45 +CPU= 23.977
  09:53:47 STEP 1264 H= 316:00 +CPU= 25.434
  09:54:14 STEP 1265 H= 316:15 +CPU= 27.304
  09:54:41 STEP 1266 H= 316:30 +CPU= 26.907
  09:55:08 STEP 1267 H= 316:45 +CPU= 26.740
  09:55:35 STEP 1268 H= 317:00 +CPU= 27.350
  09:56:08 STEP 1269 H= 317:15 +CPU= 32.563
  09:56:34 STEP 1270 H= 317:30 +CPU= 26.304
  09:57:03 STEP 1271 H= 317:45 +CPU= 29.426
  09:57:41 STEP 1272 H= 318:00 +CPU= 37.731
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001272
  09:58:04 STEP 1273 H= 318:15 +CPU= 22.954
  09:58:27 STEP 1274 H= 318:30 +CPU= 22.952
  09:58:50 STEP 1275 H= 318:45 +CPU= 22.821
  09:59:14 STEP 1276 H= 319:00 +CPU= 23.768
  09:59:36 STEP 1277 H= 319:15 +CPU= 22.033
  09:59:57 STEP 1278 H= 319:30 +CPU= 21.384
  10:00:19 STEP 1279 H= 319:45 +CPU= 22.226
  10:00:49 STEP 1280 H= 320:00 +CPU= 29.356
  10:01:11 STEP 1281 H= 320:15 +CPU= 22.266
  10:01:35 STEP 1282 H= 320:30 +CPU= 23.836
  10:02:01 STEP 1283 H= 320:45 +CPU= 25.896
  10:02:31 STEP 1284 H= 321:00 +CPU= 29.946
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001284
  10:02:56 STEP 1285 H= 321:15 +CPU= 25.352
  10:03:24 STEP 1286 H= 321:30 +CPU= 28.211
  10:03:47 STEP 1287 H= 321:45 +CPU= 22.269
  10:04:16 STEP 1288 H= 322:00 +CPU= 28.955
  10:04:39 STEP 1289 H= 322:15 +CPU= 23.001
  10:05:03 STEP 1290 H= 322:30 +CPU= 24.091
  10:05:28 STEP 1291 H= 322:45 +CPU= 25.069
  10:05:51 STEP 1292 H= 323:00 +CPU= 23.401
  10:06:16 STEP 1293 H= 323:15 +CPU= 24.742
  10:06:41 STEP 1294 H= 323:30 +CPU= 25.375
  10:07:08 STEP 1295 H= 323:45 +CPU= 27.096
  10:07:45 STEP 1296 H= 324:00 +CPU= 35.918
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001296
  10:08:15 STEP 1297 H= 324:15 +CPU= 30.722
  10:08:39 STEP 1298 H= 324:30 +CPU= 23.116
  10:09:00 STEP 1299 H= 324:45 +CPU= 21.344
  10:09:29 STEP 1300 H= 325:00 +CPU= 28.627
  10:09:51 STEP 1301 H= 325:15 +CPU= 22.448
  10:10:15 STEP 1302 H= 325:30 +CPU= 23.490
  10:10:40 STEP 1303 H= 325:45 +CPU= 24.927
  10:11:06 STEP 1304 H= 326:00 +CPU= 26.820
  10:11:28 STEP 1305 H= 326:15 +CPU= 21.584
  10:11:54 STEP 1306 H= 326:30 +CPU= 25.481
  10:12:19 STEP 1307 H= 326:45 +CPU= 25.183
  10:12:48 STEP 1308 H= 327:00 +CPU= 28.888
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001308
  10:13:12 STEP 1309 H= 327:15 +CPU= 23.932
  10:13:36 STEP 1310 H= 327:30 +CPU= 24.762
  10:14:00 STEP 1311 H= 327:45 +CPU= 23.380
  10:14:25 STEP 1312 H= 328:00 +CPU= 25.256
  10:14:48 STEP 1313 H= 328:15 +CPU= 23.025
  10:15:15 STEP 1314 H= 328:30 +CPU= 26.594
  10:15:38 STEP 1315 H= 328:45 +CPU= 23.031
  10:16:01 STEP 1316 H= 329:00 +CPU= 23.804
  10:16:26 STEP 1317 H= 329:15 +CPU= 24.105
  10:16:49 STEP 1318 H= 329:30 +CPU= 23.811
  10:17:11 STEP 1319 H= 329:45 +CPU= 21.629
  10:17:50 STEP 1320 H= 330:00 +CPU= 38.853
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001320
  10:18:17 STEP 1321 H= 330:15 +CPU= 26.771
  10:18:40 STEP 1322 H= 330:30 +CPU= 22.920
  10:19:04 STEP 1323 H= 330:45 +CPU= 24.338
  10:19:32 STEP 1324 H= 331:00 +CPU= 28.159
  10:19:58 STEP 1325 H= 331:15 +CPU= 25.626
  10:20:21 STEP 1326 H= 331:30 +CPU= 23.406
  10:20:44 STEP 1327 H= 331:45 +CPU= 22.396
  10:21:08 STEP 1328 H= 332:00 +CPU= 23.946
  10:21:30 STEP 1329 H= 332:15 +CPU= 22.476
  10:21:55 STEP 1330 H= 332:30 +CPU= 25.275
  10:22:21 STEP 1331 H= 332:45 +CPU= 25.183
  10:22:51 STEP 1332 H= 333:00 +CPU= 30.700
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001332
  10:23:15 STEP 1333 H= 333:15 +CPU= 23.415
  10:23:44 STEP 1334 H= 333:30 +CPU= 29.773
  10:24:11 STEP 1335 H= 333:45 +CPU= 26.216
  10:24:36 STEP 1336 H= 334:00 +CPU= 25.397
  10:25:04 STEP 1337 H= 334:15 +CPU= 27.490
  10:25:32 STEP 1338 H= 334:30 +CPU= 28.156
  10:25:55 STEP 1339 H= 334:45 +CPU= 22.703
  10:26:18 STEP 1340 H= 335:00 +CPU= 23.000
  10:26:39 STEP 1341 H= 335:15 +CPU= 21.947
  10:27:06 STEP 1342 H= 335:30 +CPU= 26.721
  10:27:30 STEP 1343 H= 335:45 +CPU= 24.218
  10:28:05 STEP 1344 H= 336:00 +CPU= 34.491
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_13.zip
Uploading the intermediate file: upload_file_13.zip
  10:28:30 STEP 1345 H= 336:15 +CPU= 25.386
  10:28:53 STEP 1346 H= 336:30 +CPU= 22.346
  10:29:18 STEP 1347 H= 336:45 +CPU= 25.624
  10:29:43 STEP 1348 H= 337:00 +CPU= 24.062
  10:30:09 STEP 1349 H= 337:15 +CPU= 26.385
  10:30:40 STEP 1350 H= 337:30 +CPU= 30.769
  10:31:11 STEP 1351 H= 337:45 +CPU= 31.500
  10:31:43 STEP 1352 H= 338:00 +CPU= 32.116
  10:32:06 STEP 1353 H= 338:15 +CPU= 22.337
  10:32:31 STEP 1354 H= 338:30 +CPU= 25.737
  10:33:02 STEP 1355 H= 338:45 +CPU= 31.000
  10:33:33 STEP 1356 H= 339:00 +CPU= 30.344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001356
  10:34:01 STEP 1357 H= 339:15 +CPU= 28.121
  10:34:28 STEP 1358 H= 339:30 +CPU= 26.697
  10:34:54 STEP 1359 H= 339:45 +CPU= 25.921
  10:35:20 STEP 1360 H= 340:00 +CPU= 26.433
  10:35:42 STEP 1361 H= 340:15 +CPU= 21.985
  10:36:05 STEP 1362 H= 340:30 +CPU= 22.526
  10:36:28 STEP 1363 H= 340:45 +CPU= 23.819
  10:36:51 STEP 1364 H= 341:00 +CPU= 22.972
  10:37:19 STEP 1365 H= 341:15 +CPU= 27.735
  10:37:45 STEP 1366 H= 341:30 +CPU= 25.765
  10:38:07 STEP 1367 H= 341:45 +CPU= 22.451
  10:38:35 STEP 1368 H= 342:00 +CPU= 27.598
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001368
  10:38:57 STEP 1369 H= 342:15 +CPU= 21.924
  10:39:23 STEP 1370 H= 342:30 +CPU= 26.023
  10:39:45 STEP 1371 H= 342:45 +CPU= 21.923
  10:40:12 STEP 1372 H= 343:00 +CPU= 27.035
  10:40:36 STEP 1373 H= 343:15 +CPU= 24.082
  10:41:06 STEP 1374 H= 343:30 +CPU= 29.893
  10:41:35 STEP 1375 H= 343:45 +CPU= 28.890
  10:41:59 STEP 1376 H= 344:00 +CPU= 23.697
  10:42:21 STEP 1377 H= 344:15 +CPU= 22.310
  10:42:44 STEP 1378 H= 344:30 +CPU= 22.821
  10:43:06 STEP 1379 H= 344:45 +CPU= 22.606
  10:43:36 STEP 1380 H= 345:00 +CPU= 29.481
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001380
  10:44:01 STEP 1381 H= 345:15 +CPU= 24.922
  10:44:23 STEP 1382 H= 345:30 +CPU= 22.260
  10:44:50 STEP 1383 H= 345:45 +CPU= 26.773
  10:45:16 STEP 1384 H= 346:00 +CPU= 25.861
  10:45:41 STEP 1385 H= 346:15 +CPU= 25.785
  10:46:07 STEP 1386 H= 346:30 +CPU= 25.086
  10:46:32 STEP 1387 H= 346:45 +CPU= 25.371
  10:47:01 STEP 1388 H= 347:00 +CPU= 28.945
  10:47:28 STEP 1389 H= 347:15 +CPU= 26.787
  10:47:52 STEP 1390 H= 347:30 +CPU= 24.461
  10:48:14 STEP 1391 H= 347:45 +CPU= 21.815
  10:48:47 STEP 1392 H= 348:00 +CPU= 32.483
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001392
  10:49:12 STEP 1393 H= 348:15 +CPU= 25.075
  10:49:34 STEP 1394 H= 348:30 +CPU= 22.510
  10:49:56 STEP 1395 H= 348:45 +CPU= 22.051
  10:50:19 STEP 1396 H= 349:00 +CPU= 22.581
  10:50:46 STEP 1397 H= 349:15 +CPU= 27.005
  10:51:11 STEP 1398 H= 349:30 +CPU= 24.881
  10:51:40 STEP 1399 H= 349:45 +CPU= 29.503
  10:52:03 STEP 1400 H= 350:00 +CPU= 22.489
  10:52:28 STEP 1401 H= 350:15 +CPU= 24.603
  10:52:51 STEP 1402 H= 350:30 +CPU= 23.611
  10:53:15 STEP 1403 H= 350:45 +CPU= 24.020
  10:53:45 STEP 1404 H= 351:00 +CPU= 29.649
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001404
  10:54:12 STEP 1405 H= 351:15 +CPU= 26.642
  10:54:37 STEP 1406 H= 351:30 +CPU= 25.515
  10:55:03 STEP 1407 H= 351:45 +CPU= 25.999
  10:55:29 STEP 1408 H= 352:00 +CPU= 25.772
  10:55:51 STEP 1409 H= 352:15 +CPU= 22.251
  10:56:14 STEP 1410 H= 352:30 +CPU= 22.428
  10:56:36 STEP 1411 H= 352:45 +CPU= 22.079
  10:57:03 STEP 1412 H= 353:00 +CPU= 27.471
  10:57:31 STEP 1413 H= 353:15 +CPU= 27.810
  10:57:52 STEP 1414 H= 353:30 +CPU= 21.420
  10:58:13 STEP 1415 H= 353:45 +CPU= 21.008
  10:58:40 STEP 1416 H= 354:00 +CPU= 26.833
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001416
  10:59:01 STEP 1417 H= 354:15 +CPU= 21.225
  10:59:23 STEP 1418 H= 354:30 +CPU= 22.043
  10:59:45 STEP 1419 H= 354:45 +CPU= 22.002
  11:00:10 STEP 1420 H= 355:00 +CPU= 24.624
  11:00:32 STEP 1421 H= 355:15 +CPU= 22.264
  11:00:54 STEP 1422 H= 355:30 +CPU= 21.851
  11:01:21 STEP 1423 H= 355:45 +CPU= 26.653
  11:01:48 STEP 1424 H= 356:00 +CPU= 27.043
  11:02:16 STEP 1425 H= 356:15 +CPU= 27.653
  11:02:44 STEP 1426 H= 356:30 +CPU= 28.775
  11:03:09 STEP 1427 H= 356:45 +CPU= 24.326
  11:03:39 STEP 1428 H= 357:00 +CPU= 29.967
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001428
  11:04:06 STEP 1429 H= 357:15 +CPU= 26.995
  11:04:30 STEP 1430 H= 357:30 +CPU= 23.849
  11:04:59 STEP 1431 H= 357:45 +CPU= 29.231
  11:05:27 STEP 1432 H= 358:00 +CPU= 27.482
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3um
(argv4) batchid: 1017
(argv5) wuid: 12286817
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9ee8ca81b8b6fb3206302280c1661c62 to: /var/lib/boinc-client/slots/5/oifs_43r3_bl_a3um_2016092300_20_1017_12286817.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_bl_a3um_2016092300_20_1017_12286817.zip
ic_ancil_file: ic_ancil_12286817
ifsdata_file: ifsdata_12286817
climate_data_file: clim_data_12286817
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/5/ic_ancil_12286817.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12286817.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cd9ea294ee221a254c2fc18fd9406fda to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12286817.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12286817.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12286817.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12286817.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12286817.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12286817.xml
-- Model is restarting --
Adjusting last_iter, 1433, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12286817.xml
last_cpu_time: 37456
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025
The child process has been launched with process id: 12703
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:12703:12703] [20240608:115221:1717865541:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:53:10 STEP 1392 H= 348:00 +CPU= 29.873
  11:53:36 STEP 1393 H= 348:15 +CPU= 26.185
  11:53:59 STEP 1394 H= 348:30 +CPU= 22.286
  11:54:21 STEP 1395 H= 348:45 +CPU= 22.483
  11:54:45 STEP 1396 H= 349:00 +CPU= 23.646
  11:55:07 STEP 1397 H= 349:15 +CPU= 22.363
  11:55:29 STEP 1398 H= 349:30 +CPU= 22.370
  11:55:52 STEP 1399 H= 349:45 +CPU= 22.296
  11:56:15 STEP 1400 H= 350:00 +CPU= 22.831
  11:56:37 STEP 1401 H= 350:15 +CPU= 22.441
  11:56:59 STEP 1402 H= 350:30 +CPU= 22.439
  11:57:26 STEP 1403 H= 350:45 +CPU= 26.421
  11:58:04 STEP 1404 H= 351:00 +CPU= 37.989
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001404
  11:58:31 STEP 1405 H= 351:15 +CPU= 26.854
  11:58:53 STEP 1406 H= 351:30 +CPU= 22.616
  11:59:17 STEP 1407 H= 351:45 +CPU= 23.825
  11:59:45 STEP 1408 H= 352:00 +CPU= 28.110
  12:00:12 STEP 1409 H= 352:15 +CPU= 26.248
  12:00:37 STEP 1410 H= 352:30 +CPU= 24.939
  12:01:04 STEP 1411 H= 352:45 +CPU= 27.280
  12:01:28 STEP 1412 H= 353:00 +CPU= 24.568
  12:01:55 STEP 1413 H= 353:15 +CPU= 26.989
  12:02:22 STEP 1414 H= 353:30 +CPU= 26.340
  12:02:51 STEP 1415 H= 353:45 +CPU= 29.632
  12:03:22 STEP 1416 H= 354:00 +CPU= 30.926
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001416
  12:03:49 STEP 1417 H= 354:15 +CPU= 26.633
  12:04:15 STEP 1418 H= 354:30 +CPU= 25.643
  12:04:40 STEP 1419 H= 354:45 +CPU= 25.752
  12:05:05 STEP 1420 H= 355:00 +CPU= 24.366
  12:05:29 STEP 1421 H= 355:15 +CPU= 23.822
  12:05:53 STEP 1422 H= 355:30 +CPU= 24.503
  12:06:21 STEP 1423 H= 355:45 +CPU= 27.313
  12:06:47 STEP 1424 H= 356:00 +CPU= 26.513
  12:07:11 STEP 1425 H= 356:15 +CPU= 23.487
  12:07:37 STEP 1426 H= 356:30 +CPU= 26.203
  12:08:09 STEP 1427 H= 356:45 +CPU= 32.269
  12:08:39 STEP 1428 H= 357:00 +CPU= 29.634
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001428
  12:09:03 STEP 1429 H= 357:15 +CPU= 24.326
  12:09:26 STEP 1430 H= 357:30 +CPU= 23.142
  12:09:49 STEP 1431 H= 357:45 +CPU= 22.527
  12:10:12 STEP 1432 H= 358:00 +CPU= 23.400
  12:10:38 STEP 1433 H= 358:15 +CPU= 26.116
  12:11:01 STEP 1434 H= 358:30 +CPU= 22.158
  12:11:26 STEP 1435 H= 358:45 +CPU= 25.630
  12:11:49 STEP 1436 H= 359:00 +CPU= 23.260
  12:12:15 STEP 1437 H= 359:15 +CPU= 25.859
  12:12:45 STEP 1438 H= 359:30 +CPU= 29.739
  12:13:10 STEP 1439 H= 359:45 +CPU= 24.894
  12:13:44 STEP 1440 H= 360:00 +CPU= 33.464
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.7070433732E-21      I       0.1144590051E-08      I
ICloud fractionI      0.0000000000E+00      I       0.3402355783E-04      I
I  Relat. Hum. I      0.1016602628E+00      I       0.1283182835E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.7155835498E+00      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    295854244 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS986180.8 Mbytes/s, TOTAL TIME=    0.0080( 3.7%)
 12:13:44 STEP 1440 H= 360:00 +CPU= 33.464
 END CNT3
 NSTEP =  1440 CNT0   000000000  23.058  23.058  23.074   0.999               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************     9.5     7.0     2.5     7.6     4.9
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1282222.8 1282222.8    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1273390.9 1273390.9     99.31      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    3458.7    3458.7     14.30      0.00
   9 SPCM     - SPECTRAL COMP.                      48     827.3     827.3      3.10      0.00
  10 SCAN2M - PHYSICS                               49   13632.1   13632.1     52.10      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4     774.8     774.8      0.24      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      55.5      55.5      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      31.8      31.8      0.01      0.00
  27 RESTART FILES - WRITING                         1    3098.6    3098.6      0.24      0.00
  28 RESTART FILES - READING                         1    8462.6    8462.6      0.66      0.00
  29 SU4FPOS IN CNT4                                 4       0.2       0.2      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    6462.5    6462.5      2.02      0.00
  31 POSDDH IN STEPO                                 1       3.2       3.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    4800.3    4800.3     18.34      0.00
  39 SU0YOMB                                         1     219.5     219.5      0.02      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      16.5      16.5      0.06      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     326.3     326.3      1.22      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      38.5      38.5      0.14      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.5       0.5      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     380.8     380.8      3.21      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     246.9     246.9      1.93      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      80.1      80.1      0.62      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     142.2     142.2      1.20      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      64.4      64.4      0.01      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      23.0      23.0      0.19      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      15.9      15.9      0.12      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108      79.9      79.9      0.67      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      23.5      23.5      0.18      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1282.2 CPU TIME    1266.4 VECTOR TIME     1266.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1017.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/5/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/5/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286817/ICMGGh7zg+001344
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
12:17:14 (12699): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0_r2011632025_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 18:03:01 1459164 22438441 oifs_43r3_bl_a3um_2016092300_20_1017_12286817_0 1,295,100 38,722 0.0299


©2024 climateprediction.net