climateprediction.net home page
Task 22435422

Task 22435422

Name oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0
Workunit 12283843
Created 7 Jun 2024, 13:45:47 UTC
Sent 7 Jun 2024, 13:52:53 UTC
Report deadline 6 Aug 2024, 13:52:53 UTC
Received 8 Jun 2024, 13:40:03 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1532094
Run time 13 hours 26 min 54 sec
CPU time 13 hours 26 min 54 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 4.58 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,002.04 MB
Peak swap size 5,980.80 MB
Peak disk usage 2,288.30 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
T1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:12928:12928] [20240608:110143:1717837303:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:03:43 STEP 1104 H= 276:00 +CPU= 31.016
  11:04:17 STEP 1105 H= 276:15 +CPU= 29.833
  11:04:46 STEP 1106 H= 276:30 +CPU= 29.303
  11:05:19 STEP 1107 H= 276:45 +CPU= 29.137
  11:05:49 STEP 1108 H= 277:00 +CPU= 29.750
  11:06:18 STEP 1109 H= 277:15 +CPU= 28.986
  11:06:46 STEP 1110 H= 277:30 +CPU= 28.820
  11:07:16 STEP 1111 H= 277:45 +CPU= 29.226
  11:07:45 STEP 1112 H= 278:00 +CPU= 29.603
  11:08:15 STEP 1113 H= 278:15 +CPU= 29.319
  11:08:44 STEP 1114 H= 278:30 +CPU= 29.445
  11:09:13 STEP 1115 H= 278:45 +CPU= 29.395
  11:09:53 STEP 1116 H= 279:00 +CPU= 36.666
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001116
  11:10:26 STEP 1117 H= 279:15 +CPU= 29.904
  11:10:58 STEP 1118 H= 279:30 +CPU= 29.656
  11:11:28 STEP 1119 H= 279:45 +CPU= 29.718
  11:11:59 STEP 1120 H= 280:00 +CPU= 30.110
  11:12:28 STEP 1121 H= 280:15 +CPU= 29.493
  11:12:58 STEP 1122 H= 280:30 +CPU= 30.032
  11:13:28 STEP 1123 H= 280:45 +CPU= 29.917
  11:13:59 STEP 1124 H= 281:00 +CPU= 30.790
  11:14:29 STEP 1125 H= 281:15 +CPU= 29.908
  11:14:59 STEP 1126 H= 281:30 +CPU= 30.095
  11:15:29 STEP 1127 H= 281:45 +CPU= 30.036
  11:16:10 STEP 1128 H= 282:00 +CPU= 37.506
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001128
  11:16:43 STEP 1129 H= 282:15 +CPU= 29.892
  11:17:12 STEP 1130 H= 282:30 +CPU= 28.939
  11:17:41 STEP 1131 H= 282:45 +CPU= 28.970
  11:18:11 STEP 1132 H= 283:00 +CPU= 29.514
  11:18:40 STEP 1133 H= 283:15 +CPU= 28.926
  11:19:09 STEP 1134 H= 283:30 +CPU= 28.832
  11:19:37 STEP 1135 H= 283:45 +CPU= 28.925
  11:20:07 STEP 1136 H= 284:00 +CPU= 29.595
  11:20:36 STEP 1137 H= 284:15 +CPU= 28.801
  11:21:05 STEP 1138 H= 284:30 +CPU= 28.812
  11:21:33 STEP 1139 H= 284:45 +CPU= 28.755
  11:22:09 STEP 1140 H= 285:00 +CPU= 35.669
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001140
  11:22:38 STEP 1141 H= 285:15 +CPU= 28.625
  11:23:06 STEP 1142 H= 285:30 +CPU= 28.247
  11:23:35 STEP 1143 H= 285:45 +CPU= 28.533
  11:24:04 STEP 1144 H= 286:00 +CPU= 29.191
  11:24:32 STEP 1145 H= 286:15 +CPU= 28.677
  11:25:01 STEP 1146 H= 286:30 +CPU= 28.845
  11:25:30 STEP 1147 H= 286:45 +CPU= 28.835
  11:26:00 STEP 1148 H= 287:00 +CPU= 29.731
  11:26:29 STEP 1149 H= 287:15 +CPU= 29.055
  11:26:58 STEP 1150 H= 287:30 +CPU= 28.752
  11:27:26 STEP 1151 H= 287:45 +CPU= 28.525
  11:28:10 STEP 1152 H= 288:00 +CPU= 39.125
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_11.zip
  11:28:32 STEP 1153 H= 288:15 +CPU= 21.217
  11:28:52 STEP 1154 H= 288:30 +CPU= 20.671
  11:29:22 STEP 1155 H= 288:45 +CPU= 29.011
Uploading the intermediate file: upload_file_11.zip
  11:29:56 STEP 1156 H= 289:00 +CPU= 30.730
  11:30:28 STEP 1157 H= 289:15 +CPU= 29.622
  11:31:01 STEP 1158 H= 289:30 +CPU= 30.075
  11:31:34 STEP 1159 H= 289:45 +CPU= 29.430
  11:32:07 STEP 1160 H= 290:00 +CPU= 30.575
  11:32:37 STEP 1161 H= 290:15 +CPU= 29.660
  11:33:07 STEP 1162 H= 290:30 +CPU= 29.402
  11:33:39 STEP 1163 H= 290:45 +CPU= 29.863
  11:34:19 STEP 1164 H= 291:00 +CPU= 37.351
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001164
  11:34:48 STEP 1165 H= 291:15 +CPU= 29.045
  11:35:17 STEP 1166 H= 291:30 +CPU= 29.266
  11:35:47 STEP 1167 H= 291:45 +CPU= 29.678
  11:36:17 STEP 1168 H= 292:00 +CPU= 30.449
  11:36:47 STEP 1169 H= 292:15 +CPU= 30.108
  11:37:19 STEP 1170 H= 292:30 +CPU= 29.537
  11:37:52 STEP 1171 H= 292:45 +CPU= 30.322
  11:38:23 STEP 1172 H= 293:00 +CPU= 29.888
  11:38:53 STEP 1173 H= 293:15 +CPU= 29.663
  11:39:25 STEP 1174 H= 293:30 +CPU= 30.234
  11:39:57 STEP 1175 H= 293:45 +CPU= 29.879
  11:40:38 STEP 1176 H= 294:00 +CPU= 37.670
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001176
  11:41:09 STEP 1177 H= 294:15 +CPU= 29.741
  11:41:39 STEP 1178 H= 294:30 +CPU= 29.477
  11:42:09 STEP 1179 H= 294:45 +CPU= 29.890
  11:42:41 STEP 1180 H= 295:00 +CPU= 30.412
  11:43:13 STEP 1181 H= 295:15 +CPU= 29.826
  11:43:45 STEP 1182 H= 295:30 +CPU= 30.125
  11:44:15 STEP 1183 H= 295:45 +CPU= 29.896
  11:44:46 STEP 1184 H= 296:00 +CPU= 30.559
  11:45:16 STEP 1185 H= 296:15 +CPU= 29.805
  11:45:46 STEP 1186 H= 296:30 +CPU= 29.826
  11:46:16 STEP 1187 H= 296:45 +CPU= 29.812
  11:46:54 STEP 1188 H= 297:00 +CPU= 37.228
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001188
  11:47:25 STEP 1189 H= 297:15 +CPU= 30.967
  11:47:56 STEP 1190 H= 297:30 +CPU= 31.269
  11:48:28 STEP 1191 H= 297:45 +CPU= 31.825
  11:49:00 STEP 1192 H= 298:00 +CPU= 32.334
  11:49:32 STEP 1193 H= 298:15 +CPU= 31.348
  11:50:03 STEP 1194 H= 298:30 +CPU= 31.388
  11:50:34 STEP 1195 H= 298:45 +CPU= 31.102
  11:51:06 STEP 1196 H= 299:00 +CPU= 31.225
  11:51:37 STEP 1197 H= 299:15 +CPU= 30.866
  11:52:08 STEP 1198 H= 299:30 +CPU= 31.461
  11:52:40 STEP 1199 H= 299:45 +CPU= 31.511
  11:54:01 STEP 1200 H= 300:00 +CPU= 42.036
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001200
  11:54:32 STEP 1201 H= 300:15 +CPU= 30.861
  11:55:02 STEP 1202 H= 300:30 +CPU= 30.232
  11:55:32 STEP 1203 H= 300:45 +CPU= 29.727
  11:56:02 STEP 1204 H= 301:00 +CPU= 30.102
  11:56:29 STEP 1205 H= 301:15 +CPU= 26.994
  11:56:53 STEP 1206 H= 301:30 +CPU= 24.131
  11:57:23 STEP 1207 H= 301:45 +CPU= 29.032
  11:57:54 STEP 1208 H= 302:00 +CPU= 30.253
  11:58:26 STEP 1209 H= 302:15 +CPU= 29.061
  11:58:59 STEP 1210 H= 302:30 +CPU= 29.215
  11:59:31 STEP 1211 H= 302:45 +CPU= 30.163
  12:00:10 STEP 1212 H= 303:00 +CPU= 38.879
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001212
  12:00:42 STEP 1213 H= 303:15 +CPU= 32.020
  12:01:14 STEP 1214 H= 303:30 +CPU= 31.985
  12:01:47 STEP 1215 H= 303:45 +CPU= 32.374
  12:02:18 STEP 1216 H= 304:00 +CPU= 31.782
  12:02:42 STEP 1217 H= 304:15 +CPU= 23.343
  12:03:04 STEP 1218 H= 304:30 +CPU= 22.611
  12:03:34 STEP 1219 H= 304:45 +CPU= 29.835
  12:04:04 STEP 1220 H= 305:00 +CPU= 30.020
  12:04:33 STEP 1221 H= 305:15 +CPU= 28.945
  12:05:03 STEP 1222 H= 305:30 +CPU= 29.397
  12:05:32 STEP 1223 H= 305:45 +CPU= 29.422
  12:06:09 STEP 1224 H= 306:00 +CPU= 36.419
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001224
  12:06:38 STEP 1225 H= 306:15 +CPU= 28.992
  12:07:08 STEP 1226 H= 306:30 +CPU= 29.274
  12:07:40 STEP 1227 H= 306:45 +CPU= 29.208
  12:08:11 STEP 1228 H= 307:00 +CPU= 29.902
  12:08:42 STEP 1229 H= 307:15 +CPU= 30.591
  12:09:13 STEP 1230 H= 307:30 +CPU= 30.868
  12:09:43 STEP 1231 H= 307:45 +CPU= 30.559
  12:10:19 STEP 1232 H= 308:00 +CPU= 31.821
  12:10:50 STEP 1233 H= 308:15 +CPU= 30.518
  12:11:20 STEP 1234 H= 308:30 +CPU= 30.915
  12:11:49 STEP 1235 H= 308:45 +CPU= 28.775
  12:12:30 STEP 1236 H= 309:00 +CPU= 36.091
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001236
  12:13:02 STEP 1237 H= 309:15 +CPU= 29.530
  12:13:34 STEP 1238 H= 309:30 +CPU= 29.419
  12:14:06 STEP 1239 H= 309:45 +CPU= 31.677
  12:14:38 STEP 1240 H= 310:00 +CPU= 32.237
  12:15:10 STEP 1241 H= 310:15 +CPU= 31.788
  12:15:42 STEP 1242 H= 310:30 +CPU= 31.566
  12:16:14 STEP 1243 H= 310:45 +CPU= 29.833
  12:16:46 STEP 1244 H= 311:00 +CPU= 29.522
  12:17:16 STEP 1245 H= 311:15 +CPU= 29.084
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.389288414375509E-06 0.000000000000000E+00 0.723032704805800E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.846511750243001E-06 0.000000000000000E+00 0.876462524060273E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.980415036475839E-07 0.000000000000000E+00 0.487782158444042E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.227576270313801E-08 0.000000000000000E+00 0.631587617239035E-04
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.564813173625988E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1236 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1236 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001236 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001236
 MPI-TASK:   1 -    317451876 BYTES IN      6 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.1034( 0.3%)
 NSTEP =  1236 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001236 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001236
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0069 SECONDS102521.6 Mbytes/s, TOTAL TIME=    0.3565( 1.9%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001236 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001236
 MPI-TASK:   1 -   5079230016 BYTES IN    125 RECORDS TRANSFERRED IN    0.0047 SECONDS******** Mbytes/s, TOTAL TIME=    0.2623( 1.8%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001236 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001236
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0001 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0044( 2.3%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001236 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001236
 MPI-TASK:   1 -    308872147 BYTES IN      4 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0064( 3.1%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1236 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1236 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001236 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001236
 MPI-TASK:   1 -    317451876 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS793629.7 Mbytes/s, TOTAL TIME=    0.0119( 3.4%)
 12:12:30 STEP 1236 H= 309:00 +CPU= 36.091
 NSTEP =  1237 STEPO   0AAA00AAA
 12:13:02 STEP 1237 H= 309:15 +CPU= 29.530
 NSTEP =  1238 STEPO   0AAA00AAA
 12:13:34 STEP 1238 H= 309:30 +CPU= 29.419
 NSTEP =  1239 STEPO   0AAA00AAA
 12:14:06 STEP 1239 H= 309:45 +CPU= 31.677
ISTEP=    1240 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1240 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1240 STEPO   0AAA00AAA
 12:14:38 STEP 1240 H= 310:00 +CPU= 32.237
 NSTEP =  1241 STEPO   0AAA00AAA
 12:15:10 STEP 1241 H= 310:15 +CPU= 31.788
 NSTEP =  1242 STEPO   0AAA00AAA
 12:15:42 STEP 1242 H= 310:30 +CPU= 31.566
 NSTEP =  1243 STEPO   0AAA00AAA
 12:16:14 STEP 1243 H= 310:45 +CPU= 29.833
 NSTEP =  1244 STEPO   0AAA00AAA
 12:16:46 STEP 1244 H= 311:00 +CPU= 29.522
ISTEP=    1245 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1245 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1245 STEPO   0AAA00AAA
 12:17:16 STEP 1245 H= 311:15 +CPU= 29.084
 NSTEP =  1246 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 12:14:06 0AAA00AAA STEPO     1240    31.678   31.678   31.705     68:34     72:23 0.25835240612853E-05       2GB       0MB
 12:14:38 0AAA00AAA STEPO     1241    32.237   32.237   32.261     69:07     72:55 0.25835240612853E-05       2GB       0MB
 12:15:10 0AAA00AAA STEPO     1242    31.788   31.788   31.821     69:38     73:27 0.25835240612853E-05       2GB       0MB
 12:15:42 0AAA00AAA STEPO     1243    31.566   31.566   31.603     70:10     73:59 0.25835240612853E-05       2GB       0MB
 12:16:14 0AAA00AAA STEPO     1244    29.834   29.834   31.858     70:40     74:31 0.25835240612853E-05       2GB       0MB
 12:16:46 0AAA00AAA STEPO     1245    29.523   29.523   32.359     71:09     75:03 0.25835240612853E-05       2GB       0MB
 12:17:16 0AAA00AAA STEPO     1246    29.084   29.084   30.140     71:38     75:33 0.25835240612853E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1200",
 CTIME   = "00121200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1188, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/9/progress_file_12283843.xml

<running_values>
  <last_cpu_time>41002.910000</last_cpu_time>
  <upload_file_number>12</upload_file_number>
  <last_iter>1246</last_iter>
  <last_upload>1036800</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1k0
(argv4) batchid: 1017
(argv5) wuid: 12283843
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_eb5e0c32e2d8a9343d4ea618eb25faa6 to: /var/lib/boinc-client/slots/9/oifs_43r3_bl_a1k0_2016092300_20_1017_12283843.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_bl_a1k0_2016092300_20_1017_12283843.zip
ic_ancil_file: ic_ancil_12283843
ifsdata_file: ifsdata_12283843
climate_data_file: clim_data_12283843
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/9/ic_ancil_12283843.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12283843.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b0aaac7fc21053d4ed176be27b2bc642 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12283843.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12283843.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12283843.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12283843.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12283843.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12283843.xml
-- Model is restarting --
Adjusting last_iter, 1246, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12283843.xml
last_cpu_time: 41002
upload_file_number: 12
last_iter: 1201
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490
The child process has been launched with process id: 13545
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13545:13545] [20240608:123408:1717842848:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:36:01 STEP 1200 H= 300:00 +CPU= 22.233
  12:36:24 STEP 1201 H= 300:15 +CPU= 22.710
  12:36:54 STEP 1202 H= 300:30 +CPU= 29.697
  12:37:27 STEP 1203 H= 300:45 +CPU= 29.852
  12:37:58 STEP 1204 H= 301:00 +CPU= 30.681
  12:38:29 STEP 1205 H= 301:15 +CPU= 29.515
  12:38:58 STEP 1206 H= 301:30 +CPU= 29.042
  12:39:27 STEP 1207 H= 301:45 +CPU= 28.834
  12:39:56 STEP 1208 H= 302:00 +CPU= 29.417
  12:40:25 STEP 1209 H= 302:15 +CPU= 29.242
  12:40:55 STEP 1210 H= 302:30 +CPU= 29.730
  12:41:24 STEP 1211 H= 302:45 +CPU= 29.062
  12:42:03 STEP 1212 H= 303:00 +CPU= 37.438
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001212
  12:42:33 STEP 1213 H= 303:15 +CPU= 30.371
  12:43:04 STEP 1214 H= 303:30 +CPU= 30.540
  12:43:34 STEP 1215 H= 303:45 +CPU= 30.564
  12:44:05 STEP 1216 H= 304:00 +CPU= 30.915
  12:44:36 STEP 1217 H= 304:15 +CPU= 30.346
  12:45:06 STEP 1218 H= 304:30 +CPU= 30.537
  12:45:39 STEP 1219 H= 304:45 +CPU= 30.767
  12:46:13 STEP 1220 H= 305:00 +CPU= 30.996
  12:46:46 STEP 1221 H= 305:15 +CPU= 30.523
  12:47:16 STEP 1222 H= 305:30 +CPU= 29.931
  12:47:51 STEP 1223 H= 305:45 +CPU= 31.188
  12:48:33 STEP 1224 H= 306:00 +CPU= 37.940
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001224
  12:49:08 STEP 1225 H= 306:15 +CPU= 30.967
  12:49:39 STEP 1226 H= 306:30 +CPU= 30.596
  12:50:10 STEP 1227 H= 306:45 +CPU= 30.410
  12:50:41 STEP 1228 H= 307:00 +CPU= 30.806
  12:51:11 STEP 1229 H= 307:15 +CPU= 29.967
  12:51:41 STEP 1230 H= 307:30 +CPU= 29.813
  12:52:11 STEP 1231 H= 307:45 +CPU= 30.330
  12:52:42 STEP 1232 H= 308:00 +CPU= 31.300
  12:53:13 STEP 1233 H= 308:15 +CPU= 30.529
  12:53:43 STEP 1234 H= 308:30 +CPU= 30.462
  12:54:15 STEP 1235 H= 308:45 +CPU= 31.857
  12:54:55 STEP 1236 H= 309:00 +CPU= 39.571
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001236
  12:55:28 STEP 1237 H= 309:15 +CPU= 32.183
  12:56:00 STEP 1238 H= 309:30 +CPU= 32.215
  12:56:32 STEP 1239 H= 309:45 +CPU= 32.564
  12:57:05 STEP 1240 H= 310:00 +CPU= 32.876
  12:57:34 STEP 1241 H= 310:15 +CPU= 29.028
  12:58:07 STEP 1242 H= 310:30 +CPU= 29.938
  12:58:39 STEP 1243 H= 310:45 +CPU= 30.585
  12:59:09 STEP 1244 H= 311:00 +CPU= 29.464
  12:59:38 STEP 1245 H= 311:15 +CPU= 28.976
  13:00:07 STEP 1246 H= 311:30 +CPU= 29.240
  13:00:36 STEP 1247 H= 311:45 +CPU= 29.050
  13:01:19 STEP 1248 H= 312:00 +CPU= 29.886
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_12.zip
  13:01:43 STEP 1249 H= 312:15 +CPU= 24.000
Uploading the intermediate file: upload_file_12.zip
  13:02:14 STEP 1250 H= 312:30 +CPU= 29.842
  13:02:47 STEP 1251 H= 312:45 +CPU= 29.853
  13:03:18 STEP 1252 H= 313:00 +CPU= 30.414
  13:03:48 STEP 1253 H= 313:15 +CPU= 29.938
  13:04:19 STEP 1254 H= 313:30 +CPU= 30.854
  13:04:49 STEP 1255 H= 313:45 +CPU= 30.369
  13:05:20 STEP 1256 H= 314:00 +CPU= 30.825
  13:05:50 STEP 1257 H= 314:15 +CPU= 30.230
  13:06:21 STEP 1258 H= 314:30 +CPU= 30.606
  13:06:52 STEP 1259 H= 314:45 +CPU= 30.612
  13:07:35 STEP 1260 H= 315:00 +CPU= 40.117
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001260
  13:08:07 STEP 1261 H= 315:15 +CPU= 30.767
  13:08:38 STEP 1262 H= 315:30 +CPU= 30.818
  13:09:08 STEP 1263 H= 315:45 +CPU= 30.727
  13:09:39 STEP 1264 H= 316:00 +CPU= 30.972
  13:10:10 STEP 1265 H= 316:15 +CPU= 30.568
  13:10:40 STEP 1266 H= 316:30 +CPU= 30.538
  13:11:11 STEP 1267 H= 316:45 +CPU= 30.801
  13:11:41 STEP 1268 H= 317:00 +CPU= 29.771
  13:12:10 STEP 1269 H= 317:15 +CPU= 28.648
  13:12:39 STEP 1270 H= 317:30 +CPU= 28.885
  13:13:08 STEP 1271 H= 317:45 +CPU= 28.988
  13:13:44 STEP 1272 H= 318:00 +CPU= 35.992
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001272
  13:14:12 STEP 1273 H= 318:15 +CPU= 28.561
  13:14:41 STEP 1274 H= 318:30 +CPU= 28.449
  13:15:10 STEP 1275 H= 318:45 +CPU= 28.850
  13:15:40 STEP 1276 H= 319:00 +CPU= 29.808
  13:16:09 STEP 1277 H= 319:15 +CPU= 29.189
  13:16:38 STEP 1278 H= 319:30 +CPU= 29.215
  13:17:07 STEP 1279 H= 319:45 +CPU= 28.775
  13:17:36 STEP 1280 H= 320:00 +CPU= 29.009
  13:18:05 STEP 1281 H= 320:15 +CPU= 28.482
  13:18:33 STEP 1282 H= 320:30 +CPU= 28.903
  13:19:02 STEP 1283 H= 320:45 +CPU= 28.847
  13:19:38 STEP 1284 H= 321:00 +CPU= 35.794
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001284
  13:20:07 STEP 1285 H= 321:15 +CPU= 28.598
  13:20:36 STEP 1286 H= 321:30 +CPU= 29.173
  13:21:05 STEP 1287 H= 321:45 +CPU= 28.880
  13:21:34 STEP 1288 H= 322:00 +CPU= 29.258
  13:22:03 STEP 1289 H= 322:15 +CPU= 28.783
  13:22:30 STEP 1290 H= 322:30 +CPU= 26.742
  13:22:57 STEP 1291 H= 322:45 +CPU= 27.747
  13:23:27 STEP 1292 H= 323:00 +CPU= 29.680
  13:23:57 STEP 1293 H= 323:15 +CPU= 29.285
  13:24:26 STEP 1294 H= 323:30 +CPU= 28.805
  13:24:57 STEP 1295 H= 323:45 +CPU= 30.510
  13:25:48 STEP 1296 H= 324:00 +CPU= 39.153
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001296
  13:26:20 STEP 1297 H= 324:15 +CPU= 30.252
  13:26:51 STEP 1298 H= 324:30 +CPU= 30.389
  13:27:24 STEP 1299 H= 324:45 +CPU= 31.059
  13:27:57 STEP 1300 H= 325:00 +CPU= 30.776
  13:28:27 STEP 1301 H= 325:15 +CPU= 30.452
  13:28:54 STEP 1302 H= 325:30 +CPU= 26.731
  13:29:23 STEP 1303 H= 325:45 +CPU= 29.044
  13:29:53 STEP 1304 H= 326:00 +CPU= 29.712
  13:30:22 STEP 1305 H= 326:15 +CPU= 29.324
  13:30:51 STEP 1306 H= 326:30 +CPU= 29.127
  13:31:20 STEP 1307 H= 326:45 +CPU= 29.197
  13:31:57 STEP 1308 H= 327:00 +CPU= 36.286
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001308
  13:32:26 STEP 1309 H= 327:15 +CPU= 28.969
  13:32:54 STEP 1310 H= 327:30 +CPU= 28.652
  13:33:23 STEP 1311 H= 327:45 +CPU= 28.801
  13:33:53 STEP 1312 H= 328:00 +CPU= 30.125
  13:34:23 STEP 1313 H= 328:15 +CPU= 29.499
  13:34:53 STEP 1314 H= 328:30 +CPU= 29.737
  13:35:23 STEP 1315 H= 328:45 +CPU= 29.882
  13:35:53 STEP 1316 H= 329:00 +CPU= 30.003
  13:36:24 STEP 1317 H= 329:15 +CPU= 30.261
  13:36:55 STEP 1318 H= 329:30 +CPU= 30.883
  13:37:26 STEP 1319 H= 329:45 +CPU= 29.556
  13:38:03 STEP 1320 H= 330:00 +CPU= 36.826
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001320
  13:38:33 STEP 1321 H= 330:15 +CPU= 29.622
  13:39:05 STEP 1322 H= 330:30 +CPU= 29.369
  13:39:37 STEP 1323 H= 330:45 +CPU= 30.121
  13:40:08 STEP 1324 H= 331:00 +CPU= 31.664
  13:40:39 STEP 1325 H= 331:15 +CPU= 30.820
  13:41:09 STEP 1326 H= 331:30 +CPU= 29.488
  13:41:39 STEP 1327 H= 331:45 +CPU= 29.763
  13:42:08 STEP 1328 H= 332:00 +CPU= 29.761
  13:42:38 STEP 1329 H= 332:15 +CPU= 29.336
  13:43:07 STEP 1330 H= 332:30 +CPU= 29.172
  13:43:36 STEP 1331 H= 332:45 +CPU= 28.689
  13:44:12 STEP 1332 H= 333:00 +CPU= 35.754
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001332
  13:44:40 STEP 1333 H= 333:15 +CPU= 28.930
  13:45:10 STEP 1334 H= 333:30 +CPU= 29.117
  13:45:39 STEP 1335 H= 333:45 +CPU= 28.876
  13:46:09 STEP 1336 H= 334:00 +CPU= 30.073
  13:46:38 STEP 1337 H= 334:15 +CPU= 29.499
  13:47:07 STEP 1338 H= 334:30 +CPU= 29.294
  13:47:37 STEP 1339 H= 334:45 +CPU= 29.421
  13:48:05 STEP 1340 H= 335:00 +CPU= 27.950
  13:48:34 STEP 1341 H= 335:15 +CPU= 29.140
  13:49:08 STEP 1342 H= 335:30 +CPU= 32.759
  13:49:46 STEP 1343 H= 335:45 +CPU= 32.985
  13:50:41 STEP 1344 H= 336:00 +CPU= 41.005
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_13.zip
Uploading the intermediate file: upload_file_13.zip
  13:51:12 STEP 1345 H= 336:15 +CPU= 30.024
  13:51:42 STEP 1346 H= 336:30 +CPU= 29.361
  13:52:12 STEP 1347 H= 336:45 +CPU= 30.258
  13:52:43 STEP 1348 H= 337:00 +CPU= 30.644
  13:53:13 STEP 1349 H= 337:15 +CPU= 30.515
  13:53:44 STEP 1350 H= 337:30 +CPU= 30.345
  13:54:13 STEP 1351 H= 337:45 +CPU= 29.369
  13:54:43 STEP 1352 H= 338:00 +CPU= 28.785
  13:55:16 STEP 1353 H= 338:15 +CPU= 29.431
  13:55:47 STEP 1354 H= 338:30 +CPU= 30.782
  13:56:18 STEP 1355 H= 338:45 +CPU= 31.826
  13:56:58 STEP 1356 H= 339:00 +CPU= 39.497
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001356
  13:57:30 STEP 1357 H= 339:15 +CPU= 32.154
  13:58:02 STEP 1358 H= 339:30 +CPU= 32.349
  13:58:35 STEP 1359 H= 339:45 +CPU= 32.177
  13:59:07 STEP 1360 H= 340:00 +CPU= 32.623
  13:59:40 STEP 1361 H= 340:15 +CPU= 32.386
  14:00:12 STEP 1362 H= 340:30 +CPU= 32.493
  14:00:45 STEP 1363 H= 340:45 +CPU= 32.262
  14:01:17 STEP 1364 H= 341:00 +CPU= 32.789
  14:01:49 STEP 1365 H= 341:15 +CPU= 31.895
  14:02:25 STEP 1366 H= 341:30 +CPU= 31.938
  14:02:59 STEP 1367 H= 341:45 +CPU= 31.813
  14:03:39 STEP 1368 H= 342:00 +CPU= 38.967
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001368
  14:04:09 STEP 1369 H= 342:15 +CPU= 28.937
  14:04:38 STEP 1370 H= 342:30 +CPU= 29.127
  14:05:07 STEP 1371 H= 342:45 +CPU= 29.205
  14:05:37 STEP 1372 H= 343:00 +CPU= 29.692
  14:06:05 STEP 1373 H= 343:15 +CPU= 28.792
  14:06:35 STEP 1374 H= 343:30 +CPU= 29.354
  14:07:05 STEP 1375 H= 343:45 +CPU= 29.775
  14:07:36 STEP 1376 H= 344:00 +CPU= 31.194
  14:08:07 STEP 1377 H= 344:15 +CPU= 31.253
  14:08:37 STEP 1378 H= 344:30 +CPU= 30.178
  14:09:10 STEP 1379 H= 344:45 +CPU= 32.422
  14:09:50 STEP 1380 H= 345:00 +CPU= 39.699
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001380
  14:10:23 STEP 1381 H= 345:15 +CPU= 32.348
  14:10:55 STEP 1382 H= 345:30 +CPU= 32.060
  14:11:27 STEP 1383 H= 345:45 +CPU= 32.303
  14:12:00 STEP 1384 H= 346:00 +CPU= 32.775
  14:12:32 STEP 1385 H= 346:15 +CPU= 32.114
  14:13:01 STEP 1386 H= 346:30 +CPU= 28.273
  14:13:31 STEP 1387 H= 346:45 +CPU= 30.539
  14:14:03 STEP 1388 H= 347:00 +CPU= 31.240
  14:14:35 STEP 1389 H= 347:15 +CPU= 31.588
  14:15:06 STEP 1390 H= 347:30 +CPU= 30.950
  14:15:37 STEP 1391 H= 347:45 +CPU= 30.632
  14:16:30 STEP 1392 H= 348:00 +CPU= 40.229
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001392
  14:17:01 STEP 1393 H= 348:15 +CPU= 29.386
  14:17:30 STEP 1394 H= 348:30 +CPU= 29.036
  14:17:59 STEP 1395 H= 348:45 +CPU= 29.235
  14:18:30 STEP 1396 H= 349:00 +CPU= 29.739
  14:19:01 STEP 1397 H= 349:15 +CPU= 29.776
  14:19:33 STEP 1398 H= 349:30 +CPU= 28.884
  14:20:04 STEP 1399 H= 349:45 +CPU= 30.423
  14:20:35 STEP 1400 H= 350:00 +CPU= 31.415
  14:21:05 STEP 1401 H= 350:15 +CPU= 29.844
  14:21:39 STEP 1402 H= 350:30 +CPU= 31.682
  14:22:11 STEP 1403 H= 350:45 +CPU= 32.096
  14:22:51 STEP 1404 H= 351:00 +CPU= 39.686
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001404
  14:23:22 STEP 1405 H= 351:15 +CPU= 31.316
  14:23:54 STEP 1406 H= 351:30 +CPU= 31.702
  14:24:25 STEP 1407 H= 351:45 +CPU= 31.696
  14:24:58 STEP 1408 H= 352:00 +CPU= 32.723
  14:25:30 STEP 1409 H= 352:15 +CPU= 31.996
  14:26:02 STEP 1410 H= 352:30 +CPU= 32.138
  14:26:33 STEP 1411 H= 352:45 +CPU= 30.934
  14:27:04 STEP 1412 H= 353:00 +CPU= 31.030
  14:27:34 STEP 1413 H= 353:15 +CPU= 30.036
  14:28:04 STEP 1414 H= 353:30 +CPU= 30.031
  14:28:35 STEP 1415 H= 353:45 +CPU= 30.210
  14:29:17 STEP 1416 H= 354:00 +CPU= 37.982
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001416
  14:29:48 STEP 1417 H= 354:15 +CPU= 29.079
  14:30:18 STEP 1418 H= 354:30 +CPU= 29.633
  14:30:49 STEP 1419 H= 354:45 +CPU= 30.666
  14:31:20 STEP 1420 H= 355:00 +CPU= 30.823
  14:31:51 STEP 1421 H= 355:15 +CPU= 31.452
  14:32:23 STEP 1422 H= 355:30 +CPU= 31.460
  14:32:53 STEP 1423 H= 355:45 +CPU= 30.142
  14:33:24 STEP 1424 H= 356:00 +CPU= 31.098
  14:33:54 STEP 1425 H= 356:15 +CPU= 30.165
  14:34:25 STEP 1426 H= 356:30 +CPU= 30.310
  14:34:55 STEP 1427 H= 356:45 +CPU= 30.357
  14:35:32 STEP 1428 H= 357:00 +CPU= 36.910
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001428
  14:36:02 STEP 1429 H= 357:15 +CPU= 29.995
  14:36:32 STEP 1430 H= 357:30 +CPU= 30.238
  14:37:03 STEP 1431 H= 357:45 +CPU= 30.619
  14:37:32 STEP 1432 H= 358:00 +CPU= 29.196
  14:38:02 STEP 1433 H= 358:15 +CPU= 30.032
  14:38:32 STEP 1434 H= 358:30 +CPU= 30.079
  14:39:02 STEP 1435 H= 358:45 +CPU= 29.515
  14:39:32 STEP 1436 H= 359:00 +CPU= 30.485
  14:40:02 STEP 1437 H= 359:15 +CPU= 30.131
  14:40:33 STEP 1438 H= 359:30 +CPU= 30.269
  14:41:03 STEP 1439 H= 359:45 +CPU= 30.008
  14:41:53 STEP 1440 H= 360:00 +CPU= 36.827
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1906088101E-20      I      -0.1209085326E-06      I
ICloud fractionI      0.0000000000E+00      I       0.3911791915E-01      I
I  Relat. Hum. I      0.4977339840E+00      I       0.5109344419E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.5416699788E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    327581220 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS818953.0 Mbytes/s, TOTAL TIME=    0.0119( 3.4%)
 14:41:53 STEP 1440 H= 360:00 +CPU= 36.827
 END CNT3
 NSTEP =  1440 CNT0   000000000  27.711  27.711  27.751   0.999               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         7150/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    128    0       46      1044068     7150 ************   112.9    87.0    25.9    87.5    42.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 7664959.5 7664959.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 7576368.0 7576368.0     98.84      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  261    4640.0    4640.0     15.80      0.00
   9 SPCM     - SPECTRAL COMP.                     240     900.9     900.9      2.82      0.00
  10 SCAN2M - PHYSICS                              241   15269.7   15269.7     48.01      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 20    3121.4    3121.4      0.81      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 11      57.5      57.5      0.01      0.00
  17 GRIDFPOS IN CNT4                               20      30.8      30.8      0.01      0.00
  27 RESTART FILES - WRITING                         5   12481.2   12481.2      0.81      0.00
  28 RESTART FILES - READING                         1   88052.2   88052.2      1.15      0.00
  29 SU4FPOS IN CNT4                                20       1.0       1.0      0.00      0.00
  30 DYNFPOS IN CNT4                                20    6913.2    6913.2      1.80      0.00
  31 POSDDH IN STEPO                                 2      39.3      39.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    241    6435.2    6435.2     20.23      0.00
  39 SU0YOMB                                         1     301.1     301.1      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    241      35.9      35.9      0.11      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              240     400.0     400.0      1.25      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              240      73.0      73.0      0.23      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 11       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      524     461.1     461.1      3.15      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       500     278.2     278.2      1.81      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        500      94.3      94.3      0.62      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       524     153.6     153.6      1.05      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      39.9      39.9      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            524      46.1      46.1      0.32      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            500      26.5      26.5      0.17      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            524      96.2      96.2      0.66      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            500      26.1      26.1      0.17      0.00
 400 GSTATS                                      28864       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 26802       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     7665.0 CPU TIME    7412.8 VECTOR TIME     7412.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      171.1
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/9/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/9/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283843/ICMSHh7zg+001428
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
14:45:24 (13533): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0_r1729817490_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 12:40:32 1532094 22435422 oifs_43r3_bl_a1k0_2016092300_20_1017_12283843_0 1,295,100 48,414 0.0374


©2024 climateprediction.net