climateprediction.net home page
Task 22312924

Task 22312924

Name oifs_43r3_bl_a4j1_2016092300_15_991_12212656_0
Workunit 12212656
Created 13 Feb 2023, 22:48:37 UTC
Sent 14 Feb 2023, 6:22:11 UTC
Report deadline 15 Apr 2023, 6:22:11 UTC
Received 16 Feb 2023, 18:28:23 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1506016
Run time 11 hours 30 min 36 sec
CPU time 11 hours 30 min 36 sec
Validate state Valid
Credit 1,813.14
Device peak FLOPS 5.52 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,071.89 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,396.08 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
re-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:14155:14155] [20230215:122921:1676492961:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:29:51 STEP 1104 H= 276:00 +CPU= 24.755
  12:30:17 STEP 1105 H= 276:15 +CPU= 24.442
  12:30:42 STEP 1106 H= 276:30 +CPU= 23.998
  12:31:07 STEP 1107 H= 276:45 +CPU= 24.008
  12:31:33 STEP 1108 H= 277:00 +CPU= 24.395
  12:31:59 STEP 1109 H= 277:15 +CPU= 24.018
  12:32:24 STEP 1110 H= 277:30 +CPU= 23.658
  12:32:49 STEP 1111 H= 277:45 +CPU= 23.812
  12:33:14 STEP 1112 H= 278:00 +CPU= 24.187
  12:33:39 STEP 1113 H= 278:15 +CPU= 23.654
  12:34:04 STEP 1114 H= 278:30 +CPU= 23.904
  12:34:30 STEP 1115 H= 278:45 +CPU= 23.963
  12:35:01 STEP 1116 H= 279:00 +CPU= 29.293
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001116
  12:35:26 STEP 1117 H= 279:15 +CPU= 24.040
  12:35:52 STEP 1118 H= 279:30 +CPU= 24.324
  12:36:17 STEP 1119 H= 279:45 +CPU= 24.260
  12:36:43 STEP 1120 H= 280:00 +CPU= 24.164
  12:37:08 STEP 1121 H= 280:15 +CPU= 24.037
  12:37:33 STEP 1122 H= 280:30 +CPU= 23.755
  12:37:58 STEP 1123 H= 280:45 +CPU= 23.787
  12:38:24 STEP 1124 H= 281:00 +CPU= 24.290
  12:38:49 STEP 1125 H= 281:15 +CPU= 23.680
  12:39:14 STEP 1126 H= 281:30 +CPU= 24.070
  12:39:39 STEP 1127 H= 281:45 +CPU= 23.655
  12:40:11 STEP 1128 H= 282:00 +CPU= 29.771
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001128
  12:40:36 STEP 1129 H= 282:15 +CPU= 23.641
  12:41:01 STEP 1130 H= 282:30 +CPU= 24.170
  12:41:26 STEP 1131 H= 282:45 +CPU= 23.853
  12:41:52 STEP 1132 H= 283:00 +CPU= 24.634
  12:42:18 STEP 1133 H= 283:15 +CPU= 23.876
  12:42:43 STEP 1134 H= 283:30 +CPU= 24.155
  12:43:08 STEP 1135 H= 283:45 +CPU= 23.837
  12:43:34 STEP 1136 H= 284:00 +CPU= 24.442
  12:44:00 STEP 1137 H= 284:15 +CPU= 24.104
  12:44:25 STEP 1138 H= 284:30 +CPU= 23.981
  12:44:50 STEP 1139 H= 284:45 +CPU= 24.022
  12:45:22 STEP 1140 H= 285:00 +CPU= 29.905
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001140
  12:45:47 STEP 1141 H= 285:15 +CPU= 23.977
  12:46:12 STEP 1142 H= 285:30 +CPU= 23.987
  12:46:38 STEP 1143 H= 285:45 +CPU= 24.068
  12:47:04 STEP 1144 H= 286:00 +CPU= 24.606
  12:47:29 STEP 1145 H= 286:15 +CPU= 23.811
  12:47:54 STEP 1146 H= 286:30 +CPU= 24.205
  12:48:20 STEP 1147 H= 286:45 +CPU= 24.001
  12:48:45 STEP 1148 H= 287:00 +CPU= 24.380
  12:49:11 STEP 1149 H= 287:15 +CPU= 23.888
  12:49:36 STEP 1150 H= 287:30 +CPU= 23.850
  12:50:01 STEP 1151 H= 287:45 +CPU= 23.703
  12:50:38 STEP 1152 H= 288:00 +CPU= 32.968
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4j1_2016092300_15_991_12212656_0_r2063574814_11.zip
Uploading the intermediate file: upload_file_11.zip
  12:51:05 STEP 1153 H= 288:15 +CPU= 24.193
  12:51:32 STEP 1154 H= 288:30 +CPU= 25.402
  12:52:02 STEP 1155 H= 288:45 +CPU= 26.694
  12:52:33 STEP 1156 H= 289:00 +CPU= 26.163
  12:53:00 STEP 1157 H= 289:15 +CPU= 24.828
  12:53:29 STEP 1158 H= 289:30 +CPU= 25.605
  12:53:59 STEP 1159 H= 289:45 +CPU= 25.438
  12:54:31 STEP 1160 H= 290:00 +CPU= 26.317
  12:55:01 STEP 1161 H= 290:15 +CPU= 25.497
  12:55:31 STEP 1162 H= 290:30 +CPU= 25.675
  12:56:03 STEP 1163 H= 290:45 +CPU= 25.636
  12:56:41 STEP 1164 H= 291:00 +CPU= 32.096
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001164
  12:57:13 STEP 1165 H= 291:15 +CPU= 25.894
  12:57:43 STEP 1166 H= 291:30 +CPU= 25.662
  12:58:13 STEP 1167 H= 291:45 +CPU= 25.608
  12:58:43 STEP 1168 H= 292:00 +CPU= 26.069
  12:59:12 STEP 1169 H= 292:15 +CPU= 25.690
  12:59:43 STEP 1170 H= 292:30 +CPU= 25.918
  13:00:14 STEP 1171 H= 292:45 +CPU= 25.676
  13:00:46 STEP 1172 H= 293:00 +CPU= 26.372
  13:01:16 STEP 1173 H= 293:15 +CPU= 25.660
  13:01:45 STEP 1174 H= 293:30 +CPU= 25.364
  13:02:15 STEP 1175 H= 293:45 +CPU= 25.776
  13:02:52 STEP 1176 H= 294:00 +CPU= 32.010
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001176
  13:03:21 STEP 1177 H= 294:15 +CPU= 25.089
  13:03:49 STEP 1178 H= 294:30 +CPU= 25.440
  13:04:17 STEP 1179 H= 294:45 +CPU= 25.357
  13:04:46 STEP 1180 H= 295:00 +CPU= 25.836
  13:05:14 STEP 1181 H= 295:15 +CPU= 26.034
  13:05:42 STEP 1182 H= 295:30 +CPU= 25.681
  13:06:10 STEP 1183 H= 295:45 +CPU= 25.587
  13:06:38 STEP 1184 H= 296:00 +CPU= 25.817
  13:07:06 STEP 1185 H= 296:15 +CPU= 25.767
  13:07:33 STEP 1186 H= 296:30 +CPU= 24.956
  13:08:01 STEP 1187 H= 296:45 +CPU= 25.448
  13:08:36 STEP 1188 H= 297:00 +CPU= 32.156
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001188
  13:09:03 STEP 1189 H= 297:15 +CPU= 25.616
  13:09:31 STEP 1190 H= 297:30 +CPU= 25.900
  13:09:59 STEP 1191 H= 297:45 +CPU= 25.721
  13:10:26 STEP 1192 H= 298:00 +CPU= 26.073
  13:10:55 STEP 1193 H= 298:15 +CPU= 25.789
  13:11:22 STEP 1194 H= 298:30 +CPU= 25.773
  13:11:50 STEP 1195 H= 298:45 +CPU= 25.544
  13:12:18 STEP 1196 H= 299:00 +CPU= 25.861
  13:12:45 STEP 1197 H= 299:15 +CPU= 25.662
  13:13:13 STEP 1198 H= 299:30 +CPU= 25.503
  13:13:41 STEP 1199 H= 299:45 +CPU= 25.312
  13:14:20 STEP 1200 H= 300:00 +CPU= 35.722
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001200
  13:14:47 STEP 1201 H= 300:15 +CPU= 25.604
  13:15:15 STEP 1202 H= 300:30 +CPU= 25.559
  13:15:42 STEP 1203 H= 300:45 +CPU= 25.349
  13:16:10 STEP 1204 H= 301:00 +CPU= 25.931
  13:16:37 STEP 1205 H= 301:15 +CPU= 25.630
  13:17:05 STEP 1206 H= 301:30 +CPU= 25.543
  13:17:31 STEP 1207 H= 301:45 +CPU= 25.275
  13:17:59 STEP 1208 H= 302:00 +CPU= 26.349
  13:18:26 STEP 1209 H= 302:15 +CPU= 25.019
  13:18:54 STEP 1210 H= 302:30 +CPU= 25.699
  13:19:21 STEP 1211 H= 302:45 +CPU= 25.476
  13:19:56 STEP 1212 H= 303:00 +CPU= 31.972
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001212
  13:20:24 STEP 1213 H= 303:15 +CPU= 25.361
  13:20:51 STEP 1214 H= 303:30 +CPU= 25.350
  13:21:19 STEP 1215 H= 303:45 +CPU= 25.494
  13:21:48 STEP 1216 H= 304:00 +CPU= 26.192
  13:22:15 STEP 1217 H= 304:15 +CPU= 25.438
  13:22:42 STEP 1218 H= 304:30 +CPU= 25.602
  13:23:10 STEP 1219 H= 304:45 +CPU= 25.367
  13:23:37 STEP 1220 H= 305:00 +CPU= 25.712
  13:24:05 STEP 1221 H= 305:15 +CPU= 25.499
  13:24:32 STEP 1222 H= 305:30 +CPU= 25.219
  13:24:59 STEP 1223 H= 305:45 +CPU= 25.863
  13:25:33 STEP 1224 H= 306:00 +CPU= 31.699
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001224
  13:26:00 STEP 1225 H= 306:15 +CPU= 25.123
  13:26:28 STEP 1226 H= 306:30 +CPU= 25.184
  13:26:55 STEP 1227 H= 306:45 +CPU= 25.398
  13:27:23 STEP 1228 H= 307:00 +CPU= 25.948
  13:27:50 STEP 1229 H= 307:15 +CPU= 25.186
  13:28:17 STEP 1230 H= 307:30 +CPU= 24.981
  13:28:44 STEP 1231 H= 307:45 +CPU= 25.216
  13:29:12 STEP 1232 H= 308:00 +CPU= 25.809
  13:29:39 STEP 1233 H= 308:15 +CPU= 25.326
  13:30:06 STEP 1234 H= 308:30 +CPU= 25.399
  13:30:34 STEP 1235 H= 308:45 +CPU= 25.434
  13:31:12 STEP 1236 H= 309:00 +CPU= 31.650
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001236
  13:31:39 STEP 1237 H= 309:15 +CPU= 25.180
  13:32:05 STEP 1238 H= 309:30 +CPU= 25.127
  13:32:32 STEP 1239 H= 309:45 +CPU= 25.113
  13:32:59 STEP 1240 H= 310:00 +CPU= 25.446
  13:33:27 STEP 1241 H= 310:15 +CPU= 25.559
  13:33:54 STEP 1242 H= 310:30 +CPU= 25.580
  13:34:22 STEP 1243 H= 310:45 +CPU= 25.363
  13:34:49 STEP 1244 H= 311:00 +CPU= 25.890
  13:35:18 STEP 1245 H= 311:15 +CPU= 25.968
  13:35:46 STEP 1246 H= 311:30 +CPU= 26.208
  13:36:14 STEP 1247 H= 311:45 +CPU= 26.117
  13:36:54 STEP 1248 H= 312:00 +CPU= 35.166
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4j1_2016092300_15_991_12212656_0_r2063574814_12.zip
Uploading the intermediate file: upload_file_12.zip
  13:37:23 STEP 1249 H= 312:15 +CPU= 25.972
  13:37:51 STEP 1250 H= 312:30 +CPU= 25.700
  13:38:19 STEP 1251 H= 312:45 +CPU= 26.038
  13:38:48 STEP 1252 H= 313:00 +CPU= 26.596
  13:39:15 STEP 1253 H= 313:15 +CPU= 25.679
  13:39:42 STEP 1254 H= 313:30 +CPU= 25.785
  13:40:10 STEP 1255 H= 313:45 +CPU= 25.883
  13:40:39 STEP 1256 H= 314:00 +CPU= 26.492
  13:41:06 STEP 1257 H= 314:15 +CPU= 25.659
  13:41:35 STEP 1258 H= 314:30 +CPU= 26.076
  13:42:02 STEP 1259 H= 314:45 +CPU= 25.254
  13:42:37 STEP 1260 H= 315:00 +CPU= 32.575
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001260
  13:43:05 STEP 1261 H= 315:15 +CPU= 25.801
  13:43:33 STEP 1262 H= 315:30 +CPU= 25.798
  13:44:01 STEP 1263 H= 315:45 +CPU= 26.166
  13:44:29 STEP 1264 H= 316:00 +CPU= 26.372
  13:44:57 STEP 1265 H= 316:15 +CPU= 25.914
  13:45:25 STEP 1266 H= 316:30 +CPU= 25.977
  13:45:54 STEP 1267 H= 316:45 +CPU= 26.114
  13:46:21 STEP 1268 H= 317:00 +CPU= 26.137
  13:46:49 STEP 1269 H= 317:15 +CPU= 25.921
  13:47:17 STEP 1270 H= 317:30 +CPU= 25.841
  13:47:45 STEP 1271 H= 317:45 +CPU= 26.094
  13:48:20 STEP 1272 H= 318:00 +CPU= 32.520
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001272
  13:48:47 STEP 1273 H= 318:15 +CPU= 26.043
  13:49:15 STEP 1274 H= 318:30 +CPU= 25.978
  13:49:42 STEP 1275 H= 318:45 +CPU= 25.692
  13:50:11 STEP 1276 H= 319:00 +CPU= 26.468
  13:50:39 STEP 1277 H= 319:15 +CPU= 25.904
  13:51:06 STEP 1278 H= 319:30 +CPU= 25.648
  13:51:36 STEP 1279 H= 319:45 +CPU= 26.056
  13:52:05 STEP 1280 H= 320:00 +CPU= 26.297
  13:52:32 STEP 1281 H= 320:15 +CPU= 25.530
  13:53:00 STEP 1282 H= 320:30 +CPU= 25.657
  13:53:30 STEP 1283 H= 320:45 +CPU= 27.206
  13:54:04 STEP 1284 H= 321:00 +CPU= 31.766
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001284
  13:54:32 STEP 1285 H= 321:15 +CPU= 26.637
  13:55:00 STEP 1286 H= 321:30 +CPU= 25.846
  13:55:28 STEP 1287 H= 321:45 +CPU= 25.803
  13:56:01 STEP 1288 H= 322:00 +CPU= 29.134
  13:56:29 STEP 1289 H= 322:15 +CPU= 26.939
  13:57:00 STEP 1290 H= 322:30 +CPU= 27.316
  13:57:28 STEP 1291 H= 322:45 +CPU= 25.665
  13:57:58 STEP 1292 H= 323:00 +CPU= 27.740
  13:58:28 STEP 1293 H= 323:15 +CPU= 27.471
  13:58:57 STEP 1294 H= 323:30 +CPU= 27.297
  13:59:25 STEP 1295 H= 323:45 +CPU= 25.555
  14:00:03 STEP 1296 H= 324:00 +CPU= 35.347
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001296
  14:00:31 STEP 1297 H= 324:15 +CPU= 26.074
  14:00:59 STEP 1298 H= 324:30 +CPU= 25.658
  14:01:27 STEP 1299 H= 324:45 +CPU= 25.719
  14:01:56 STEP 1300 H= 325:00 +CPU= 26.420
  14:02:23 STEP 1301 H= 325:15 +CPU= 25.544
  14:02:52 STEP 1302 H= 325:30 +CPU= 25.920
  14:03:21 STEP 1303 H= 325:45 +CPU= 26.407
  14:03:53 STEP 1304 H= 326:00 +CPU= 29.008
  14:04:21 STEP 1305 H= 326:15 +CPU= 26.049
  14:04:49 STEP 1306 H= 326:30 +CPU= 26.441
  14:05:17 STEP 1307 H= 326:45 +CPU= 25.730
  14:05:52 STEP 1308 H= 327:00 +CPU= 32.202
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001308
  14:06:20 STEP 1309 H= 327:15 +CPU= 25.574
  14:06:49 STEP 1310 H= 327:30 +CPU= 26.667
  14:07:17 STEP 1311 H= 327:45 +CPU= 25.819
  14:07:46 STEP 1312 H= 328:00 +CPU= 27.017
  14:08:13 STEP 1313 H= 328:15 +CPU= 25.672
  14:08:44 STEP 1314 H= 328:30 +CPU= 27.985
  14:09:11 STEP 1315 H= 328:45 +CPU= 25.901
  14:09:39 STEP 1316 H= 329:00 +CPU= 26.129
  14:10:07 STEP 1317 H= 329:15 +CPU= 25.528
  14:10:36 STEP 1318 H= 329:30 +CPU= 27.043
  14:11:05 STEP 1319 H= 329:45 +CPU= 26.289
  14:11:39 STEP 1320 H= 330:00 +CPU= 32.239
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001320
  14:12:07 STEP 1321 H= 330:15 +CPU= 25.752
  14:12:37 STEP 1322 H= 330:30 +CPU= 27.528
  14:13:04 STEP 1323 H= 330:45 +CPU= 25.581
  14:13:34 STEP 1324 H= 331:00 +CPU= 26.639
  14:14:01 STEP 1325 H= 331:15 +CPU= 25.525
  14:14:29 STEP 1326 H= 331:30 +CPU= 25.668
  14:14:57 STEP 1327 H= 331:45 +CPU= 25.920
  14:15:25 STEP 1328 H= 332:00 +CPU= 26.021
  14:15:53 STEP 1329 H= 332:15 +CPU= 26.481
  14:16:21 STEP 1330 H= 332:30 +CPU= 25.824
  14:16:49 STEP 1331 H= 332:45 +CPU= 25.645
  14:17:24 STEP 1332 H= 333:00 +CPU= 32.763
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001332
  14:17:54 STEP 1333 H= 333:15 +CPU= 28.554
  14:18:23 STEP 1334 H= 333:30 +CPU= 25.940
  14:18:50 STEP 1335 H= 333:45 +CPU= 26.038
  14:19:20 STEP 1336 H= 334:00 +CPU= 27.653
  14:19:48 STEP 1337 H= 334:15 +CPU= 26.325
  14:20:16 STEP 1338 H= 334:30 +CPU= 25.686
  14:20:44 STEP 1339 H= 334:45 +CPU= 26.047
  14:21:13 STEP 1340 H= 335:00 +CPU= 27.568
  14:21:42 STEP 1341 H= 335:15 +CPU= 26.384
  14:22:11 STEP 1342 H= 335:30 +CPU= 25.663
  14:22:41 STEP 1343 H= 335:45 +CPU= 28.136
  14:23:21 STEP 1344 H= 336:00 +CPU= 36.815
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4j1_2016092300_15_991_12212656_0_r2063574814_13.zip
Uploading the intermediate file: upload_file_13.zip
  14:23:52 STEP 1345 H= 336:15 +CPU= 27.192
  14:24:20 STEP 1346 H= 336:30 +CPU= 25.690
  14:24:51 STEP 1347 H= 336:45 +CPU= 28.279
  14:25:20 STEP 1348 H= 337:00 +CPU= 26.412
  14:25:48 STEP 1349 H= 337:15 +CPU= 25.892
  14:26:16 STEP 1350 H= 337:30 +CPU= 26.959
  14:26:47 STEP 1351 H= 337:45 +CPU= 28.518
  14:27:15 STEP 1352 H= 338:00 +CPU= 26.032
  14:27:44 STEP 1353 H= 338:15 +CPU= 26.635
  14:28:11 STEP 1354 H= 338:30 +CPU= 25.637
  14:28:39 STEP 1355 H= 338:45 +CPU= 25.989
  14:29:13 STEP 1356 H= 339:00 +CPU= 31.697
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001356
  14:29:43 STEP 1357 H= 339:15 +CPU= 26.583
  14:30:12 STEP 1358 H= 339:30 +CPU= 26.704
  14:30:42 STEP 1359 H= 339:45 +CPU= 26.581
  14:31:11 STEP 1360 H= 340:00 +CPU= 26.876
  14:31:40 STEP 1361 H= 340:15 +CPU= 26.167
  14:32:08 STEP 1362 H= 340:30 +CPU= 26.342
  14:32:37 STEP 1363 H= 340:45 +CPU= 26.707
  14:33:05 STEP 1364 H= 341:00 +CPU= 26.406
  14:33:33 STEP 1365 H= 341:15 +CPU= 26.579
  14:34:01 STEP 1366 H= 341:30 +CPU= 26.111
  14:34:29 STEP 1367 H= 341:45 +CPU= 26.106
  14:35:04 STEP 1368 H= 342:00 +CPU= 32.115
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001368
  14:35:32 STEP 1369 H= 342:15 +CPU= 25.582
  14:36:00 STEP 1370 H= 342:30 +CPU= 26.987
  14:36:28 STEP 1371 H= 342:45 +CPU= 25.446
  14:36:57 STEP 1372 H= 343:00 +CPU= 26.556
  14:37:27 STEP 1373 H= 343:15 +CPU= 27.355
  14:37:55 STEP 1374 H= 343:30 +CPU= 26.239
  14:38:23 STEP 1375 H= 343:45 +CPU= 26.170
  14:38:55 STEP 1376 H= 344:00 +CPU= 28.437
  14:39:23 STEP 1377 H= 344:15 +CPU= 25.996
  14:39:50 STEP 1378 H= 344:30 +CPU= 25.773
  14:40:18 STEP 1379 H= 344:45 +CPU= 25.675
  14:40:52 STEP 1380 H= 345:00 +CPU= 32.232
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001380
  14:41:20 STEP 1381 H= 345:15 +CPU= 26.092
  14:41:49 STEP 1382 H= 345:30 +CPU= 26.334
  14:42:21 STEP 1383 H= 345:45 +CPU= 28.290
  14:42:52 STEP 1384 H= 346:00 +CPU= 28.448
  14:43:22 STEP 1385 H= 346:15 +CPU= 28.413
  14:43:51 STEP 1386 H= 346:30 +CPU= 26.577
  14:44:21 STEP 1387 H= 346:45 +CPU= 28.144
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.211621407621329E-05 0.000000000000000E+00 0.683005137176531E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.902000272222187E-06 0.000000000000000E+00 0.334431936287866E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.180138580032741E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.566784660413444E-06 0.000000000000000E+00 0.152550773030670E-02
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.224313901549074E-05 0.000000000000000E+00 0.101602340545141E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.318799944468210E-06 0.000000000000000E+00 0.641952584077931E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.106232377866544E-06 0.000000000000000E+00 0.565596517303526E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.101371605568146E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1380 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1380 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001380 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001380
 MPI-TASK:   1 -    294782666 BYTES IN      6 RECORDS TRANSFERRED IN    0.0005 SECONDS589565.3 Mbytes/s, TOTAL TIME=    0.0095( 5.3%)
 NSTEP =  1380 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001380 MODE=a
 IO-STREAM CLOSED - ./ICMSHh7zg+001380
 MPI-TASK:   1 -    707765250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0102 SECONDS 69388.7 Mbytes/s, TOTAL TIME=    0.3061( 3.3%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001380 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001380
 MPI-TASK:   1 -   4716522647 BYTES IN    125 RECORDS TRANSFERRED IN    0.0092 SECONDS512665.5 Mbytes/s, TOTAL TIME=    0.2554( 3.6%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001380 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001380
 MPI-TASK:   1 -    286209033 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0035( 5.7%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001380 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001380
 MPI-TASK:   1 -    294782666 BYTES IN      4 RECORDS TRANSFERRED IN    0.0003 SECONDS982608.8 Mbytes/s, TOTAL TIME=    0.0063( 4.8%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1380 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1380 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001380 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001380
 MPI-TASK:   1 -    294782657 BYTES IN      7 RECORDS TRANSFERRED IN    0.0005 SECONDS589565.3 Mbytes/s, TOTAL TIME=    0.0102( 4.9%)
 14:40:52 STEP 1380 H= 345:00 +CPU= 32.232
 NSTEP =  1381 STEPO   0AAA00AAA
 14:41:20 STEP 1381 H= 345:15 +CPU= 26.092
 NSTEP =  1382 STEPO   0AAA00AAA
 14:41:49 STEP 1382 H= 345:30 +CPU= 26.334
 NSTEP =  1383 STEPO   0AAA00AAA
 14:42:21 STEP 1383 H= 345:45 +CPU= 28.290
 NSTEP =  1384 STEPO   0AAA00AAA
 14:42:52 STEP 1384 H= 346:00 +CPU= 28.448
ISTEP=    1385 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1385 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1385 STEPO   0AAA00AAA
 14:43:22 STEP 1385 H= 346:15 +CPU= 28.413
 NSTEP =  1386 STEPO   0AAA00AAA
 14:43:51 STEP 1386 H= 346:30 +CPU= 26.577
 NSTEP =  1387 STEPO   0AAA00AAA
 14:44:21 STEP 1387 H= 346:45 +CPU= 28.144
 NSTEP =  1388 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 14:41:20 0AAA00AAA STEPO     1382    26.092   26.092   28.044    121:36    131:59 0.89300333908169E-05       2GB       0MB
 14:41:49 0AAA00AAA STEPO     1383    26.334   26.334   28.165    122:03    132:27 0.89300333908169E-05       2GB       0MB
 14:42:21 0AAA00AAA STEPO     1384    28.290   28.290   32.175    122:31    132:59 0.89300333908169E-05       2GB       0MB
 14:42:52 0AAA00AAA STEPO     1385    28.448   28.448   30.796    123:00    133:30 0.89300333908169E-05       2GB       0MB
 14:43:22 0AAA00AAA STEPO     1386    28.413   28.413   30.698    123:28    134:01 0.89300333908169E-05       2GB       0MB
 14:43:51 0AAA00AAA STEPO     1387    26.577   26.577   28.824    123:55    134:30 0.89300333908169E-05       2GB       0MB
 14:44:21 0AAA00AAA STEPO     1388    28.144   28.144   30.319    124:23    135:00 0.89300333908169E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1344",
 CTIME   = "00140000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1332, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/7/progress_file_12212656.xml

<running_values>
  <last_cpu_time>39209.720000</last_cpu_time>
  <upload_file_number>14</upload_file_number>
  <last_iter>1388</last_iter>
  <last_upload>1209600</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a4j1
(argv4) batchid: 991
(argv5) wuid: 12212656
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c09485d514c24da722b14c8df1ec101d to: /var/lib/boinc-client/slots/7/oifs_43r3_bl_a4j1_2016092300_15_991_12212656.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_bl_a4j1_2016092300_15_991_12212656.zip
ic_ancil_file: ic_ancil_12212656
ifsdata_file: ifsdata_12212656
climate_data_file: clim_data_12212656
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/7/ic_ancil_12212656.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12212656.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_597e49dca3a4964398b33c5f1c8bfd98 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12212656.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12212656.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12212656.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12212656.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12212656.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12212656.xml
-- Model is restarting --
Adjusting last_iter, 1388, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12212656.xml
last_cpu_time: 39209
upload_file_number: 14
last_iter: 1345
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a4j1_2016092300_15_991_12212656_0_r2063574814
The child process has been launched with process id: 5805
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:naturerick:1:1:5805:5805] [20230215:231522:1676531722:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:15:56 STEP 1344 H= 336:00 +CPU= 26.067
  23:16:20 STEP 1345 H= 336:15 +CPU= 24.145
  23:16:45 STEP 1346 H= 336:30 +CPU= 24.801
  23:17:10 STEP 1347 H= 336:45 +CPU= 24.602
  23:17:35 STEP 1348 H= 337:00 +CPU= 25.146
  23:18:00 STEP 1349 H= 337:15 +CPU= 24.622
  23:18:24 STEP 1350 H= 337:30 +CPU= 24.502
  23:18:49 STEP 1351 H= 337:45 +CPU= 24.202
  23:19:14 STEP 1352 H= 338:00 +CPU= 25.461
  23:19:39 STEP 1353 H= 338:15 +CPU= 24.385
  23:20:03 STEP 1354 H= 338:30 +CPU= 24.168
  23:20:28 STEP 1355 H= 338:45 +CPU= 24.469
  23:20:58 STEP 1356 H= 339:00 +CPU= 30.086
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001356
  23:21:22 STEP 1357 H= 339:15 +CPU= 24.179
  23:21:47 STEP 1358 H= 339:30 +CPU= 24.821
  23:22:12 STEP 1359 H= 339:45 +CPU= 24.355
  23:22:37 STEP 1360 H= 340:00 +CPU= 25.012
  23:23:02 STEP 1361 H= 340:15 +CPU= 24.541
  23:23:27 STEP 1362 H= 340:30 +CPU= 24.690
  23:23:51 STEP 1363 H= 340:45 +CPU= 24.498
  23:24:16 STEP 1364 H= 341:00 +CPU= 24.967
  23:24:41 STEP 1365 H= 341:15 +CPU= 24.451
  23:25:06 STEP 1366 H= 341:30 +CPU= 24.609
  23:25:31 STEP 1367 H= 341:45 +CPU= 24.836
  23:26:02 STEP 1368 H= 342:00 +CPU= 30.728
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001368
  23:26:27 STEP 1369 H= 342:15 +CPU= 24.800
  23:26:51 STEP 1370 H= 342:30 +CPU= 24.589
  23:27:16 STEP 1371 H= 342:45 +CPU= 24.365
  23:27:41 STEP 1372 H= 343:00 +CPU= 25.234
  23:28:06 STEP 1373 H= 343:15 +CPU= 24.363
  23:28:30 STEP 1374 H= 343:30 +CPU= 24.588
  23:28:55 STEP 1375 H= 343:45 +CPU= 24.505
  23:29:20 STEP 1376 H= 344:00 +CPU= 24.904
  23:29:45 STEP 1377 H= 344:15 +CPU= 24.894
  23:30:10 STEP 1378 H= 344:30 +CPU= 24.226
  23:30:34 STEP 1379 H= 344:45 +CPU= 24.257
  23:31:05 STEP 1380 H= 345:00 +CPU= 30.453
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001380
  23:31:29 STEP 1381 H= 345:15 +CPU= 24.484
  23:31:54 STEP 1382 H= 345:30 +CPU= 24.470
  23:32:19 STEP 1383 H= 345:45 +CPU= 24.912
  23:32:44 STEP 1384 H= 346:00 +CPU= 24.883
  23:33:09 STEP 1385 H= 346:15 +CPU= 24.637
  23:33:34 STEP 1386 H= 346:30 +CPU= 24.743
  23:33:58 STEP 1387 H= 346:45 +CPU= 24.511
  23:34:24 STEP 1388 H= 347:00 +CPU= 25.196
  23:34:49 STEP 1389 H= 347:15 +CPU= 25.166
  23:35:13 STEP 1390 H= 347:30 +CPU= 23.689
  23:35:37 STEP 1391 H= 347:45 +CPU= 24.318
  23:36:07 STEP 1392 H= 348:00 +CPU= 29.480
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001392
  23:36:28 STEP 1393 H= 348:15 +CPU= 21.407
  23:36:50 STEP 1394 H= 348:30 +CPU= 21.861
  23:37:11 STEP 1395 H= 348:45 +CPU= 20.648
  23:37:33 STEP 1396 H= 349:00 +CPU= 21.323
  23:37:54 STEP 1397 H= 349:15 +CPU= 21.080
  23:38:15 STEP 1398 H= 349:30 +CPU= 20.720
  23:38:34 STEP 1399 H= 349:45 +CPU= 19.788
  23:38:55 STEP 1400 H= 350:00 +CPU= 20.742
  23:39:16 STEP 1401 H= 350:15 +CPU= 20.306
  23:39:36 STEP 1402 H= 350:30 +CPU= 20.534
  23:39:57 STEP 1403 H= 350:45 +CPU= 20.537
  23:40:23 STEP 1404 H= 351:00 +CPU= 25.647
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001404
  23:40:44 STEP 1405 H= 351:15 +CPU= 20.708
  23:41:04 STEP 1406 H= 351:30 +CPU= 20.182
  23:41:25 STEP 1407 H= 351:45 +CPU= 20.522
  23:41:46 STEP 1408 H= 352:00 +CPU= 21.201
  23:42:06 STEP 1409 H= 352:15 +CPU= 20.385
  23:42:27 STEP 1410 H= 352:30 +CPU= 20.551
  23:42:48 STEP 1411 H= 352:45 +CPU= 21.076
  23:43:09 STEP 1412 H= 353:00 +CPU= 20.748
  23:43:30 STEP 1413 H= 353:15 +CPU= 21.003
  23:43:51 STEP 1414 H= 353:30 +CPU= 20.455
  23:44:12 STEP 1415 H= 353:45 +CPU= 20.666
  23:44:37 STEP 1416 H= 354:00 +CPU= 25.629
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001416
  23:44:57 STEP 1417 H= 354:15 +CPU= 20.039
  23:45:18 STEP 1418 H= 354:30 +CPU= 20.798
  23:45:39 STEP 1419 H= 354:45 +CPU= 20.606
  23:46:00 STEP 1420 H= 355:00 +CPU= 21.045
  23:46:21 STEP 1421 H= 355:15 +CPU= 20.629
  23:46:41 STEP 1422 H= 355:30 +CPU= 20.045
  23:47:02 STEP 1423 H= 355:45 +CPU= 20.587
  23:47:24 STEP 1424 H= 356:00 +CPU= 21.615
  23:47:44 STEP 1425 H= 356:15 +CPU= 20.720
  23:48:05 STEP 1426 H= 356:30 +CPU= 20.962
  23:48:26 STEP 1427 H= 356:45 +CPU= 20.815
  23:48:52 STEP 1428 H= 357:00 +CPU= 25.124
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001428
  23:49:12 STEP 1429 H= 357:15 +CPU= 20.360
  23:49:33 STEP 1430 H= 357:30 +CPU= 20.536
  23:49:54 STEP 1431 H= 357:45 +CPU= 20.934
  23:50:14 STEP 1432 H= 358:00 +CPU= 20.073
  23:50:32 STEP 1433 H= 358:15 +CPU= 18.200
  23:50:50 STEP 1434 H= 358:30 +CPU= 17.863
  23:51:09 STEP 1435 H= 358:45 +CPU= 18.269
  23:51:27 STEP 1436 H= 359:00 +CPU= 18.370
  23:51:45 STEP 1437 H= 359:15 +CPU= 17.793
  23:52:04 STEP 1438 H= 359:30 +CPU= 18.844
  23:52:23 STEP 1439 H= 359:45 +CPU= 19.063
  23:52:46 STEP 1440 H= 360:00 +CPU= 22.533
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1276843850E-20      I       0.1499767596E-04      I
ICloud fractionI      0.0000000000E+00      I       0.8698836877E-01      I
I  Relat. Hum. I      0.2140346345E+00      I       0.3622219916E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1201501303E+04      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    309327553 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0060( 5.0%)
 23:52:46 STEP 1440 H= 360:00 +CPU= 22.533
 END CNT3
 NSTEP =  1440 CNT0   000000000  16.487  16.487  16.570   0.995               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN         2836/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     53    0       34      1044068     2836 ************     9.4     5.3     4.1     6.2     8.6
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 2243583.1 2243583.1    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 2236216.9 2236216.9     99.67      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  105    3824.1    3824.1     17.90      0.00
   9 SPCM     - SPECTRAL COMP.                      96     639.8     639.8      2.74      0.00
  10 SCAN2M - PHYSICS                               97   10300.8   10300.8     44.53      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  8     733.1     733.1      0.26      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  5      23.4      23.4      0.01      0.00
  17 GRIDFPOS IN CNT4                                8      21.9      21.9      0.01      0.00
  27 RESTART FILES - WRITING                         2    2932.6    2932.6      0.26      0.00
  28 RESTART FILES - READING                         1    6888.2    6888.2      0.31      0.00
  29 SU4FPOS IN CNT4                                 8       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                 8    4673.3    4673.3      1.67      0.00
  31 POSDDH IN STEPO                                 1       1.3       1.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     97    5171.8    5171.8     22.36      0.00
  39 SU0YOMB                                         1     318.1     318.1      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                     97      21.5      21.5      0.09      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               96     344.4     344.4      1.47      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               96      71.4      71.4      0.31      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  5       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      212     369.3     369.3      3.49      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       200     218.9     218.9      1.95      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        200      80.2      80.2      0.71      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       212     114.5     114.5      1.08      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      35.1      35.1      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            212      57.7      57.7      0.55      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            200      28.6      28.6      0.25      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            212      36.7      36.7      0.35      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            200      17.8      17.8      0.16      0.00
 400 GSTATS                                      12155       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 11263       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     2243.6 CPU TIME    2227.1 VECTOR TIME     2227.1


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      579.6
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/7/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/7/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12212656/ICMSHh7zg+001356
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4j1_2016092300_15_991_12212656_0_r2063574814_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
23:56:16 (5788): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:52 1506016 22312924 oifs_43r3_bl_a4j1_2016092300_15_991_12212656_0 1,295,100 41,436 0.0320


©2024 climateprediction.net