climateprediction.net home page
Task 22309067

Task 22309067

Name oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0
Workunit 12208887
Created 13 Feb 2023, 15:52:42 UTC
Sent 13 Feb 2023, 16:08:02 UTC
Report deadline 14 Apr 2023, 16:08:02 UTC
Received 15 Feb 2023, 2:32:48 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1509857
Run time 11 hours 59 min 40 sec
CPU time 11 hours 59 min 40 sec
Validate state Valid
Credit 1,813.14
Device peak FLOPS 5.50 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,069.94 MB
Peak swap size 5,930.79 MB
Peak disk usage 2,126.20 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
fs_43r3_bl_12208887/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0_r75061972_10.zip
Uploading the intermediate file: upload_file_10.zip
  16:08:23 STEP 1057 H= 264:15 +CPU= 28.055
  16:08:48 STEP 1058 H= 264:30 +CPU= 25.369
  16:09:19 STEP 1059 H= 264:45 +CPU= 30.362
  16:09:48 STEP 1060 H= 265:00 +CPU= 28.833
  16:10:13 STEP 1061 H= 265:15 +CPU= 25.061
  16:10:45 STEP 1062 H= 265:30 +CPU= 31.521
  16:11:15 STEP 1063 H= 265:45 +CPU= 30.778
  16:11:47 STEP 1064 H= 266:00 +CPU= 31.560
  16:12:19 STEP 1065 H= 266:15 +CPU= 31.669
  16:12:50 STEP 1066 H= 266:30 +CPU= 31.196
  16:13:19 STEP 1067 H= 266:45 +CPU= 28.511
  16:13:57 STEP 1068 H= 267:00 +CPU= 37.926
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001068
  16:14:23 STEP 1069 H= 267:15 +CPU= 25.616
  16:14:49 STEP 1070 H= 267:30 +CPU= 25.811
  16:15:19 STEP 1071 H= 267:45 +CPU= 30.795
  16:15:48 STEP 1072 H= 268:00 +CPU= 28.590
  16:16:18 STEP 1073 H= 268:15 +CPU= 29.916
  16:16:50 STEP 1074 H= 268:30 +CPU= 31.745
  16:17:22 STEP 1075 H= 268:45 +CPU= 31.539
  16:17:52 STEP 1076 H= 269:00 +CPU= 30.073
  16:18:17 STEP 1077 H= 269:15 +CPU= 25.310
  16:18:43 STEP 1078 H= 269:30 +CPU= 25.649
  16:19:12 STEP 1079 H= 269:45 +CPU= 28.379
  16:19:43 STEP 1080 H= 270:00 +CPU= 31.553
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001080
  16:20:09 STEP 1081 H= 270:15 +CPU= 25.612
  16:20:37 STEP 1082 H= 270:30 +CPU= 27.660
  16:21:05 STEP 1083 H= 270:45 +CPU= 28.436
  16:21:35 STEP 1084 H= 271:00 +CPU= 29.631
  16:22:05 STEP 1085 H= 271:15 +CPU= 29.116
  16:22:32 STEP 1086 H= 271:30 +CPU= 26.815
  16:22:59 STEP 1087 H= 271:45 +CPU= 27.264
  16:23:28 STEP 1088 H= 272:00 +CPU= 28.633
  16:23:56 STEP 1089 H= 272:15 +CPU= 27.854
  16:24:22 STEP 1090 H= 272:30 +CPU= 25.862
  16:24:47 STEP 1091 H= 272:45 +CPU= 25.341
  16:25:19 STEP 1092 H= 273:00 +CPU= 31.730
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001092
  16:25:46 STEP 1093 H= 273:15 +CPU= 27.523
  16:26:16 STEP 1094 H= 273:30 +CPU= 29.515
  16:26:42 STEP 1095 H= 273:45 +CPU= 25.729
  16:27:08 STEP 1096 H= 274:00 +CPU= 25.820
  16:27:35 STEP 1097 H= 274:15 +CPU= 26.812
  16:28:05 STEP 1098 H= 274:30 +CPU= 30.287
  16:28:32 STEP 1099 H= 274:45 +CPU= 27.057
  16:29:03 STEP 1100 H= 275:00 +CPU= 30.227
  16:29:29 STEP 1101 H= 275:15 +CPU= 26.767
  16:30:00 STEP 1102 H= 275:30 +CPU= 30.472
  16:30:28 STEP 1103 H= 275:45 +CPU= 26.821
  16:31:06 STEP 1104 H= 276:00 +CPU= 38.019
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001104
  16:31:32 STEP 1105 H= 276:15 +CPU= 26.119
  16:31:58 STEP 1106 H= 276:30 +CPU= 25.321
  16:32:24 STEP 1107 H= 276:45 +CPU= 26.097
  16:32:50 STEP 1108 H= 277:00 +CPU= 26.045
  16:33:19 STEP 1109 H= 277:15 +CPU= 28.423
  16:33:46 STEP 1110 H= 277:30 +CPU= 26.925
  16:34:14 STEP 1111 H= 277:45 +CPU= 28.707
  16:34:43 STEP 1112 H= 278:00 +CPU= 28.124
  16:35:10 STEP 1113 H= 278:15 +CPU= 27.165
  16:35:39 STEP 1114 H= 278:30 +CPU= 29.150
  16:36:08 STEP 1115 H= 278:45 +CPU= 28.320
  16:36:41 STEP 1116 H= 279:00 +CPU= 33.804
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001116
  16:37:08 STEP 1117 H= 279:15 +CPU= 26.025
  16:37:33 STEP 1118 H= 279:30 +CPU= 25.384
  16:38:01 STEP 1119 H= 279:45 +CPU= 27.624
  16:38:29 STEP 1120 H= 280:00 +CPU= 28.061
  16:38:55 STEP 1121 H= 280:15 +CPU= 25.367
  16:39:22 STEP 1122 H= 280:30 +CPU= 27.486
  16:39:53 STEP 1123 H= 280:45 +CPU= 30.734
  16:40:22 STEP 1124 H= 281:00 +CPU= 29.037
  16:40:51 STEP 1125 H= 281:15 +CPU= 28.370
  16:41:20 STEP 1126 H= 281:30 +CPU= 28.889
  16:41:46 STEP 1127 H= 281:45 +CPU= 26.119
  16:42:23 STEP 1128 H= 282:00 +CPU= 36.933
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001128
  16:42:51 STEP 1129 H= 282:15 +CPU= 27.986
  16:43:19 STEP 1130 H= 282:30 +CPU= 27.407
  16:43:47 STEP 1131 H= 282:45 +CPU= 28.109
  16:44:14 STEP 1132 H= 283:00 +CPU= 26.365
  16:44:40 STEP 1133 H= 283:15 +CPU= 26.464
  16:45:09 STEP 1134 H= 283:30 +CPU= 29.026
  16:45:38 STEP 1135 H= 283:45 +CPU= 28.792
  16:46:04 STEP 1136 H= 284:00 +CPU= 25.632
  16:46:29 STEP 1137 H= 284:15 +CPU= 24.979
  16:46:54 STEP 1138 H= 284:30 +CPU= 25.119
  16:47:25 STEP 1139 H= 284:45 +CPU= 30.326
  16:47:56 STEP 1140 H= 285:00 +CPU= 31.605
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001140
  16:48:27 STEP 1141 H= 285:15 +CPU= 30.745
  16:48:53 STEP 1142 H= 285:30 +CPU= 25.816
  16:49:19 STEP 1143 H= 285:45 +CPU= 25.744
  16:49:48 STEP 1144 H= 286:00 +CPU= 28.723
  16:50:15 STEP 1145 H= 286:15 +CPU= 26.753
  16:50:40 STEP 1146 H= 286:30 +CPU= 25.346
  16:51:10 STEP 1147 H= 286:45 +CPU= 30.074
  16:51:43 STEP 1148 H= 287:00 +CPU= 32.420
  16:52:10 STEP 1149 H= 287:15 +CPU= 26.743
  16:52:37 STEP 1150 H= 287:30 +CPU= 27.670
  16:53:05 STEP 1151 H= 287:45 +CPU= 27.599
  16:53:47 STEP 1152 H= 288:00 +CPU= 41.819
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0_r75061972_11.zip
Uploading the intermediate file: upload_file_11.zip
  16:54:19 STEP 1153 H= 288:15 +CPU= 29.827
  16:54:47 STEP 1154 H= 288:30 +CPU= 28.070
  16:55:15 STEP 1155 H= 288:45 +CPU= 27.941
  16:55:45 STEP 1156 H= 289:00 +CPU= 30.561
  16:56:14 STEP 1157 H= 289:15 +CPU= 28.228
  16:56:44 STEP 1158 H= 289:30 +CPU= 29.607
  16:57:08 STEP 1159 H= 289:45 +CPU= 24.715
  16:57:34 STEP 1160 H= 290:00 +CPU= 25.550
  16:58:04 STEP 1161 H= 290:15 +CPU= 29.469
  16:58:29 STEP 1162 H= 290:30 +CPU= 25.623
  16:58:54 STEP 1163 H= 290:45 +CPU= 25.024
  16:59:28 STEP 1164 H= 291:00 +CPU= 33.342
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001164
  16:59:55 STEP 1165 H= 291:15 +CPU= 27.276
  17:00:24 STEP 1166 H= 291:30 +CPU= 28.359
  17:00:51 STEP 1167 H= 291:45 +CPU= 27.429
  17:01:19 STEP 1168 H= 292:00 +CPU= 27.871
  17:01:46 STEP 1169 H= 292:15 +CPU= 26.760
  17:02:16 STEP 1170 H= 292:30 +CPU= 29.897
  17:02:47 STEP 1171 H= 292:45 +CPU= 30.320
  17:03:14 STEP 1172 H= 293:00 +CPU= 27.552
  17:03:43 STEP 1173 H= 293:15 +CPU= 28.817
  17:04:10 STEP 1174 H= 293:30 +CPU= 26.323
  17:04:35 STEP 1175 H= 293:45 +CPU= 25.577
  17:05:07 STEP 1176 H= 294:00 +CPU= 31.604
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001176
  17:05:36 STEP 1177 H= 294:15 +CPU= 29.000
  17:06:08 STEP 1178 H= 294:30 +CPU= 31.883
  17:06:34 STEP 1179 H= 294:45 +CPU= 25.522
  17:07:00 STEP 1180 H= 295:00 +CPU= 25.732
  17:07:25 STEP 1181 H= 295:15 +CPU= 25.521
  17:07:53 STEP 1182 H= 295:30 +CPU= 27.686
  17:08:22 STEP 1183 H= 295:45 +CPU= 28.617
  17:08:48 STEP 1184 H= 296:00 +CPU= 25.621
  17:09:16 STEP 1185 H= 296:15 +CPU= 28.184
  17:09:42 STEP 1186 H= 296:30 +CPU= 25.866
  17:10:09 STEP 1187 H= 296:45 +CPU= 26.930
  17:10:44 STEP 1188 H= 297:00 +CPU= 34.825
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001188
  17:11:13 STEP 1189 H= 297:15 +CPU= 29.013
  17:11:43 STEP 1190 H= 297:30 +CPU= 29.282
  17:12:11 STEP 1191 H= 297:45 +CPU= 27.630
  17:12:42 STEP 1192 H= 298:00 +CPU= 29.611
  17:13:13 STEP 1193 H= 298:15 +CPU= 30.536
  17:13:43 STEP 1194 H= 298:30 +CPU= 30.183
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5702MB (heap), 5076MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil))
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 5702 MB (maxheap), 5076 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171407:1676391247:35169.410] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171408:1676391248:35170.411] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:bart-MintII:1:1:339163:339163] [20230214:171408:1676391248:35170.411] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  000000000081FCF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8BB20  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D5BB2D  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001060EB2  larmes_                   543  larmes.F90
oifs_43r3_model.e  0000000001058F5D  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000FF4E6C  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000001081A7D  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000E67648  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000015C1E62  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  00000000015AC40E  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  00000000006620B4  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004A0E7C  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  000000000049DC29  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  000000000049CE72  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049CD68  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  00000000004078AD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040276F  MAIN__                     96  master.F90
oifs_43r3_model.e  000000000040270D  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D8287A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D840D7  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000004025E5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1mc
(argv4) batchid: 991
(argv5) wuid: 12208887
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2413ea032cd9d90a7bc0876470774359 to: /var/lib/boinc-client/slots/9/oifs_43r3_bl_a1mc_2016092300_15_991_12208887.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_bl_a1mc_2016092300_15_991_12208887.zip
ic_ancil_file: ic_ancil_12208887
ifsdata_file: ifsdata_12208887
climate_data_file: clim_data_12208887
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/9/ic_ancil_12208887.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12208887.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_75cbf4537bbf87005baa0ec4999ac5ae to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12208887.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12208887.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12208887.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12208887.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12208887.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12208887.xml
-- Model is restarting --
Adjusting last_iter, 1195, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12208887.xml
last_cpu_time: 34960
upload_file_number: 12
last_iter: 1153
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0_r75061972
The child process has been launched with process id: 1695
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:bart-MintII:1:1:1695:1695] [20230214:171512:1676391312:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:15:43 STEP 1152 H= 288:00 +CPU= 25.581
  17:16:11 STEP 1153 H= 288:15 +CPU= 25.059
  17:16:39 STEP 1154 H= 288:30 +CPU= 28.180
  17:17:06 STEP 1155 H= 288:45 +CPU= 27.103
  17:17:36 STEP 1156 H= 289:00 +CPU= 29.351
  17:18:06 STEP 1157 H= 289:15 +CPU= 28.409
  17:18:37 STEP 1158 H= 289:30 +CPU= 28.102
  17:19:10 STEP 1159 H= 289:45 +CPU= 28.471
  17:19:41 STEP 1160 H= 290:00 +CPU= 28.784
  17:20:09 STEP 1161 H= 290:15 +CPU= 27.391
  17:20:39 STEP 1162 H= 290:30 +CPU= 28.919
  17:21:09 STEP 1163 H= 290:45 +CPU= 28.353
  17:21:44 STEP 1164 H= 291:00 +CPU= 34.144
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001164
  17:22:11 STEP 1165 H= 291:15 +CPU= 27.208
  17:22:42 STEP 1166 H= 291:30 +CPU= 28.510
  17:23:11 STEP 1167 H= 291:45 +CPU= 28.056
  17:23:41 STEP 1168 H= 292:00 +CPU= 29.507
  17:24:10 STEP 1169 H= 292:15 +CPU= 27.809
  17:24:39 STEP 1170 H= 292:30 +CPU= 28.457
  17:25:08 STEP 1171 H= 292:45 +CPU= 28.793
  17:25:39 STEP 1172 H= 293:00 +CPU= 30.141
  17:26:07 STEP 1173 H= 293:15 +CPU= 26.993
  17:26:36 STEP 1174 H= 293:30 +CPU= 28.377
  17:27:04 STEP 1175 H= 293:45 +CPU= 27.769
  17:27:41 STEP 1176 H= 294:00 +CPU= 35.244
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001176
  17:28:09 STEP 1177 H= 294:15 +CPU= 27.998
  17:28:37 STEP 1178 H= 294:30 +CPU= 28.069
  17:29:04 STEP 1179 H= 294:45 +CPU= 27.016
  17:29:34 STEP 1180 H= 295:00 +CPU= 29.444
  17:30:03 STEP 1181 H= 295:15 +CPU= 28.388
  17:30:29 STEP 1182 H= 295:30 +CPU= 26.310
  17:30:59 STEP 1183 H= 295:45 +CPU= 28.309
  17:31:28 STEP 1184 H= 296:00 +CPU= 29.244
  17:31:57 STEP 1185 H= 296:15 +CPU= 28.146
  17:32:24 STEP 1186 H= 296:30 +CPU= 27.135
  17:32:54 STEP 1187 H= 296:45 +CPU= 28.729
  17:33:29 STEP 1188 H= 297:00 +CPU= 34.498
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001188
  17:33:59 STEP 1189 H= 297:15 +CPU= 29.417
  17:34:28 STEP 1190 H= 297:30 +CPU= 28.191
  17:34:57 STEP 1191 H= 297:45 +CPU= 28.559
  17:35:26 STEP 1192 H= 298:00 +CPU= 27.926
  17:35:54 STEP 1193 H= 298:15 +CPU= 27.707
  17:36:24 STEP 1194 H= 298:30 +CPU= 29.297
  17:36:52 STEP 1195 H= 298:45 +CPU= 27.687
  17:37:20 STEP 1196 H= 299:00 +CPU= 26.867
  17:37:47 STEP 1197 H= 299:15 +CPU= 27.396
  17:38:15 STEP 1198 H= 299:30 +CPU= 27.141
  17:38:45 STEP 1199 H= 299:45 +CPU= 29.246
  17:39:27 STEP 1200 H= 300:00 +CPU= 40.093
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001200
  17:39:56 STEP 1201 H= 300:15 +CPU= 28.389
  17:40:23 STEP 1202 H= 300:30 +CPU= 26.453
  17:40:52 STEP 1203 H= 300:45 +CPU= 28.238
  17:41:24 STEP 1204 H= 301:00 +CPU= 30.851
  17:41:51 STEP 1205 H= 301:15 +CPU= 26.912
  17:42:21 STEP 1206 H= 301:30 +CPU= 29.451
  17:42:49 STEP 1207 H= 301:45 +CPU= 27.441
  17:43:19 STEP 1208 H= 302:00 +CPU= 29.126
  17:43:48 STEP 1209 H= 302:15 +CPU= 28.124
  17:44:17 STEP 1210 H= 302:30 +CPU= 29.286
  17:44:45 STEP 1211 H= 302:45 +CPU= 26.689
  17:45:21 STEP 1212 H= 303:00 +CPU= 35.570
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001212
  17:45:49 STEP 1213 H= 303:15 +CPU= 27.461
  17:46:18 STEP 1214 H= 303:30 +CPU= 28.370
  17:46:49 STEP 1215 H= 303:45 +CPU= 29.918
  17:47:17 STEP 1216 H= 304:00 +CPU= 28.032
  17:47:46 STEP 1217 H= 304:15 +CPU= 27.313
  17:48:12 STEP 1218 H= 304:30 +CPU= 25.948
  17:48:43 STEP 1219 H= 304:45 +CPU= 30.118
  17:49:13 STEP 1220 H= 305:00 +CPU= 30.621
  17:49:42 STEP 1221 H= 305:15 +CPU= 28.366
  17:50:08 STEP 1222 H= 305:30 +CPU= 26.465
  17:50:37 STEP 1223 H= 305:45 +CPU= 28.022
  17:51:10 STEP 1224 H= 306:00 +CPU= 32.671
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001224
  17:51:37 STEP 1225 H= 306:15 +CPU= 27.075
  17:52:05 STEP 1226 H= 306:30 +CPU= 27.686
  17:52:32 STEP 1227 H= 306:45 +CPU= 26.771
  17:53:01 STEP 1228 H= 307:00 +CPU= 28.868
  17:53:33 STEP 1229 H= 307:15 +CPU= 31.982
  17:54:00 STEP 1230 H= 307:30 +CPU= 26.388
  17:54:25 STEP 1231 H= 307:45 +CPU= 25.264
  17:54:51 STEP 1232 H= 308:00 +CPU= 26.294
  17:55:20 STEP 1233 H= 308:15 +CPU= 28.147
  17:55:47 STEP 1234 H= 308:30 +CPU= 27.467
  17:56:15 STEP 1235 H= 308:45 +CPU= 27.238
  17:56:52 STEP 1236 H= 309:00 +CPU= 37.052
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001236
  17:57:20 STEP 1237 H= 309:15 +CPU= 27.783
  17:57:48 STEP 1238 H= 309:30 +CPU= 28.130
  17:58:16 STEP 1239 H= 309:45 +CPU= 27.942
  17:58:44 STEP 1240 H= 310:00 +CPU= 27.333
  17:59:12 STEP 1241 H= 310:15 +CPU= 28.086
  17:59:39 STEP 1242 H= 310:30 +CPU= 27.038
  18:00:07 STEP 1243 H= 310:45 +CPU= 27.492
  18:00:36 STEP 1244 H= 311:00 +CPU= 28.816
  18:01:04 STEP 1245 H= 311:15 +CPU= 27.872
  18:01:35 STEP 1246 H= 311:30 +CPU= 30.221
  18:02:04 STEP 1247 H= 311:45 +CPU= 28.334
  18:02:45 STEP 1248 H= 312:00 +CPU= 40.235
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0_r75061972_12.zip
Uploading the intermediate file: upload_file_12.zip
  18:03:14 STEP 1249 H= 312:15 +CPU= 28.062
  18:03:41 STEP 1250 H= 312:30 +CPU= 26.943
  18:04:10 STEP 1251 H= 312:45 +CPU= 29.460
  18:04:39 STEP 1252 H= 313:00 +CPU= 28.282
  18:05:06 STEP 1253 H= 313:15 +CPU= 27.010
  18:05:36 STEP 1254 H= 313:30 +CPU= 29.245
  18:06:04 STEP 1255 H= 313:45 +CPU= 27.810
  18:06:32 STEP 1256 H= 314:00 +CPU= 28.128
  18:07:01 STEP 1257 H= 314:15 +CPU= 28.991
  18:07:32 STEP 1258 H= 314:30 +CPU= 30.463
  18:08:02 STEP 1259 H= 314:45 +CPU= 29.742
  18:08:37 STEP 1260 H= 315:00 +CPU= 35.204
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001260
  18:09:06 STEP 1261 H= 315:15 +CPU= 28.483
  18:09:35 STEP 1262 H= 315:30 +CPU= 28.501
  18:10:03 STEP 1263 H= 315:45 +CPU= 28.507
  18:10:32 STEP 1264 H= 316:00 +CPU= 28.190
  18:11:03 STEP 1265 H= 316:15 +CPU= 30.698
  18:11:31 STEP 1266 H= 316:30 +CPU= 27.794
  18:12:00 STEP 1267 H= 316:45 +CPU= 29.006
  18:12:29 STEP 1268 H= 317:00 +CPU= 29.224
  18:12:58 STEP 1269 H= 317:15 +CPU= 28.122
  18:13:24 STEP 1270 H= 317:30 +CPU= 26.083
  18:13:52 STEP 1271 H= 317:45 +CPU= 27.443
  18:14:28 STEP 1272 H= 318:00 +CPU= 36.040
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001272
  18:14:56 STEP 1273 H= 318:15 +CPU= 28.492
  18:15:23 STEP 1274 H= 318:30 +CPU= 26.710
  18:15:51 STEP 1275 H= 318:45 +CPU= 27.590
  18:16:17 STEP 1276 H= 319:00 +CPU= 25.992
  18:16:45 STEP 1277 H= 319:15 +CPU= 27.289
  18:17:14 STEP 1278 H= 319:30 +CPU= 28.617
  18:17:41 STEP 1279 H= 319:45 +CPU= 27.644
  18:18:09 STEP 1280 H= 320:00 +CPU= 27.679
  18:18:37 STEP 1281 H= 320:15 +CPU= 27.612
  18:19:06 STEP 1282 H= 320:30 +CPU= 28.436
  18:19:32 STEP 1283 H= 320:45 +CPU= 26.330
  18:20:08 STEP 1284 H= 321:00 +CPU= 35.764
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001284
  18:20:40 STEP 1285 H= 321:15 +CPU= 31.634
  18:21:09 STEP 1286 H= 321:30 +CPU= 29.057
  18:21:37 STEP 1287 H= 321:45 +CPU= 27.503
  18:22:07 STEP 1288 H= 322:00 +CPU= 29.776
  18:22:36 STEP 1289 H= 322:15 +CPU= 28.608
  18:23:04 STEP 1290 H= 322:30 +CPU= 27.735
  18:23:32 STEP 1291 H= 322:45 +CPU= 27.967
  18:23:58 STEP 1292 H= 323:00 +CPU= 26.176
  18:24:25 STEP 1293 H= 323:15 +CPU= 26.879
  18:24:54 STEP 1294 H= 323:30 +CPU= 28.618
  18:25:20 STEP 1295 H= 323:45 +CPU= 26.185
  18:25:59 STEP 1296 H= 324:00 +CPU= 38.940
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001296
  18:26:29 STEP 1297 H= 324:15 +CPU= 29.217
  18:26:59 STEP 1298 H= 324:30 +CPU= 29.811
  18:27:27 STEP 1299 H= 324:45 +CPU= 27.386
  18:27:54 STEP 1300 H= 325:00 +CPU= 26.263
  18:28:23 STEP 1301 H= 325:15 +CPU= 28.951
  18:28:51 STEP 1302 H= 325:30 +CPU= 27.953
  18:29:19 STEP 1303 H= 325:45 +CPU= 27.780
  18:29:47 STEP 1304 H= 326:00 +CPU= 27.773
  18:30:14 STEP 1305 H= 326:15 +CPU= 26.858
  18:30:45 STEP 1306 H= 326:30 +CPU= 31.155
  18:31:13 STEP 1307 H= 326:45 +CPU= 27.932
  18:31:46 STEP 1308 H= 327:00 +CPU= 33.085
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001308
  18:32:15 STEP 1309 H= 327:15 +CPU= 28.386
  18:32:42 STEP 1310 H= 327:30 +CPU= 27.072
  18:33:08 STEP 1311 H= 327:45 +CPU= 25.798
  18:33:38 STEP 1312 H= 328:00 +CPU= 29.757
  18:34:04 STEP 1313 H= 328:15 +CPU= 25.912
  18:34:33 STEP 1314 H= 328:30 +CPU= 28.326
  18:35:00 STEP 1315 H= 328:45 +CPU= 27.059
  18:35:27 STEP 1316 H= 329:00 +CPU= 26.735
  18:35:52 STEP 1317 H= 329:15 +CPU= 25.557
  18:36:20 STEP 1318 H= 329:30 +CPU= 27.206
  18:36:48 STEP 1319 H= 329:45 +CPU= 27.953
  18:37:21 STEP 1320 H= 330:00 +CPU= 33.583
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001320
  18:37:53 STEP 1321 H= 330:15 +CPU= 31.512
  18:38:23 STEP 1322 H= 330:30 +CPU= 29.845
  18:38:52 STEP 1323 H= 330:45 +CPU= 29.060
  18:39:19 STEP 1324 H= 331:00 +CPU= 27.094
  18:39:46 STEP 1325 H= 331:15 +CPU= 26.279
  18:40:15 STEP 1326 H= 331:30 +CPU= 28.727
  18:40:42 STEP 1327 H= 331:45 +CPU= 27.688
  18:41:10 STEP 1328 H= 332:00 +CPU= 27.172
  18:41:39 STEP 1329 H= 332:15 +CPU= 29.079
  18:42:10 STEP 1330 H= 332:30 +CPU= 30.606
  18:42:37 STEP 1331 H= 332:45 +CPU= 27.332
  18:43:12 STEP 1332 H= 333:00 +CPU= 33.920
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001332
  18:43:41 STEP 1333 H= 333:15 +CPU= 29.694
  18:44:09 STEP 1334 H= 333:30 +CPU= 27.430
  18:44:39 STEP 1335 H= 333:45 +CPU= 29.538
  18:45:09 STEP 1336 H= 334:00 +CPU= 30.124
  18:45:40 STEP 1337 H= 334:15 +CPU= 31.233
  18:46:10 STEP 1338 H= 334:30 +CPU= 29.667
  18:46:41 STEP 1339 H= 334:45 +CPU= 30.876
  18:47:12 STEP 1340 H= 335:00 +CPU= 30.937
  18:47:42 STEP 1341 H= 335:15 +CPU= 29.362
  18:48:10 STEP 1342 H= 335:30 +CPU= 27.615
  18:48:38 STEP 1343 H= 335:45 +CPU= 28.605
  18:49:16 STEP 1344 H= 336:00 +CPU= 37.408
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0_r75061972_13.zip
Uploading the intermediate file: upload_file_13.zip
  18:49:43 STEP 1345 H= 336:15 +CPU= 26.176
  18:50:10 STEP 1346 H= 336:30 +CPU= 26.487
  18:50:38 STEP 1347 H= 336:45 +CPU= 28.081
  18:51:07 STEP 1348 H= 337:00 +CPU= 28.521
  18:51:36 STEP 1349 H= 337:15 +CPU= 28.243
  18:52:02 STEP 1350 H= 337:30 +CPU= 25.964
  18:52:28 STEP 1351 H= 337:45 +CPU= 25.842
  18:52:56 STEP 1352 H= 338:00 +CPU= 28.641
  18:53:24 STEP 1353 H= 338:15 +CPU= 27.302
  18:53:50 STEP 1354 H= 338:30 +CPU= 26.559
  18:54:20 STEP 1355 H= 338:45 +CPU= 28.871
  18:54:59 STEP 1356 H= 339:00 +CPU= 38.230
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001356
  18:55:27 STEP 1357 H= 339:15 +CPU= 27.270
  18:55:55 STEP 1358 H= 339:30 +CPU= 28.064
  18:56:23 STEP 1359 H= 339:45 +CPU= 27.907
  18:56:52 STEP 1360 H= 340:00 +CPU= 28.526
  18:57:17 STEP 1361 H= 340:15 +CPU= 25.813
  18:57:46 STEP 1362 H= 340:30 +CPU= 28.644
  18:58:15 STEP 1363 H= 340:45 +CPU= 29.021
  18:58:42 STEP 1364 H= 341:00 +CPU= 26.808
  18:59:13 STEP 1365 H= 341:15 +CPU= 30.721
  18:59:41 STEP 1366 H= 341:30 +CPU= 27.636
  19:00:09 STEP 1367 H= 341:45 +CPU= 28.148
  19:00:44 STEP 1368 H= 342:00 +CPU= 34.450
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001368
  19:01:12 STEP 1369 H= 342:15 +CPU= 27.958
  19:01:42 STEP 1370 H= 342:30 +CPU= 30.058
  19:02:13 STEP 1371 H= 342:45 +CPU= 30.673
  19:02:44 STEP 1372 H= 343:00 +CPU= 31.417
  19:03:16 STEP 1373 H= 343:15 +CPU= 31.859
  19:03:48 STEP 1374 H= 343:30 +CPU= 31.623
  19:04:17 STEP 1375 H= 343:45 +CPU= 29.123
  19:04:47 STEP 1376 H= 344:00 +CPU= 29.793
  19:05:13 STEP 1377 H= 344:15 +CPU= 25.738
  19:05:43 STEP 1378 H= 344:30 +CPU= 29.866
  19:06:12 STEP 1379 H= 344:45 +CPU= 28.339
  19:06:46 STEP 1380 H= 345:00 +CPU= 34.318
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001380
  19:07:12 STEP 1381 H= 345:15 +CPU= 25.548
  19:07:39 STEP 1382 H= 345:30 +CPU= 27.744
  19:08:06 STEP 1383 H= 345:45 +CPU= 26.599
  19:08:33 STEP 1384 H= 346:00 +CPU= 26.545
  19:09:03 STEP 1385 H= 346:15 +CPU= 29.418
  19:09:31 STEP 1386 H= 346:30 +CPU= 28.477
  19:09:57 STEP 1387 H= 346:45 +CPU= 25.876
  19:10:24 STEP 1388 H= 347:00 +CPU= 26.568
  19:10:51 STEP 1389 H= 347:15 +CPU= 26.696
  19:11:16 STEP 1390 H= 347:30 +CPU= 25.630
  19:11:45 STEP 1391 H= 347:45 +CPU= 28.118
  19:12:28 STEP 1392 H= 348:00 +CPU= 42.743
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001392
  19:12:56 STEP 1393 H= 348:15 +CPU= 28.401
  19:13:22 STEP 1394 H= 348:30 +CPU= 26.103
  19:13:48 STEP 1395 H= 348:45 +CPU= 25.646
  19:14:14 STEP 1396 H= 349:00 +CPU= 25.954
  19:14:38 STEP 1397 H= 349:15 +CPU= 24.148
  19:15:02 STEP 1398 H= 349:30 +CPU= 23.584
  19:15:26 STEP 1399 H= 349:45 +CPU= 23.968
  19:15:53 STEP 1400 H= 350:00 +CPU= 27.298
  19:16:18 STEP 1401 H= 350:15 +CPU= 25.082
  19:16:45 STEP 1402 H= 350:30 +CPU= 25.999
  19:17:09 STEP 1403 H= 350:45 +CPU= 24.490
  19:17:42 STEP 1404 H= 351:00 +CPU= 32.443
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001404
  19:18:08 STEP 1405 H= 351:15 +CPU= 26.287
  19:18:34 STEP 1406 H= 351:30 +CPU= 25.953
  19:19:03 STEP 1407 H= 351:45 +CPU= 28.741
  19:19:30 STEP 1408 H= 352:00 +CPU= 26.847
  19:19:56 STEP 1409 H= 352:15 +CPU= 25.949
  19:20:20 STEP 1410 H= 352:30 +CPU= 24.314
  19:20:46 STEP 1411 H= 352:45 +CPU= 25.664
  19:21:13 STEP 1412 H= 353:00 +CPU= 26.341
  19:21:38 STEP 1413 H= 353:15 +CPU= 24.993
  19:22:03 STEP 1414 H= 353:30 +CPU= 25.155
  19:22:30 STEP 1415 H= 353:45 +CPU= 27.157
  19:23:06 STEP 1416 H= 354:00 +CPU= 35.343
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001416
  19:23:30 STEP 1417 H= 354:15 +CPU= 24.653
  19:23:55 STEP 1418 H= 354:30 +CPU= 24.470
  19:24:21 STEP 1419 H= 354:45 +CPU= 26.181
  19:24:47 STEP 1420 H= 355:00 +CPU= 25.026
  19:25:11 STEP 1421 H= 355:15 +CPU= 24.556
  19:25:38 STEP 1422 H= 355:30 +CPU= 26.125
  19:26:03 STEP 1423 H= 355:45 +CPU= 25.050
  19:26:28 STEP 1424 H= 356:00 +CPU= 25.181
  19:26:53 STEP 1425 H= 356:15 +CPU= 24.642
  19:27:20 STEP 1426 H= 356:30 +CPU= 26.611
  19:27:46 STEP 1427 H= 356:45 +CPU= 26.251
  19:28:18 STEP 1428 H= 357:00 +CPU= 31.586
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001428
  19:28:43 STEP 1429 H= 357:15 +CPU= 25.307
  19:29:08 STEP 1430 H= 357:30 +CPU= 24.770
  19:29:35 STEP 1431 H= 357:45 +CPU= 26.569
  19:30:01 STEP 1432 H= 358:00 +CPU= 25.802
  19:30:27 STEP 1433 H= 358:15 +CPU= 26.477
  19:30:53 STEP 1434 H= 358:30 +CPU= 25.444
  19:31:22 STEP 1435 H= 358:45 +CPU= 28.316
  19:31:47 STEP 1436 H= 359:00 +CPU= 25.296
  19:32:11 STEP 1437 H= 359:15 +CPU= 24.459
  19:32:40 STEP 1438 H= 359:30 +CPU= 28.406
  19:33:11 STEP 1439 H= 359:45 +CPU= 30.814
  19:33:48 STEP 1440 H= 360:00 +CPU= 36.738
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.4025085285E-21      I       0.3535544204E-06      I
ICloud fractionI      0.0000000000E+00      I       0.2118107371E-01      I
I  Relat. Hum. I      0.1236013923E+00      I       0.1668769035E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1177449919E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    319985441 BYTES IN      7 RECORDS TRANSFERRED IN    0.0005 SECONDS639970.9 Mbytes/s, TOTAL TIME=    0.0125( 4.0%)
 19:33:48 STEP 1440 H= 360:00 +CPU= 36.738
 END CNT3
 NSTEP =  1440 CNT0   000000000  25.487  25.487  25.598   0.996               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN         8508/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    153    0       34      1044068     8508 ************    20.8     2.1    18.7     2.9    36.3
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 8316251.8 8316251.8    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 8312058.3 8312058.3     99.95      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  313    4816.2    4816.2     18.13      0.00
   9 SPCM     - SPECTRAL COMP.                     288     776.5     776.5      2.69      0.00
  10 SCAN2M - PHYSICS                              289   12814.2   12814.2     44.53      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 24    1085.0    1085.0      0.31      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 13      28.3      28.3      0.00      0.00
  17 GRIDFPOS IN CNT4                               24      32.3      32.3      0.01      0.00
  27 RESTART FILES - WRITING                         6    4337.6    4337.6      0.31      0.00
  28 RESTART FILES - READING                         1    3578.4    3578.4      0.04      0.00
  29 SU4FPOS IN CNT4                                24       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                24    6492.1    6492.1      1.87      0.00
  31 POSDDH IN STEPO                                 2      14.3      14.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    289    6392.1    6392.1     22.21      0.00
  39 SU0YOMB                                         1     394.3     394.3      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    289      23.0      23.0      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              288     327.3     327.3      1.13      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              288      79.0      79.0      0.27      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 13       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      628     492.4     492.4      3.72      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       600     290.6     290.6      2.10      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        600     108.9     108.9      0.79      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       628     149.0     149.0      1.12      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      35.6      35.6      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            628      71.7      71.7      0.54      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            600      34.6      34.6      0.25      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            628      44.6      44.6      0.34      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            600      21.8      21.8      0.16      0.00
 400 GSTATS                                      34304       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 31868       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     8316.3 CPU TIME    8220.7 VECTOR TIME     8220.7


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      155.9
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/9/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/9/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208887/ICMUAh7zg+001344
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0_r75061972_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
19:37:21 (1638): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:22 1509857 22309067 oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0 1,295,100 43,180 0.0333
09 May 2023 16:51:21 1509857 22309067 oifs_43r3_bl_a1mc_2016092300_15_991_12208887_0 864,000 28,290 0.0327


©2024 climateprediction.net