climateprediction.net home page
Task 22436268

Task 22436268

Name oifs_43r3_bl_a279_2016092300_20_1017_12284680_0
Workunit 12284680
Created 7 Jun 2024, 14:58:12 UTC
Sent 7 Jun 2024, 15:02:55 UTC
Report deadline 6 Aug 2024, 15:02:55 UTC
Received 9 Jun 2024, 6:21:54 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1491072
Run time 14 hours 42 min 54 sec
CPU time 14 hours 42 min 54 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 4.69 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,260.64 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,348.20 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<stderr_txt>
EP 1031 H= 257:45 +CPU= 32.470
  01:32:39 STEP 1032 H= 258:00 +CPU= 40.874
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001032
  01:33:12 STEP 1033 H= 258:15 +CPU= 32.450
  01:33:45 STEP 1034 H= 258:30 +CPU= 32.706
  01:34:19 STEP 1035 H= 258:45 +CPU= 33.701
  01:34:54 STEP 1036 H= 259:00 +CPU= 34.281
  01:35:27 STEP 1037 H= 259:15 +CPU= 32.668
  01:35:59 STEP 1038 H= 259:30 +CPU= 32.661
  01:36:32 STEP 1039 H= 259:45 +CPU= 32.545
  01:37:06 STEP 1040 H= 260:00 +CPU= 33.622
  01:37:39 STEP 1041 H= 260:15 +CPU= 32.574
  01:38:12 STEP 1042 H= 260:30 +CPU= 32.544
  01:38:45 STEP 1043 H= 260:45 +CPU= 32.587
  01:39:26 STEP 1044 H= 261:00 +CPU= 40.810
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001044
  01:39:59 STEP 1045 H= 261:15 +CPU= 32.331
  01:40:32 STEP 1046 H= 261:30 +CPU= 32.726
  01:41:05 STEP 1047 H= 261:45 +CPU= 32.695
  01:41:39 STEP 1048 H= 262:00 +CPU= 33.505
  01:42:11 STEP 1049 H= 262:15 +CPU= 32.382
  01:42:44 STEP 1050 H= 262:30 +CPU= 32.422
  01:43:17 STEP 1051 H= 262:45 +CPU= 32.466
  01:43:51 STEP 1052 H= 263:00 +CPU= 33.520
  01:44:23 STEP 1053 H= 263:15 +CPU= 32.411
  01:44:56 STEP 1054 H= 263:30 +CPU= 32.619
  01:45:29 STEP 1055 H= 263:45 +CPU= 32.386
  01:46:13 STEP 1056 H= 264:00 +CPU= 43.370
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001056
  01:46:45 STEP 1057 H= 264:15 +CPU= 32.059
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001056
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_10.zip
Uploading the intermediate file: upload_file_10.zip
  01:47:18 STEP 1058 H= 264:30 +CPU= 31.690
  01:47:51 STEP 1059 H= 264:45 +CPU= 32.724
  01:48:25 STEP 1060 H= 265:00 +CPU= 33.337
  01:48:58 STEP 1061 H= 265:15 +CPU= 32.603
  01:49:31 STEP 1062 H= 265:30 +CPU= 32.888
  01:50:04 STEP 1063 H= 265:45 +CPU= 32.761
  01:50:38 STEP 1064 H= 266:00 +CPU= 33.525
  01:51:11 STEP 1065 H= 266:15 +CPU= 33.049
  01:51:44 STEP 1066 H= 266:30 +CPU= 32.950
  01:52:18 STEP 1067 H= 266:45 +CPU= 33.080
  01:53:01 STEP 1068 H= 267:00 +CPU= 42.783
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001068
  01:53:35 STEP 1069 H= 267:15 +CPU= 33.527
  01:54:09 STEP 1070 H= 267:30 +CPU= 34.041
  01:54:44 STEP 1071 H= 267:45 +CPU= 34.158
  01:55:19 STEP 1072 H= 268:00 +CPU= 34.756
  01:55:53 STEP 1073 H= 268:15 +CPU= 34.269
  01:56:28 STEP 1074 H= 268:30 +CPU= 34.093
  01:57:03 STEP 1075 H= 268:45 +CPU= 34.565
  01:57:38 STEP 1076 H= 269:00 +CPU= 35.103
  01:58:13 STEP 1077 H= 269:15 +CPU= 34.229
  01:58:47 STEP 1078 H= 269:30 +CPU= 33.925
  01:59:21 STEP 1079 H= 269:45 +CPU= 33.922
  02:00:04 STEP 1080 H= 270:00 +CPU= 42.657
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001080
  02:00:38 STEP 1081 H= 270:15 +CPU= 33.700
  02:01:13 STEP 1082 H= 270:30 +CPU= 34.489
  02:01:47 STEP 1083 H= 270:45 +CPU= 33.312
  02:02:18 STEP 1084 H= 271:00 +CPU= 30.518
  02:02:48 STEP 1085 H= 271:15 +CPU= 29.696
  02:03:17 STEP 1086 H= 271:30 +CPU= 29.478
  02:03:47 STEP 1087 H= 271:45 +CPU= 29.625
  02:04:18 STEP 1088 H= 272:00 +CPU= 30.343
  02:04:47 STEP 1089 H= 272:15 +CPU= 29.704
  02:05:18 STEP 1090 H= 272:30 +CPU= 30.148
  02:05:50 STEP 1091 H= 272:45 +CPU= 32.015
  02:06:31 STEP 1092 H= 273:00 +CPU= 40.554
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001092
  02:07:04 STEP 1093 H= 273:15 +CPU= 32.465
  02:07:36 STEP 1094 H= 273:30 +CPU= 32.086
  02:08:09 STEP 1095 H= 273:45 +CPU= 32.312
  02:08:42 STEP 1096 H= 274:00 +CPU= 33.476
  02:09:15 STEP 1097 H= 274:15 +CPU= 32.612
  02:09:48 STEP 1098 H= 274:30 +CPU= 32.548
  02:10:21 STEP 1099 H= 274:45 +CPU= 32.532
  02:10:55 STEP 1100 H= 275:00 +CPU= 33.279
  02:11:27 STEP 1101 H= 275:15 +CPU= 32.175
  02:12:00 STEP 1102 H= 275:30 +CPU= 33.090
  02:12:34 STEP 1103 H= 275:45 +CPU= 32.808
  02:13:19 STEP 1104 H= 276:00 +CPU= 44.563
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001104
  02:13:52 STEP 1105 H= 276:15 +CPU= 32.776
  02:14:25 STEP 1106 H= 276:30 +CPU= 32.536
  02:14:57 STEP 1107 H= 276:45 +CPU= 32.415
  02:15:31 STEP 1108 H= 277:00 +CPU= 33.488
  02:16:04 STEP 1109 H= 277:15 +CPU= 33.039
  02:16:38 STEP 1110 H= 277:30 +CPU= 32.977
  02:17:10 STEP 1111 H= 277:45 +CPU= 32.516
  02:17:45 STEP 1112 H= 278:00 +CPU= 33.892
  02:18:17 STEP 1113 H= 278:15 +CPU= 32.407
  02:18:51 STEP 1114 H= 278:30 +CPU= 32.746
  02:19:23 STEP 1115 H= 278:45 +CPU= 32.597
  02:20:04 STEP 1116 H= 279:00 +CPU= 40.757
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001116
  02:20:38 STEP 1117 H= 279:15 +CPU= 32.928
  02:21:11 STEP 1118 H= 279:30 +CPU= 33.139
  02:21:44 STEP 1119 H= 279:45 +CPU= 32.902
  02:22:18 STEP 1120 H= 280:00 +CPU= 33.616
  02:22:51 STEP 1121 H= 280:15 +CPU= 32.787
  02:23:24 STEP 1122 H= 280:30 +CPU= 32.673
  02:23:57 STEP 1123 H= 280:45 +CPU= 32.406
  02:24:31 STEP 1124 H= 281:00 +CPU= 33.987
  02:25:04 STEP 1125 H= 281:15 +CPU= 32.738
  02:25:37 STEP 1126 H= 281:30 +CPU= 32.942
  02:26:11 STEP 1127 H= 281:45 +CPU= 32.831
  02:26:52 STEP 1128 H= 282:00 +CPU= 40.840
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001128
  02:27:25 STEP 1129 H= 282:15 +CPU= 32.631
  02:27:58 STEP 1130 H= 282:30 +CPU= 32.551
  02:28:31 STEP 1131 H= 282:45 +CPU= 33.065
  02:29:05 STEP 1132 H= 283:00 +CPU= 33.536
  02:29:38 STEP 1133 H= 283:15 +CPU= 32.567
  02:30:10 STEP 1134 H= 283:30 +CPU= 32.409
  02:30:44 STEP 1135 H= 283:45 +CPU= 32.772
  02:31:18 STEP 1136 H= 284:00 +CPU= 33.770
  02:31:51 STEP 1137 H= 284:15 +CPU= 32.942
  02:32:24 STEP 1138 H= 284:30 +CPU= 33.239
  02:32:57 STEP 1139 H= 284:45 +CPU= 32.737
  02:33:39 STEP 1140 H= 285:00 +CPU= 41.020
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001140
  02:34:12 STEP 1141 H= 285:15 +CPU= 32.602
  02:34:45 STEP 1142 H= 285:30 +CPU= 32.619
  02:35:17 STEP 1143 H= 285:45 +CPU= 32.482
  02:35:51 STEP 1144 H= 286:00 +CPU= 33.196
  02:36:24 STEP 1145 H= 286:15 +CPU= 32.849
  02:36:57 STEP 1146 H= 286:30 +CPU= 32.842
  02:37:30 STEP 1147 H= 286:45 +CPU= 32.553
  02:38:04 STEP 1148 H= 287:00 +CPU= 33.757
  02:38:37 STEP 1149 H= 287:15 +CPU= 33.056
  02:39:11 STEP 1150 H= 287:30 +CPU= 32.831
  02:39:44 STEP 1151 H= 287:45 +CPU= 32.706
  02:40:29 STEP 1152 H= 288:00 +CPU= 44.575
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001152
  02:41:01 STEP 1153 H= 288:15 +CPU= 32.584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001152
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_11.zip
Uploading the intermediate file: upload_file_11.zip
  02:41:35 STEP 1154 H= 288:30 +CPU= 32.406
  02:42:09 STEP 1155 H= 288:45 +CPU= 33.035
  02:42:43 STEP 1156 H= 289:00 +CPU= 33.686
  02:43:15 STEP 1157 H= 289:15 +CPU= 32.529
  02:43:48 STEP 1158 H= 289:30 +CPU= 32.462
  02:44:21 STEP 1159 H= 289:45 +CPU= 32.494
  02:44:55 STEP 1160 H= 290:00 +CPU= 33.465
  02:45:28 STEP 1161 H= 290:15 +CPU= 32.806
  02:46:01 STEP 1162 H= 290:30 +CPU= 32.960
  02:46:34 STEP 1163 H= 290:45 +CPU= 32.768
  02:47:15 STEP 1164 H= 291:00 +CPU= 40.730
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001164
  02:47:49 STEP 1165 H= 291:15 +CPU= 32.912
  02:48:22 STEP 1166 H= 291:30 +CPU= 32.778
  02:48:54 STEP 1167 H= 291:45 +CPU= 32.233
  02:49:29 STEP 1168 H= 292:00 +CPU= 34.066
  02:50:08 STEP 1169 H= 292:15 +CPU= 39.101
  02:50:49 STEP 1170 H= 292:30 +CPU= 40.104
  02:51:31 STEP 1171 H= 292:45 +CPU= 41.712
  02:52:12 STEP 1172 H= 293:00 +CPU= 40.240
  02:52:54 STEP 1173 H= 293:15 +CPU= 41.516
  02:53:36 STEP 1174 H= 293:30 +CPU= 41.720
  02:54:17 STEP 1175 H= 293:45 +CPU= 40.244
  02:55:07 STEP 1176 H= 294:00 +CPU= 50.369
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001176
  02:55:49 STEP 1177 H= 294:15 +CPU= 41.136
  02:56:32 STEP 1178 H= 294:30 +CPU= 42.308
  02:57:11 STEP 1179 H= 294:45 +CPU= 38.627
  02:57:52 STEP 1180 H= 295:00 +CPU= 41.223
  02:58:32 STEP 1181 H= 295:15 +CPU= 39.452
  02:59:11 STEP 1182 H= 295:30 +CPU= 38.496
  02:59:53 STEP 1183 H= 295:45 +CPU= 40.985
  03:00:35 STEP 1184 H= 296:00 +CPU= 42.421
  03:01:18 STEP 1185 H= 296:15 +CPU= 41.704
  03:01:59 STEP 1186 H= 296:30 +CPU= 40.938
  03:02:40 STEP 1187 H= 296:45 +CPU= 40.829
  03:03:29 STEP 1188 H= 297:00 +CPU= 48.421
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001188
  03:04:07 STEP 1189 H= 297:15 +CPU= 37.419
  03:04:45 STEP 1190 H= 297:30 +CPU= 37.118
  03:05:22 STEP 1191 H= 297:45 +CPU= 36.858
  03:06:01 STEP 1192 H= 298:00 +CPU= 38.641
  03:06:41 STEP 1193 H= 298:15 +CPU= 39.558
  03:07:20 STEP 1194 H= 298:30 +CPU= 38.643
  03:07:56 STEP 1195 H= 298:45 +CPU= 35.520
  03:08:33 STEP 1196 H= 299:00 +CPU= 36.693
  03:09:12 STEP 1197 H= 299:15 +CPU= 38.269
  03:09:55 STEP 1198 H= 299:30 +CPU= 42.400
  03:10:37 STEP 1199 H= 299:45 +CPU= 41.579
  03:11:34 STEP 1200 H= 300:00 +CPU= 56.317
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001200
  03:12:17 STEP 1201 H= 300:15 +CPU= 42.686
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

I     I Latent heat     II              I              I      1.40303 I              I               I
I     Iof which neg q isII              I              I      0.00000 I      0.00233 I      0.00290  I
DDH--------------------------------------------------------------------------------------------------I
I     I Transport       II     -0.00247 I              I     -0.00053 I      0.00000 I      0.00000  I
I     I Dissipation     II      0.00031 I     -0.00031 I              I              I               I
I CONV- Cond. Water     II      0.02190 I              I     -0.06942 I      0.06942 I               I
I     I Rain            II              I              I              I     -0.00876 I               I
I     I Cond. Ice       II      0.00070 I              I     -0.03558 I              I      0.03558  I
I     I Snow            II              I              I              I              I     -0.00025  I
I     I Sens. heat rain II      0.00000 I              I              I              I               I
I     I Sens. heat snow II      0.00000 I              I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I     I Cond. Water     II      0.02947 I              I     -0.00442 I      0.00442 I               I
I STRA- Rain            II              I              I              I     -0.01179 I               I
I     I Cond. Ice       II      0.45942 I              I     -0.03959 I              I      0.03959  I
I     I Snow            II              I              I              I              I     -0.16208  I
I     I Sens. heat rain II      0.00000 I              I              I              I               I
I     I Sens. heat snow II      0.00000 I              I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I PREC- Pot. and kin. E.II      0.00000 I      0.00000 I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I   Orographic drag     II      0.00000 I      0.00000 I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I      I Balance        II      0.00002 I              I              I              I               I
I ADIAB- of which       II              I              I              I              I               I
I      I Hor. Flux of PEII     -0.00000 I              I              I              I               I
I      I Vert.Flux of PEII      0.00000 I              I              I              I               I
I      I Work Pres.G.F  II              I      0.21806 I              I              I               I
I      I Conv. Ad.-Int. II     -0.21804 I              I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I DYN- Horizont. Flux   II      0.00958 I      0.00432 I      0.14661 I     -0.00039 I      0.00030  I
I    I Vertical Flux    II      0.00000 I      0.00000 I      0.00000 I      0.00000 I      0.00000  I
DDH--------------------------------------------------------------------------------------------------I
  
DDH----------------------------------------------------------------------------------------------------------------------------------
  I   - Mass balance    (10**(-6)  KG/M**2) 
  
  I INITIAL      I HORIZ. FLUX  I DYN. FLUX    I DYN. FLUX    I PHYS. FLUX   I PHYS. FLUX   I CURRENT      I I I RESIDUAL     I
  I MASS         I              I TOA          I SURFACE      I TOA          I SURFACE      I MASS         I I I              I
  ---------------------------------------------------------------------------------------------------------- I ----------------
  -              -              -              -              -              -              -              - I -              -
  - 10341776742. I          -0. .           0. .           0. .           0. .           0. I 10341981280. - I -      204538. -
  -              -              -              -              -              -              -              - I -              -
  ---------------------------------------------------------------------------------------------------------- I ----------------
 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  300.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.8394171730E-22      I       0.3282125457E-05      I
ICloud fractionI      0.0000000000E+00      I       0.1046943676E-01      I
I  Relat. Hum. I      0.1395409102E+00      I       0.1848833392E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1975192664E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1200 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001200 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001200
 MPI-TASK:   1 -    313664404 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS448092.0 Mbytes/s, TOTAL TIME=    0.0177( 4.0%)
 03:11:34 STEP 1200 H= 300:00 +CPU= 56.317
 NSTEP =  1201 STEPO   0AAA00AAA
 03:12:17 STEP 1201 H= 300:15 +CPU= 42.686
 NSTEP =  1202 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 03:10:41 0AA000000 STEPO     1200     4.434    4.434    4.504    409:13    413:43 0.11197450229326E-05       2GB       0MB
 03:10:44 FULLPOS-B DYNFPOS   1200     2.426    2.426    2.479    409:15    413:45 0.11197450229326E-05       2GB       0MB
 03:10:49 FULLPOS-V DYNFPOS   1200     5.331    5.331    5.382    409:21    413:51 0.11197450229326E-05       2GB       0MB
 03:10:50 FULLPOS-T DYNFPOS   1200     1.044    1.044    1.049    409:22    413:52 0.11197450229326E-05       2GB       0MB
 03:10:51 0AAA00AAA STEPO     1200     0.902    0.902    0.921    409:23    413:53 0.11197450229326E-05       2GB       0MB
 03:11:34 0AAA00AAA STEPO     1201    41.920   41.920   42.457    410:05    414:35 0.11197450229326E-05       2GB       0MB
 03:12:17 0AAA00AAA STEPO     1202    42.686   42.686   43.255    410:47    415:19 0.11197450229326E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1200",
 CTIME   = "00121200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1188, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/5/progress_file_12284680.xml

<running_values>
  <last_cpu_time>44903.100000</last_cpu_time>
  <upload_file_number>12</upload_file_number>
  <last_iter>1202</last_iter>
  <last_upload>1037700</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a279
(argv4) batchid: 1017
(argv5) wuid: 12284680
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4bbc3d5800473d7b0b5872f27e197dd9 to: /var/lib/boinc-client/slots/5/oifs_43r3_bl_a279_2016092300_20_1017_12284680.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_bl_a279_2016092300_20_1017_12284680.zip
ic_ancil_file: ic_ancil_12284680
ifsdata_file: ifsdata_12284680
climate_data_file: clim_data_12284680
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/5/ic_ancil_12284680.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12284680.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b0be9501c1e2b3c19baf6a9f5251300a to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12284680.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12284680.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12284680.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12284680.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12284680.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12284680.xml
-- Model is restarting --
Adjusting last_iter, 1202, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12284680.xml
last_cpu_time: 44903
upload_file_number: 12
last_iter: 1201
last_upload: 1037700
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687
The child process has been launched with process id: 887961
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Debian17:1:1:887961:887961] [20240609:040154:1717902114:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:02:26 STEP 1200 H= 300:00 +CPU= 29.116
  04:02:54 STEP 1201 H= 300:15 +CPU= 27.839
  04:03:22 STEP 1202 H= 300:30 +CPU= 27.725
  04:03:50 STEP 1203 H= 300:45 +CPU= 27.799
  04:04:19 STEP 1204 H= 301:00 +CPU= 28.609
  04:04:47 STEP 1205 H= 301:15 +CPU= 27.893
  04:05:15 STEP 1206 H= 301:30 +CPU= 27.990
  04:05:43 STEP 1207 H= 301:45 +CPU= 27.891
  04:06:12 STEP 1208 H= 302:00 +CPU= 28.466
  04:06:40 STEP 1209 H= 302:15 +CPU= 27.717
  04:07:08 STEP 1210 H= 302:30 +CPU= 27.805
  04:07:35 STEP 1211 H= 302:45 +CPU= 27.641
  04:08:11 STEP 1212 H= 303:00 +CPU= 35.085
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001212
  04:08:39 STEP 1213 H= 303:15 +CPU= 27.639
  04:09:06 STEP 1214 H= 303:30 +CPU= 27.685
  04:09:35 STEP 1215 H= 303:45 +CPU= 27.946
  04:10:03 STEP 1216 H= 304:00 +CPU= 28.609
  04:10:31 STEP 1217 H= 304:15 +CPU= 27.753
  04:10:59 STEP 1218 H= 304:30 +CPU= 27.922
  04:11:28 STEP 1219 H= 304:45 +CPU= 28.091
  04:11:57 STEP 1220 H= 305:00 +CPU= 28.644
  04:12:25 STEP 1221 H= 305:15 +CPU= 27.867
  04:12:53 STEP 1222 H= 305:30 +CPU= 28.047
  04:13:21 STEP 1223 H= 305:45 +CPU= 28.056
  04:13:56 STEP 1224 H= 306:00 +CPU= 35.132
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001224
  04:14:25 STEP 1225 H= 306:15 +CPU= 28.003
  04:14:53 STEP 1226 H= 306:30 +CPU= 28.451
  04:15:26 STEP 1227 H= 306:45 +CPU= 32.041
  04:15:59 STEP 1228 H= 307:00 +CPU= 33.236
  04:16:32 STEP 1229 H= 307:15 +CPU= 32.424
  04:17:05 STEP 1230 H= 307:30 +CPU= 32.549
  04:17:38 STEP 1231 H= 307:45 +CPU= 32.573
  04:18:11 STEP 1232 H= 308:00 +CPU= 33.249
  04:18:44 STEP 1233 H= 308:15 +CPU= 32.824
  04:19:17 STEP 1234 H= 308:30 +CPU= 32.372
  04:19:50 STEP 1235 H= 308:45 +CPU= 32.853
  04:20:31 STEP 1236 H= 309:00 +CPU= 40.793
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001236
  04:21:04 STEP 1237 H= 309:15 +CPU= 32.733
  04:21:37 STEP 1238 H= 309:30 +CPU= 32.763
  04:22:10 STEP 1239 H= 309:45 +CPU= 32.516
  04:22:44 STEP 1240 H= 310:00 +CPU= 33.549
  04:23:17 STEP 1241 H= 310:15 +CPU= 32.621
  04:23:50 STEP 1242 H= 310:30 +CPU= 32.582
  04:24:23 STEP 1243 H= 310:45 +CPU= 32.511
  04:24:56 STEP 1244 H= 311:00 +CPU= 33.452
  04:25:29 STEP 1245 H= 311:15 +CPU= 32.640
  04:26:02 STEP 1246 H= 311:30 +CPU= 32.548
  04:26:35 STEP 1247 H= 311:45 +CPU= 32.526
  04:27:20 STEP 1248 H= 312:00 +CPU= 44.709
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001248
  04:27:53 STEP 1249 H= 312:15 +CPU= 32.665
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001248
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_12.zip
Uploading the intermediate file: upload_file_12.zip
  04:28:27 STEP 1250 H= 312:30 +CPU= 32.402
  04:28:59 STEP 1251 H= 312:45 +CPU= 32.495
  04:29:33 STEP 1252 H= 313:00 +CPU= 33.711
  04:30:06 STEP 1253 H= 313:15 +CPU= 32.441
  04:30:39 STEP 1254 H= 313:30 +CPU= 32.597
  04:31:12 STEP 1255 H= 313:45 +CPU= 32.955
  04:31:46 STEP 1256 H= 314:00 +CPU= 32.999
  04:32:19 STEP 1257 H= 314:15 +CPU= 32.817
  04:32:52 STEP 1258 H= 314:30 +CPU= 32.778
  04:33:25 STEP 1259 H= 314:45 +CPU= 32.730
  04:34:06 STEP 1260 H= 315:00 +CPU= 40.648
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001260
  04:34:39 STEP 1261 H= 315:15 +CPU= 32.660
  04:35:11 STEP 1262 H= 315:30 +CPU= 32.242
  04:35:50 STEP 1263 H= 315:45 +CPU= 37.942
  04:36:29 STEP 1264 H= 316:00 +CPU= 38.710
  04:37:07 STEP 1265 H= 316:15 +CPU= 38.278
  04:37:46 STEP 1266 H= 316:30 +CPU= 38.370
  04:38:26 STEP 1267 H= 316:45 +CPU= 39.205
  04:39:08 STEP 1268 H= 317:00 +CPU= 41.423
  04:39:46 STEP 1269 H= 317:15 +CPU= 37.837
  04:40:26 STEP 1270 H= 317:30 +CPU= 39.109
  04:41:06 STEP 1271 H= 317:45 +CPU= 39.511
  04:41:54 STEP 1272 H= 318:00 +CPU= 48.105
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001272
  04:42:32 STEP 1273 H= 318:15 +CPU= 37.425
  04:43:12 STEP 1274 H= 318:30 +CPU= 39.158
  04:43:47 STEP 1275 H= 318:45 +CPU= 35.165
  04:44:27 STEP 1276 H= 319:00 +CPU= 38.928
  04:45:07 STEP 1277 H= 319:15 +CPU= 39.951
  04:45:47 STEP 1278 H= 319:30 +CPU= 39.762
  04:46:28 STEP 1279 H= 319:45 +CPU= 40.217
  04:47:08 STEP 1280 H= 320:00 +CPU= 39.360
  04:47:46 STEP 1281 H= 320:15 +CPU= 38.278
  04:48:21 STEP 1282 H= 320:30 +CPU= 34.366
  04:48:56 STEP 1283 H= 320:45 +CPU= 34.919
  04:49:41 STEP 1284 H= 321:00 +CPU= 43.842
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001284
  04:50:16 STEP 1285 H= 321:15 +CPU= 35.046
  04:50:55 STEP 1286 H= 321:30 +CPU= 38.363
  04:51:30 STEP 1287 H= 321:45 +CPU= 35.187
  04:52:08 STEP 1288 H= 322:00 +CPU= 37.309
  04:52:42 STEP 1289 H= 322:15 +CPU= 33.813
  04:53:17 STEP 1290 H= 322:30 +CPU= 34.308
  04:53:56 STEP 1291 H= 322:45 +CPU= 39.044
  04:54:37 STEP 1292 H= 323:00 +CPU= 40.612
  04:55:18 STEP 1293 H= 323:15 +CPU= 40.513
  04:55:58 STEP 1294 H= 323:30 +CPU= 38.910
  04:56:39 STEP 1295 H= 323:45 +CPU= 40.707
  04:57:30 STEP 1296 H= 324:00 +CPU= 50.835
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001296
  04:58:10 STEP 1297 H= 324:15 +CPU= 39.693
  04:58:50 STEP 1298 H= 324:30 +CPU= 39.401
  04:59:30 STEP 1299 H= 324:45 +CPU= 39.753
  05:00:13 STEP 1300 H= 325:00 +CPU= 42.461
  05:00:54 STEP 1301 H= 325:15 +CPU= 40.302
  05:01:35 STEP 1302 H= 325:30 +CPU= 40.204
  05:02:16 STEP 1303 H= 325:45 +CPU= 40.711
  05:02:51 STEP 1304 H= 326:00 +CPU= 35.008
  05:03:27 STEP 1305 H= 326:15 +CPU= 34.982
  05:04:05 STEP 1306 H= 326:30 +CPU= 37.763
  05:04:42 STEP 1307 H= 326:45 +CPU= 37.153
  05:05:29 STEP 1308 H= 327:00 +CPU= 46.628
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001308
  05:06:05 STEP 1309 H= 327:15 +CPU= 34.899
  05:06:40 STEP 1310 H= 327:30 +CPU= 35.177
  05:07:14 STEP 1311 H= 327:45 +CPU= 33.463
  05:07:49 STEP 1312 H= 328:00 +CPU= 34.891
  05:08:24 STEP 1313 H= 328:15 +CPU= 34.553
  05:08:59 STEP 1314 H= 328:30 +CPU= 34.840
  05:09:34 STEP 1315 H= 328:45 +CPU= 34.533
  05:10:10 STEP 1316 H= 329:00 +CPU= 35.753
  05:10:46 STEP 1317 H= 329:15 +CPU= 35.316
  05:11:21 STEP 1318 H= 329:30 +CPU= 34.831
  05:11:56 STEP 1319 H= 329:45 +CPU= 34.555
  05:12:40 STEP 1320 H= 330:00 +CPU= 43.689
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001320
  05:13:16 STEP 1321 H= 330:15 +CPU= 35.404
  05:13:49 STEP 1322 H= 330:30 +CPU= 32.967
  05:14:24 STEP 1323 H= 330:45 +CPU= 34.812
  05:14:59 STEP 1324 H= 331:00 +CPU= 34.199
  05:15:32 STEP 1325 H= 331:15 +CPU= 32.790
  05:16:05 STEP 1326 H= 331:30 +CPU= 33.169
  05:16:39 STEP 1327 H= 331:45 +CPU= 33.429
  05:17:14 STEP 1328 H= 332:00 +CPU= 34.195
  05:17:47 STEP 1329 H= 332:15 +CPU= 33.009
  05:18:20 STEP 1330 H= 332:30 +CPU= 33.123
  05:18:54 STEP 1331 H= 332:45 +CPU= 33.375
  05:19:36 STEP 1332 H= 333:00 +CPU= 41.100
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001332
  05:20:09 STEP 1333 H= 333:15 +CPU= 32.993
  05:20:42 STEP 1334 H= 333:30 +CPU= 33.152
  05:21:16 STEP 1335 H= 333:45 +CPU= 33.255
  05:21:50 STEP 1336 H= 334:00 +CPU= 33.512
  05:22:23 STEP 1337 H= 334:15 +CPU= 32.749
  05:22:56 STEP 1338 H= 334:30 +CPU= 32.928
  05:23:30 STEP 1339 H= 334:45 +CPU= 33.337
  05:24:04 STEP 1340 H= 335:00 +CPU= 33.717
  05:24:37 STEP 1341 H= 335:15 +CPU= 33.272
  05:25:11 STEP 1342 H= 335:30 +CPU= 33.407
  05:25:45 STEP 1343 H= 335:45 +CPU= 33.039
  05:26:30 STEP 1344 H= 336:00 +CPU= 45.043
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001344
  05:27:04 STEP 1345 H= 336:15 +CPU= 33.446
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001344
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_13.zip
Uploading the intermediate file: upload_file_13.zip
  05:27:38 STEP 1346 H= 336:30 +CPU= 32.751
  05:28:11 STEP 1347 H= 336:45 +CPU= 32.955
  05:28:46 STEP 1348 H= 337:00 +CPU= 34.116
  05:29:19 STEP 1349 H= 337:15 +CPU= 32.943
  05:29:53 STEP 1350 H= 337:30 +CPU= 33.286
  05:30:26 STEP 1351 H= 337:45 +CPU= 32.889
  05:31:00 STEP 1352 H= 338:00 +CPU= 33.801
  05:31:34 STEP 1353 H= 338:15 +CPU= 33.110
  05:32:07 STEP 1354 H= 338:30 +CPU= 33.420
  05:32:41 STEP 1355 H= 338:45 +CPU= 33.240
  05:33:23 STEP 1356 H= 339:00 +CPU= 41.572
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001356
  05:33:57 STEP 1357 H= 339:15 +CPU= 33.375
  05:34:30 STEP 1358 H= 339:30 +CPU= 33.158
  05:35:04 STEP 1359 H= 339:45 +CPU= 33.270
  05:35:38 STEP 1360 H= 340:00 +CPU= 33.954
  05:36:12 STEP 1361 H= 340:15 +CPU= 33.427
  05:36:45 STEP 1362 H= 340:30 +CPU= 33.220
  05:37:19 STEP 1363 H= 340:45 +CPU= 33.800
  05:37:54 STEP 1364 H= 341:00 +CPU= 34.095
  05:38:27 STEP 1365 H= 341:15 +CPU= 33.332
  05:39:01 STEP 1366 H= 341:30 +CPU= 33.325
  05:39:35 STEP 1367 H= 341:45 +CPU= 33.281
  05:40:16 STEP 1368 H= 342:00 +CPU= 41.338
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001368
  05:40:50 STEP 1369 H= 342:15 +CPU= 33.344
  05:41:23 STEP 1370 H= 342:30 +CPU= 33.117
  05:41:57 STEP 1371 H= 342:45 +CPU= 33.083
  05:42:31 STEP 1372 H= 343:00 +CPU= 34.033
  05:43:05 STEP 1373 H= 343:15 +CPU= 33.401
  05:43:39 STEP 1374 H= 343:30 +CPU= 33.287
  05:44:12 STEP 1375 H= 343:45 +CPU= 33.225
  05:44:46 STEP 1376 H= 344:00 +CPU= 34.016
  05:45:20 STEP 1377 H= 344:15 +CPU= 33.332
  05:45:53 STEP 1378 H= 344:30 +CPU= 32.858
  05:46:27 STEP 1379 H= 344:45 +CPU= 33.055
  05:47:08 STEP 1380 H= 345:00 +CPU= 41.526
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001380
  05:47:42 STEP 1381 H= 345:15 +CPU= 33.229
  05:48:16 STEP 1382 H= 345:30 +CPU= 33.635
  05:48:49 STEP 1383 H= 345:45 +CPU= 33.062
  05:49:24 STEP 1384 H= 346:00 +CPU= 33.929
  05:49:57 STEP 1385 H= 346:15 +CPU= 32.707
  05:50:30 STEP 1386 H= 346:30 +CPU= 33.074
  05:51:04 STEP 1387 H= 346:45 +CPU= 33.507
  05:51:38 STEP 1388 H= 347:00 +CPU= 33.942
  05:52:10 STEP 1389 H= 347:15 +CPU= 31.654
  05:52:40 STEP 1390 H= 347:30 +CPU= 29.346
  05:53:10 STEP 1391 H= 347:45 +CPU= 29.789
  05:53:49 STEP 1392 H= 348:00 +CPU= 39.188
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001392
  05:54:18 STEP 1393 H= 348:15 +CPU= 28.857
  05:54:47 STEP 1394 H= 348:30 +CPU= 28.699
  05:55:16 STEP 1395 H= 348:45 +CPU= 28.713
  05:55:46 STEP 1396 H= 349:00 +CPU= 29.573
  05:56:15 STEP 1397 H= 349:15 +CPU= 28.644
  05:56:44 STEP 1398 H= 349:30 +CPU= 28.976
  05:57:13 STEP 1399 H= 349:45 +CPU= 28.815
  05:57:43 STEP 1400 H= 350:00 +CPU= 29.595
  05:58:12 STEP 1401 H= 350:15 +CPU= 29.118
  05:58:41 STEP 1402 H= 350:30 +CPU= 28.961
  05:59:10 STEP 1403 H= 350:45 +CPU= 28.853
  05:59:47 STEP 1404 H= 351:00 +CPU= 36.145
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001404
  06:00:16 STEP 1405 H= 351:15 +CPU= 28.758
  06:00:45 STEP 1406 H= 351:30 +CPU= 28.783
  06:01:14 STEP 1407 H= 351:45 +CPU= 28.906
  06:01:44 STEP 1408 H= 352:00 +CPU= 29.530
  06:02:13 STEP 1409 H= 352:15 +CPU= 28.830
  06:02:42 STEP 1410 H= 352:30 +CPU= 28.924
  06:03:11 STEP 1411 H= 352:45 +CPU= 28.867
  06:03:41 STEP 1412 H= 353:00 +CPU= 29.355
  06:04:10 STEP 1413 H= 353:15 +CPU= 28.704
  06:04:39 STEP 1414 H= 353:30 +CPU= 29.050
  06:05:08 STEP 1415 H= 353:45 +CPU= 29.007
  06:05:45 STEP 1416 H= 354:00 +CPU= 36.444
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001416
  06:06:14 STEP 1417 H= 354:15 +CPU= 28.891
  06:06:43 STEP 1418 H= 354:30 +CPU= 28.981
  06:07:12 STEP 1419 H= 354:45 +CPU= 28.998
  06:07:42 STEP 1420 H= 355:00 +CPU= 29.614
  06:08:11 STEP 1421 H= 355:15 +CPU= 28.879
  06:08:40 STEP 1422 H= 355:30 +CPU= 28.816
  06:09:09 STEP 1423 H= 355:45 +CPU= 28.779
  06:09:39 STEP 1424 H= 356:00 +CPU= 29.465
  06:10:08 STEP 1425 H= 356:15 +CPU= 28.809
  06:10:37 STEP 1426 H= 356:30 +CPU= 28.668
  06:11:06 STEP 1427 H= 356:45 +CPU= 28.863
  06:11:42 STEP 1428 H= 357:00 +CPU= 36.185
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001428
  06:12:11 STEP 1429 H= 357:15 +CPU= 28.903
  06:12:40 STEP 1430 H= 357:30 +CPU= 28.744
  06:13:09 STEP 1431 H= 357:45 +CPU= 28.693
  06:13:39 STEP 1432 H= 358:00 +CPU= 29.533
  06:14:08 STEP 1433 H= 358:15 +CPU= 28.834
  06:14:37 STEP 1434 H= 358:30 +CPU= 28.932
  06:15:06 STEP 1435 H= 358:45 +CPU= 28.855
  06:15:36 STEP 1436 H= 359:00 +CPU= 29.717
  06:16:06 STEP 1437 H= 359:15 +CPU= 28.891
  06:16:35 STEP 1438 H= 359:30 +CPU= 28.937
  06:17:04 STEP 1439 H= 359:45 +CPU= 28.991
  06:17:42 STEP 1440 H= 360:00 +CPU= 37.352
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.8394171730E-22      I       0.1979633670E-05      I
ICloud fractionI      0.0000000000E+00      I       0.1493254810E-01      I
I  Relat. Hum. I      0.1395409102E+00      I       0.1979988770E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.2832442313E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    305460116 BYTES IN      7 RECORDS TRANSFERRED IN    0.0005 SECONDS610920.2 Mbytes/s, TOTAL TIME=    0.0113( 4.4%)
 06:17:42 STEP 1440 H= 360:00 +CPU= 37.352
 END CNT3
 NSTEP =  1440 CNT0   000000000  27.788  27.788  28.050   0.991               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         7150/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    128    0       46      1044068     7150 ************    14.0     1.4    12.6     1.9    28.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 8147837.5 8147837.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 8144877.1 8144877.1     99.96      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  261    4608.3    4608.3     14.76      0.00
   9 SPCM     - SPECTRAL COMP.                     240    1020.5    1020.5      3.01      0.00
  10 SCAN2M - PHYSICS                              241   17417.3   17417.3     51.52      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 20     843.3     843.3      0.21      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 11      65.4      65.4      0.01      0.00
  17 GRIDFPOS IN CNT4                               20      39.0      39.0      0.01      0.00
  27 RESTART FILES - WRITING                         5    3373.0    3373.0      0.21      0.00
  28 RESTART FILES - READING                         1    2559.0    2559.0      0.03      0.00
  29 SU4FPOS IN CNT4                                20       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                20    7187.5    7187.5      1.76      0.00
  31 POSDDH IN STEPO                                 2       1.2       1.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    241    6535.5    6535.5     19.33      0.00
  39 SU0YOMB                                         1     276.1     276.1      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    241      21.0      21.0      0.06      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              240     396.6     396.6      1.17      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              240      56.5      56.5      0.17      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 11       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      524     504.4     504.4      3.24      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       500     305.0     305.0      1.87      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        500      97.5      97.5      0.60      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       524     180.5     180.5      1.16      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      41.3      41.3      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            524      31.4      31.4      0.20      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            500      21.4      21.4      0.13      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            524     128.9     128.9      0.83      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            500      34.4      34.4      0.21      0.00
 400 GSTATS                                      28864       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 26802       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     8147.8 CPU TIME    8071.5 VECTOR TIME     8071.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      159.1
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/5/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/5/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284680/ICMUAh7zg+001392
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
06:21:10 (887958): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a279_2016092300_20_1017_12284680_0_r485840687_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 05:19:14 1491072 22436268 oifs_43r3_bl_a279_2016092300_20_1017_12284680_0 1,295,100 52,974 0.0409


©2024 climateprediction.net