climateprediction.net home page
Task 22255165

Task 22255165

Name oifs_43r3_ps_0193_1985050100_123_954_12170837_0
Workunit 12170837
Created 21 Dec 2022, 5:22:53 UTC
Sent 21 Dec 2022, 5:30:21 UTC
Report deadline 20 Jan 2023, 5:30:21 UTC
Received 27 Dec 2022, 23:32:08 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1488917
Run time 5 days 17 hours 59 min 17 sec
CPU time 23 hours 49 min 38 sec
Validate state Invalid
Credit 4,838.40
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,406.57 MB
Peak swap size 4,974.80 MB
Peak disk usage 2,446.42 MB
Stderr
<core_client_version>7.16.17</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
49.008
  06:38:04 STEP  966 H= 966:00 +CPU= 79.535
  06:42:57 STEP  967 H= 967:00 +CPU= 49.695
  06:48:14 STEP  968 H= 968:00 +CPU= 52.305
  06:55:59 STEP  969 H= 969:00 +CPU= 77.778
  07:01:05 STEP  970 H= 970:00 +CPU= 51.236
  07:06:25 STEP  971 H= 971:00 +CPU= 52.971
  07:15:07 STEP  972 H= 972:00 +CPU= 87.912
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+000972
  07:20:22 STEP  973 H= 973:00 +CPU= 50.830
  07:25:25 STEP  974 H= 974:00 +CPU= 50.121
  07:32:57 STEP  975 H= 975:00 +CPU= 76.340
  07:37:54 STEP  976 H= 976:00 +CPU= 50.434
  07:43:00 STEP  977 H= 977:00 +CPU= 50.351
  07:51:22 STEP  978 H= 978:00 +CPU= 81.461
  07:56:09 STEP  979 H= 979:00 +CPU= 48.278
  08:01:26 STEP  980 H= 980:00 +CPU= 52.970
  08:09:26 STEP  981 H= 981:00 +CPU= 79.224
  08:14:17 STEP  982 H= 982:00 +CPU= 49.817
  08:19:49 STEP  983 H= 983:00 +CPU= 54.019
  08:28:56 STEP  984 H= 984:00 +CPU= 91.978
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_40.zip
Uploading the intermediate file: upload_file_40.zip
  08:34:03 STEP  985 H= 985:00 +CPU= 51.163
  08:39:14 STEP  986 H= 986:00 +CPU= 50.936
  08:47:04 STEP  987 H= 987:00 +CPU= 77.113
  08:52:46 STEP  988 H= 988:00 +CPU= 53.581
  08:57:27 STEP  989 H= 989:00 +CPU= 46.980
  09:05:33 STEP  990 H= 990:00 +CPU= 80.284
  09:10:33 STEP  991 H= 991:00 +CPU= 49.496
  09:15:37 STEP  992 H= 992:00 +CPU= 51.038
  09:23:45 STEP  993 H= 993:00 +CPU= 80.844
  09:28:46 STEP  994 H= 994:00 +CPU= 52.076
  09:33:57 STEP  995 H= 995:00 +CPU= 51.299
  09:42:08 STEP  996 H= 996:00 +CPU= 83.964
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+000996
  09:47:18 STEP  997 H= 997:00 +CPU= 50.418
  09:52:29 STEP  998 H= 998:00 +CPU= 52.087
  10:00:46 STEP  999 H= 999:00 +CPU= 80.646
  10:05:46 STEP 1000 H=1000:00 +CPU= 50.344
  10:10:50 STEP 1001 H=1001:00 +CPU= 49.565
  10:19:20 STEP 1002 H=1002:00 +CPU= 83.684
  10:24:28 STEP 1003 H=1003:00 +CPU= 50.849
  10:29:52 STEP 1004 H=1004:00 +CPU= 53.751
  10:37:55 STEP 1005 H=1005:00 +CPU= 79.291
  10:43:05 STEP 1006 H=1006:00 +CPU= 52.355
  10:48:02 STEP 1007 H=1007:00 +CPU= 50.510
  10:57:28 STEP 1008 H=1008:00 +CPU= 91.413
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_41.zip
Uploading the intermediate file: upload_file_41.zip
  11:02:35 STEP 1009 H=1009:00 +CPU= 51.431
  11:07:30 STEP 1010 H=1010:00 +CPU= 50.737
  11:15:04 STEP 1011 H=1011:00 +CPU= 77.466
  11:20:04 STEP 1012 H=1012:00 +CPU= 51.299
  11:24:47 STEP 1013 H=1013:00 +CPU= 48.275
  11:32:52 STEP 1014 H=1014:00 +CPU= 80.864
  11:37:41 STEP 1015 H=1015:00 +CPU= 48.944
  11:42:53 STEP 1016 H=1016:00 +CPU= 51.288
  11:50:56 STEP 1017 H=1017:00 +CPU= 79.990
  11:55:52 STEP 1018 H=1018:00 +CPU= 49.945
  12:00:40 STEP 1019 H=1019:00 +CPU= 49.978
  12:09:16 STEP 1020 H=1020:00 +CPU= 86.802
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001020
  12:14:26 STEP 1021 H=1021:00 +CPU= 51.047
  12:19:45 STEP 1022 H=1022:00 +CPU= 53.466
  12:27:14 STEP 1023 H=1023:00 +CPU= 76.391
  12:32:46 STEP 1024 H=1024:00 +CPU= 54.228
  12:38:06 STEP 1025 H=1025:00 +CPU= 51.364
  12:45:42 STEP 1026 H=1026:00 +CPU= 78.502
  12:50:52 STEP 1027 H=1027:00 +CPU= 51.197
  12:56:16 STEP 1028 H=1028:00 +CPU= 51.848
  13:03:59 STEP 1029 H=1029:00 +CPU= 77.380
  13:08:55 STEP 1030 H=1030:00 +CPU= 50.525
  13:14:18 STEP 1031 H=1031:00 +CPU= 52.294
  13:23:19 STEP 1032 H=1032:00 +CPU= 91.497
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_42.zip
Uploading the intermediate file: upload_file_42.zip
  13:28:33 STEP 1033 H=1033:00 +CPU= 50.044
  13:33:58 STEP 1034 H=1034:00 +CPU= 53.154
  13:42:05 STEP 1035 H=1035:00 +CPU= 78.884
  13:47:23 STEP 1036 H=1036:00 +CPU= 52.894
  13:52:16 STEP 1037 H=1037:00 +CPU= 48.828
  14:00:19 STEP 1038 H=1038:00 +CPU= 81.643
  14:05:01 STEP 1039 H=1039:00 +CPU= 49.096
  14:10:16 STEP 1040 H=1040:00 +CPU= 50.884
  14:18:39 STEP 1041 H=1041:00 +CPU= 80.338
  14:24:04 STEP 1042 H=1042:00 +CPU= 53.257
  14:29:28 STEP 1043 H=1043:00 +CPU= 51.156
  14:38:09 STEP 1044 H=1044:00 +CPU= 87.651
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001044
  14:43:35 STEP 1045 H=1045:00 +CPU= 51.836
  14:48:43 STEP 1046 H=1046:00 +CPU= 51.452
  14:57:11 STEP 1047 H=1047:00 +CPU= 82.106
  15:02:14 STEP 1048 H=1048:00 +CPU= 52.099
  15:07:10 STEP 1049 H=1049:00 +CPU= 49.202
  15:15:06 STEP 1050 H=1050:00 +CPU= 80.254
  15:19:50 STEP 1051 H=1051:00 +CPU= 47.742
  15:24:59 STEP 1052 H=1052:00 +CPU= 50.861
  15:33:01 STEP 1053 H=1053:00 +CPU= 79.456
  15:37:56 STEP 1054 H=1054:00 +CPU= 51.018
  15:43:16 STEP 1055 H=1055:00 +CPU= 51.830
  15:52:23 STEP 1056 H=1056:00 +CPU= 90.160
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_43.zip
Uploading the intermediate file: upload_file_43.zip
  15:57:10 STEP 1057 H=1057:00 +CPU= 48.841
  16:02:24 STEP 1058 H=1058:00 +CPU= 52.036
  16:10:25 STEP 1059 H=1059:00 +CPU= 80.814
  16:15:33 STEP 1060 H=1060:00 +CPU= 50.889
  16:20:24 STEP 1061 H=1061:00 +CPU= 48.326
  16:28:43 STEP 1062 H=1062:00 +CPU= 82.529
  16:33:26 STEP 1063 H=1063:00 +CPU= 49.003
  16:38:30 STEP 1064 H=1064:00 +CPU= 51.171
  16:46:01 STEP 1065 H=1065:00 +CPU= 76.248
  16:51:06 STEP 1066 H=1066:00 +CPU= 51.774
  16:56:09 STEP 1067 H=1067:00 +CPU= 50.464
  17:04:58 STEP 1068 H=1068:00 +CPU= 87.753
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001068
  17:09:56 STEP 1069 H=1069:00 +CPU= 48.849
  17:14:59 STEP 1070 H=1070:00 +CPU= 51.437
  17:22:40 STEP 1071 H=1071:00 +CPU= 76.433
  17:28:00 STEP 1072 H=1072:00 +CPU= 51.550
  17:33:19 STEP 1073 H=1073:00 +CPU= 52.105
  17:41:12 STEP 1074 H=1074:00 +CPU= 79.505
  17:46:06 STEP 1075 H=1075:00 +CPU= 49.258
  17:51:26 STEP 1076 H=1076:00 +CPU= 52.933
  17:59:03 STEP 1077 H=1077:00 +CPU= 77.828
  18:04:08 STEP 1078 H=1078:00 +CPU= 50.817
  18:09:40 STEP 1079 H=1079:00 +CPU= 51.471
  18:19:19 STEP 1080 H=1080:00 +CPU= 92.825
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_44.zip
Uploading the intermediate file: upload_file_44.zip
  18:24:23 STEP 1081 H=1081:00 +CPU= 49.386
  18:29:27 STEP 1082 H=1082:00 +CPU= 50.372
  18:36:59 STEP 1083 H=1083:00 +CPU= 76.264
  18:41:59 STEP 1084 H=1084:00 +CPU= 50.250
  18:46:57 STEP 1085 H=1085:00 +CPU= 50.025
  18:55:13 STEP 1086 H=1086:00 +CPU= 83.900
  19:00:18 STEP 1087 H=1087:00 +CPU= 50.686
  19:05:44 STEP 1088 H=1088:00 +CPU= 53.176
  19:13:44 STEP 1089 H=1089:00 +CPU= 78.333
  19:19:06 STEP 1090 H=1090:00 +CPU= 54.142
  19:24:06 STEP 1091 H=1091:00 +CPU= 50.383
  19:33:18 STEP 1092 H=1092:00 +CPU= 88.643
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001092
  19:38:44 STEP 1093 H=1093:00 +CPU= 52.657
  19:44:12 STEP 1094 H=1094:00 +CPU= 52.678
  19:52:25 STEP 1095 H=1095:00 +CPU= 81.121
  19:57:21 STEP 1096 H=1096:00 +CPU= 50.626
  20:02:31 STEP 1097 H=1097:00 +CPU= 50.389
  20:11:06 STEP 1098 H=1098:00 +CPU= 83.072
  20:16:10 STEP 1099 H=1099:00 +CPU= 50.412
  20:21:18 STEP 1100 H=1100:00 +CPU= 51.688
  20:29:05 STEP 1101 H=1101:00 +CPU= 78.090
  20:34:39 STEP 1102 H=1102:00 +CPU= 53.002
  20:39:21 STEP 1103 H=1103:00 +CPU= 48.414
  20:48:36 STEP 1104 H=1104:00 +CPU= 92.636
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_45.zip
Uploading the intermediate file: upload_file_45.zip
  20:54:01 STEP 1105 H=1105:00 +CPU= 51.970
  20:59:02 STEP 1106 H=1106:00 +CPU= 50.094
  21:06:43 STEP 1107 H=1107:00 +CPU= 76.368
  21:11:38 STEP 1108 H=1108:00 +CPU= 50.324
  21:16:38 STEP 1109 H=1109:00 +CPU= 49.721
  21:24:31 STEP 1110 H=1110:00 +CPU= 81.116
  21:29:23 STEP 1111 H=1111:00 +CPU= 49.500
  21:34:48 STEP 1112 H=1112:00 +CPU= 52.930
  21:42:24 STEP 1113 H=1113:00 +CPU= 78.247
  21:47:15 STEP 1114 H=1114:00 +CPU= 49.100
  21:52:20 STEP 1115 H=1115:00 +CPU= 50.555
  22:00:37 STEP 1116 H=1116:00 +CPU= 82.990
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001116
  22:05:14 STEP 1117 H=1117:00 +CPU= 47.167
  22:10:52 STEP 1118 H=1118:00 +CPU= 53.550
  22:19:01 STEP 1119 H=1119:00 +CPU= 80.969
  22:24:08 STEP 1120 H=1120:00 +CPU= 50.564
  22:29:15 STEP 1121 H=1121:00 +CPU= 50.250
  22:37:28 STEP 1122 H=1122:00 +CPU= 81.848
  22:42:34 STEP 1123 H=1123:00 +CPU= 49.647
  22:47:49 STEP 1124 H=1124:00 +CPU= 51.637
  22:56:05 STEP 1125 H=1125:00 +CPU= 80.220
  23:01:08 STEP 1126 H=1126:00 +CPU= 51.314
  23:06:30 STEP 1127 H=1127:00 +CPU= 53.539
  23:16:00 STEP 1128 H=1128:00 +CPU= 92.440
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_46.zip
Uploading the intermediate file: upload_file_46.zip
  23:21:19 STEP 1129 H=1129:00 +CPU= 52.017
  23:26:40 STEP 1130 H=1130:00 +CPU= 54.094
  23:34:52 STEP 1131 H=1131:00 +CPU= 79.742
  23:39:56 STEP 1132 H=1132:00 +CPU= 51.506
  23:44:53 STEP 1133 H=1133:00 +CPU= 48.880
  23:52:44 STEP 1134 H=1134:00 +CPU= 79.613
  23:57:48 STEP 1135 H=1135:00 +CPU= 50.756
  00:02:44 STEP 1136 H=1136:00 +CPU= 51.106
  00:10:34 STEP 1137 H=1137:00 +CPU= 79.964
  00:15:35 STEP 1138 H=1138:00 +CPU= 51.773
  00:20:56 STEP 1139 H=1139:00 +CPU= 50.950
  00:29:55 STEP 1140 H=1140:00 +CPU= 88.503
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001140
  00:34:58 STEP 1141 H=1141:00 +CPU= 50.938
  00:40:09 STEP 1142 H=1142:00 +CPU= 52.254
  00:48:11 STEP 1143 H=1143:00 +CPU= 79.921
  00:53:19 STEP 1144 H=1144:00 +CPU= 52.948
  00:58:25 STEP 1145 H=1145:00 +CPU= 49.225
  01:06:35 STEP 1146 H=1146:00 +CPU= 80.850
  01:11:40 STEP 1147 H=1147:00 +CPU= 50.287
  01:16:48 STEP 1148 H=1148:00 +CPU= 51.049
  01:24:31 STEP 1149 H=1149:00 +CPU= 78.201
  01:29:49 STEP 1150 H=1150:00 +CPU= 51.891
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0193
(argv4) batchid: 954
(argv5) wuid: 12170837
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/51
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/51/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/51/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_195604768c17856686e5f5335db4557a to: /var/lib/boinc/slots/51/oifs_43r3_ps_0193_1985050100_123_954_12170837.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/51/oifs_43r3_ps_0193_1985050100_123_954_12170837.zip
ic_ancil_file: ic_ancil_12170837
ifsdata_file: ifsdata_12170837
climate_data_file: clim_data_12170837
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_58bde664c39c1b869542abb4827bddc4 to: /var/lib/boinc/slots/51/ic_ancil_12170837.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/51/ic_ancil_12170837.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_27b16cfe92038e0b10b6510dec0c3835 to: /var/lib/boinc/slots/51/ifsdata/ifsdata_12170837.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/51/ifsdata/ifsdata_12170837.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/51/159l_2/clim_data_12170837.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/51/159l_2/clim_data_12170837.zip
Checking for progress XML file: /var/lib/boinc/slots/51/progress_file_12170837.xml
Opened progress file ok : /var/lib/boinc/slots/51/progress_file_12170837.xml
-- Model is restarting --
Adjusting last_iter, 1151, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/51/progress_file_12170837.xml
last_cpu_time: 73572
upload_file_number: 47
last_iter: 1127
last_upload: 4060800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476
The child process has been launched with process id: 58
Executing the command: /var/lib/boinc/slots/51/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:58:58] [20221227:013521:1672104921:0.004] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0193
(argv4) batchid: 954
(argv5) wuid: 12170837
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/51
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/51/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/51/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_195604768c17856686e5f5335db4557a to: /var/lib/boinc/slots/51/oifs_43r3_ps_0193_1985050100_123_954_12170837.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/51/oifs_43r3_ps_0193_1985050100_123_954_12170837.zip
ic_ancil_file: ic_ancil_12170837
ifsdata_file: ifsdata_12170837
climate_data_file: clim_data_12170837
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_58bde664c39c1b869542abb4827bddc4 to: /var/lib/boinc/slots/51/ic_ancil_12170837.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/51/ic_ancil_12170837.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_27b16cfe92038e0b10b6510dec0c3835 to: /var/lib/boinc/slots/51/ifsdata/ifsdata_12170837.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/51/ifsdata/ifsdata_12170837.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/51/159l_2/clim_data_12170837.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/51/159l_2/clim_data_12170837.zip
Checking for progress XML file: /var/lib/boinc/slots/51/progress_file_12170837.xml
Opened progress file ok : /var/lib/boinc/slots/51/progress_file_12170837.xml
-- Model is restarting --
Adjusting last_iter, 1128, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/51/progress_file_12170837.xml
last_cpu_time: 73638
upload_file_number: 47
last_iter: 1127
last_upload: 4060800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476
The child process has been launched with process id: 55
Executing the command: /var/lib/boinc/slots/51/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:022623:1672107983:0.004] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0193
(argv4) batchid: 954
(argv5) wuid: 12170837
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/51
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/51/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/51/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_195604768c17856686e5f5335db4557a to: /var/lib/boinc/slots/51/oifs_43r3_ps_0193_1985050100_123_954_12170837.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/51/oifs_43r3_ps_0193_1985050100_123_954_12170837.zip
ic_ancil_file: ic_ancil_12170837
ifsdata_file: ifsdata_12170837
climate_data_file: clim_data_12170837
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_58bde664c39c1b869542abb4827bddc4 to: /var/lib/boinc/slots/51/ic_ancil_12170837.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/51/ic_ancil_12170837.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_27b16cfe92038e0b10b6510dec0c3835 to: /var/lib/boinc/slots/51/ifsdata/ifsdata_12170837.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/51/ifsdata/ifsdata_12170837.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/51/159l_2/clim_data_12170837.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/51/159l_2/clim_data_12170837.zip
Checking for progress XML file: /var/lib/boinc/slots/51/progress_file_12170837.xml
Opened progress file ok : /var/lib/boinc/slots/51/progress_file_12170837.xml
-- Model is restarting --
Adjusting last_iter, 1128, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/51/progress_file_12170837.xml
last_cpu_time: 73651
upload_file_number: 47
last_iter: 1127
last_upload: 4060800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476
The child process has been launched with process id: 54
Executing the command: /var/lib/boinc/slots/51/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:54:54] [20221227:023540:1672108540:0.003] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:46:11 STEP 1128 H=1128:00 +CPU=104.309
  02:51:44 STEP 1129 H=1129:00 +CPU= 57.489
  02:57:54 STEP 1130 H=1130:00 +CPU= 62.218
  03:07:16 STEP 1131 H=1131:00 +CPU= 93.183
  03:13:20 STEP 1132 H=1132:00 +CPU= 61.375
  03:19:00 STEP 1133 H=1133:00 +CPU= 58.119
  03:28:52 STEP 1134 H=1134:00 +CPU=102.803
  03:34:33 STEP 1135 H=1135:00 +CPU= 58.036
  03:40:28 STEP 1136 H=1136:00 +CPU= 61.169
  03:48:44 STEP 1137 H=1137:00 +CPU= 85.679
  03:54:24 STEP 1138 H=1138:00 +CPU= 57.236
  03:59:20 STEP 1139 H=1139:00 +CPU= 50.882
  04:09:41 STEP 1140 H=1140:00 +CPU=105.665
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001140
  04:15:58 STEP 1141 H=1141:00 +CPU= 62.834
  04:22:32 STEP 1142 H=1142:00 +CPU= 66.440
  04:32:30 STEP 1143 H=1143:00 +CPU=100.058
  04:38:43 STEP 1144 H=1144:00 +CPU= 60.735
  04:44:48 STEP 1145 H=1145:00 +CPU= 61.351
  04:53:20 STEP 1146 H=1146:00 +CPU= 88.489
  04:58:22 STEP 1147 H=1147:00 +CPU= 52.584
  05:03:12 STEP 1148 H=1148:00 +CPU= 50.510
  05:10:59 STEP 1149 H=1149:00 +CPU= 79.308
  05:15:58 STEP 1150 H=1150:00 +CPU= 52.066
  05:20:46 STEP 1151 H=1151:00 +CPU= 50.188
  05:30:02 STEP 1152 H=1152:00 +CPU= 93.195
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_47.zip
Uploading the intermediate file: upload_file_47.zip
  05:34:38 STEP 1153 H=1153:00 +CPU= 48.285
  05:39:33 STEP 1154 H=1154:00 +CPU= 49.848
  05:46:42 STEP 1155 H=1155:00 +CPU= 74.574
  05:51:34 STEP 1156 H=1156:00 +CPU= 50.627
  05:56:22 STEP 1157 H=1157:00 +CPU= 49.659
  06:03:53 STEP 1158 H=1158:00 +CPU= 78.656
  06:08:37 STEP 1159 H=1159:00 +CPU= 48.993
  06:14:02 STEP 1160 H=1160:00 +CPU= 54.275
  06:21:29 STEP 1161 H=1161:00 +CPU= 77.037
  06:26:23 STEP 1162 H=1162:00 +CPU= 49.655
  06:30:58 STEP 1163 H=1163:00 +CPU= 46.696
  06:40:04 STEP 1164 H=1164:00 +CPU= 88.695
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001164
  06:45:23 STEP 1165 H=1165:00 +CPU= 50.902
  06:50:19 STEP 1166 H=1166:00 +CPU= 49.868
  06:57:22 STEP 1167 H=1167:00 +CPU= 73.624
  07:02:09 STEP 1168 H=1168:00 +CPU= 50.086
  07:07:09 STEP 1169 H=1169:00 +CPU= 50.568
  07:15:06 STEP 1170 H=1170:00 +CPU= 81.129
  07:19:50 STEP 1171 H=1171:00 +CPU= 47.599
  07:24:19 STEP 1172 H=1172:00 +CPU= 47.078
  07:31:36 STEP 1173 H=1173:00 +CPU= 75.548
  07:36:32 STEP 1174 H=1174:00 +CPU= 50.854
  07:41:14 STEP 1175 H=1175:00 +CPU= 48.821
  07:49:40 STEP 1176 H=1176:00 +CPU= 86.790
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_48.zip
Uploading the intermediate file: upload_file_48.zip
  07:54:24 STEP 1177 H=1177:00 +CPU= 47.318
  07:59:09 STEP 1178 H=1178:00 +CPU= 49.478
  08:07:11 STEP 1179 H=1179:00 +CPU= 80.240
  08:12:08 STEP 1180 H=1180:00 +CPU= 50.644
  08:17:07 STEP 1181 H=1181:00 +CPU= 49.408
  08:24:08 STEP 1182 H=1182:00 +CPU= 73.272
  08:28:34 STEP 1183 H=1183:00 +CPU= 46.056
  08:33:19 STEP 1184 H=1184:00 +CPU= 48.796
  08:40:55 STEP 1185 H=1185:00 +CPU= 75.925
  08:45:56 STEP 1186 H=1186:00 +CPU= 51.719
  08:50:29 STEP 1187 H=1187:00 +CPU= 46.379
  08:58:24 STEP 1188 H=1188:00 +CPU= 81.883
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001188
  09:03:09 STEP 1189 H=1189:00 +CPU= 48.825
  09:07:49 STEP 1190 H=1190:00 +CPU= 48.775
  09:15:14 STEP 1191 H=1191:00 +CPU= 75.215
  09:19:44 STEP 1192 H=1192:00 +CPU= 46.173
  09:24:18 STEP 1193 H=1193:00 +CPU= 47.224
  09:32:01 STEP 1194 H=1194:00 +CPU= 77.622
  09:36:40 STEP 1195 H=1195:00 +CPU= 48.661
  09:41:41 STEP 1196 H=1196:00 +CPU= 51.310
  09:49:06 STEP 1197 H=1197:00 +CPU= 77.375
  09:53:56 STEP 1198 H=1198:00 +CPU= 50.103
  09:58:39 STEP 1199 H=1199:00 +CPU= 47.805
  10:07:12 STEP 1200 H=1200:00 +CPU= 87.441
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_49.zip
Uploading the intermediate file: upload_file_49.zip
  10:11:55 STEP 1201 H=1201:00 +CPU= 48.416
  10:16:46 STEP 1202 H=1202:00 +CPU= 48.896
  10:24:35 STEP 1203 H=1203:00 +CPU= 77.106
  10:29:46 STEP 1204 H=1204:00 +CPU= 51.356
  10:35:06 STEP 1205 H=1205:00 +CPU= 49.958
  10:42:15 STEP 1206 H=1206:00 +CPU= 74.926
  10:47:02 STEP 1207 H=1207:00 +CPU= 48.428
  10:51:57 STEP 1208 H=1208:00 +CPU= 50.031
  10:59:22 STEP 1209 H=1209:00 +CPU= 76.273
  11:04:24 STEP 1210 H=1210:00 +CPU= 51.197
  11:09:22 STEP 1211 H=1211:00 +CPU= 48.987
  11:17:43 STEP 1212 H=1212:00 +CPU= 84.309
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001212
  11:22:14 STEP 1213 H=1213:00 +CPU= 46.549
  11:26:59 STEP 1214 H=1214:00 +CPU= 48.274
  11:34:07 STEP 1215 H=1215:00 +CPU= 74.445
  11:39:00 STEP 1216 H=1216:00 +CPU= 50.723
  11:44:10 STEP 1217 H=1217:00 +CPU= 50.037
  11:52:16 STEP 1218 H=1218:00 +CPU= 80.869
  11:56:43 STEP 1219 H=1219:00 +CPU= 46.239
  12:01:21 STEP 1220 H=1220:00 +CPU= 48.444
  12:09:09 STEP 1221 H=1221:00 +CPU= 75.934
  12:14:42 STEP 1222 H=1222:00 +CPU= 52.447
  12:19:40 STEP 1223 H=1223:00 +CPU= 51.395
  12:28:24 STEP 1224 H=1224:00 +CPU= 88.762
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_50.zip
Uploading the intermediate file: upload_file_50.zip
  12:33:21 STEP 1225 H=1225:00 +CPU= 50.106
  12:38:08 STEP 1226 H=1226:00 +CPU= 49.285
  12:45:18 STEP 1227 H=1227:00 +CPU= 73.648
  12:50:04 STEP 1228 H=1228:00 +CPU= 48.240
  12:55:11 STEP 1229 H=1229:00 +CPU= 49.796
  13:02:38 STEP 1230 H=1230:00 +CPU= 77.011
  13:07:10 STEP 1231 H=1231:00 +CPU= 46.698
  13:11:42 STEP 1232 H=1232:00 +CPU= 47.139
  13:18:55 STEP 1233 H=1233:00 +CPU= 74.610
  13:23:19 STEP 1234 H=1234:00 +CPU= 46.210
  13:28:01 STEP 1235 H=1235:00 +CPU= 47.250
  13:36:44 STEP 1236 H=1236:00 +CPU= 87.153
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001236
  13:41:51 STEP 1237 H=1237:00 +CPU= 49.780
  13:46:58 STEP 1238 H=1238:00 +CPU= 50.599
  13:54:19 STEP 1239 H=1239:00 +CPU= 75.162
  13:59:04 STEP 1240 H=1240:00 +CPU= 49.285
  14:03:17 STEP 1241 H=1241:00 +CPU= 44.285
  14:10:41 STEP 1242 H=1242:00 +CPU= 75.380
  14:14:56 STEP 1243 H=1243:00 +CPU= 44.621
  14:19:53 STEP 1244 H=1244:00 +CPU= 50.798
  14:27:23 STEP 1245 H=1245:00 +CPU= 77.159
  14:32:19 STEP 1246 H=1246:00 +CPU= 49.860
  14:37:11 STEP 1247 H=1247:00 +CPU= 48.944
  14:45:50 STEP 1248 H=1248:00 +CPU= 88.168
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_51.zip
Uploading the intermediate file: upload_file_51.zip
  14:50:42 STEP 1249 H=1249:00 +CPU= 48.031
  14:55:29 STEP 1250 H=1250:00 +CPU= 48.791
  15:03:08 STEP 1251 H=1251:00 +CPU= 77.810
  15:08:11 STEP 1252 H=1252:00 +CPU= 52.188
  15:13:13 STEP 1253 H=1253:00 +CPU= 49.948
  15:21:10 STEP 1254 H=1254:00 +CPU= 80.972
  15:25:54 STEP 1255 H=1255:00 +CPU= 47.565
  15:30:34 STEP 1256 H=1256:00 +CPU= 48.068
  15:38:15 STEP 1257 H=1257:00 +CPU= 76.262
  15:43:19 STEP 1258 H=1258:00 +CPU= 50.367
  15:47:44 STEP 1259 H=1259:00 +CPU= 45.690
  15:56:22 STEP 1260 H=1260:00 +CPU= 86.690
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001260
  16:01:18 STEP 1261 H=1261:00 +CPU= 49.798
  16:06:21 STEP 1262 H=1262:00 +CPU= 50.294
  16:13:50 STEP 1263 H=1263:00 +CPU= 76.037
  16:19:14 STEP 1264 H=1264:00 +CPU= 51.943
  16:24:20 STEP 1265 H=1265:00 +CPU= 49.801
  16:32:06 STEP 1266 H=1266:00 +CPU= 78.936
  16:37:02 STEP 1267 H=1267:00 +CPU= 49.895
  16:42:18 STEP 1268 H=1268:00 +CPU= 53.033
  16:50:01 STEP 1269 H=1269:00 +CPU= 77.691
  16:55:00 STEP 1270 H=1270:00 +CPU= 50.908
  16:59:48 STEP 1271 H=1271:00 +CPU= 48.940
  17:08:20 STEP 1272 H=1272:00 +CPU= 88.272
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_52.zip
Uploading the intermediate file: upload_file_52.zip
  17:13:04 STEP 1273 H=1273:00 +CPU= 47.903
  17:18:14 STEP 1274 H=1274:00 +CPU= 49.820
  17:26:02 STEP 1275 H=1275:00 +CPU= 77.148
  17:31:10 STEP 1276 H=1276:00 +CPU= 50.603
  17:35:24 STEP 1277 H=1277:00 +CPU= 44.237
  17:43:00 STEP 1278 H=1278:00 +CPU= 78.036
  17:47:50 STEP 1279 H=1279:00 +CPU= 49.689
  17:52:51 STEP 1280 H=1280:00 +CPU= 51.895
  18:00:31 STEP 1281 H=1281:00 +CPU= 76.756
  18:05:55 STEP 1282 H=1282:00 +CPU= 51.589
  18:10:51 STEP 1283 H=1283:00 +CPU= 49.456
  18:19:07 STEP 1284 H=1284:00 +CPU= 83.896
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001284
  18:24:04 STEP 1285 H=1285:00 +CPU= 49.158
  18:29:19 STEP 1286 H=1286:00 +CPU= 50.323
  18:36:21 STEP 1287 H=1287:00 +CPU= 73.793
  18:40:49 STEP 1288 H=1288:00 +CPU= 46.486
  18:45:23 STEP 1289 H=1289:00 +CPU= 47.906
  18:53:09 STEP 1290 H=1290:00 +CPU= 78.719
  18:58:01 STEP 1291 H=1291:00 +CPU= 49.570
  19:03:04 STEP 1292 H=1292:00 +CPU= 50.536
  19:10:40 STEP 1293 H=1293:00 +CPU= 77.224
  19:15:39 STEP 1294 H=1294:00 +CPU= 49.497
  19:20:37 STEP 1295 H=1295:00 +CPU= 50.234
  19:29:12 STEP 1296 H=1296:00 +CPU= 87.838
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_53.zip
Uploading the intermediate file: upload_file_53.zip
  19:34:19 STEP 1297 H=1297:00 +CPU= 48.119
  19:39:03 STEP 1298 H=1298:00 +CPU= 47.937
  19:46:47 STEP 1299 H=1299:00 +CPU= 76.778
  19:51:27 STEP 1300 H=1300:00 +CPU= 47.455
  19:56:23 STEP 1301 H=1301:00 +CPU= 49.149
  20:04:08 STEP 1302 H=1302:00 +CPU= 77.377
  20:09:06 STEP 1303 H=1303:00 +CPU= 49.246
  20:13:34 STEP 1304 H=1304:00 +CPU= 46.828
  20:20:57 STEP 1305 H=1305:00 +CPU= 76.872
  20:25:47 STEP 1306 H=1306:00 +CPU= 49.945
  20:30:17 STEP 1307 H=1307:00 +CPU= 46.103
  20:38:56 STEP 1308 H=1308:00 +CPU= 85.484
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001308
  20:44:10 STEP 1309 H=1309:00 +CPU= 49.373
  20:49:20 STEP 1310 H=1310:00 +CPU= 51.799
  20:56:51 STEP 1311 H=1311:00 +CPU= 76.282
  21:01:55 STEP 1312 H=1312:00 +CPU= 51.247
  21:06:40 STEP 1313 H=1313:00 +CPU= 49.448
  21:14:10 STEP 1314 H=1314:00 +CPU= 76.569
  21:18:51 STEP 1315 H=1315:00 +CPU= 47.139
  21:23:34 STEP 1316 H=1316:00 +CPU= 48.276
  21:31:10 STEP 1317 H=1317:00 +CPU= 75.835
  21:36:00 STEP 1318 H=1318:00 +CPU= 48.004
  21:41:03 STEP 1319 H=1319:00 +CPU= 49.185
  21:49:38 STEP 1320 H=1320:00 +CPU= 85.728
Moving to projects directory: /var/lib/boinc/slots/51/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/51/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/51/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12170837/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0193_1985050100_123_954_12170837_0_r1043582476_54.zip
Uploading the intermediate file: upload_file_54.zip
  21:54:25 STEP 1321 H=1321:00 +CPU= 46.915
  21:59:22 STEP 1322 H=1322:00 +CPU= 49.054
  22:07:09 STEP 1323 H=1323:00 +CPU= 76.095
  22:12:13 STEP 1324 H=1324:00 +CPU= 49.986
  22:17:00 STEP 1325 H=1325:00 +CPU= 48.331
  22:25:11 STEP 1326 H=1326:00 +CPU= 81.031
  22:30:12 STEP 1327 H=1327:00 +CPU= 48.698
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:56:28 1488917 22255165 oifs_43r3_ps_0193_1985050100_123_954_12170837_0 3,456,000 61,895 0.0179
09 May 2023 15:54:16 1488917 22255165 oifs_43r3_ps_0193_1985050100_123_954_12170837_0 2,592,000 47,327 0.0183
09 May 2023 15:51:47 1488917 22255165 oifs_43r3_ps_0193_1985050100_123_954_12170837_0 1,728,000 32,950 0.0191
09 May 2023 15:49:42 1488917 22255165 oifs_43r3_ps_0193_1985050100_123_954_12170837_0 864,000 19,647 0.0227


©2024 climateprediction.net