Name | oifs_43r3_bl_a0s7_2016092300_20_1018_12289342_0 |
Workunit | 12289342 |
Created | 12 Jun 2024, 21:26:17 UTC |
Sent | 14 Jun 2024, 6:34:15 UTC |
Report deadline | 13 Aug 2024, 6:34:15 UTC |
Received | 14 Jun 2024, 13:07:51 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 1 (0x00000001) Unknown error code |
Computer ID | 1541000 |
Run time | 5 hours 41 min 34 sec |
CPU time | 5 hours 40 min 3 sec |
Validate state | Invalid |
Credit | 1,318.46 |
Device peak FLOPS | 4.91 GFLOPS |
Application version | OpenIFS 43r3 Baroclinic Lifecycle v1.13 x86_64-pc-linux-gnu |
Peak working set size | 5,025.16 MB |
Peak swap size | 5,981.07 MB |
Peak disk usage | 1,286.23 MB |
Stderr | <core_client_version>8.0.2</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> 6 03:22:55 STEP 937 H= 234:15 +CPU= 14.226 03:23:10 STEP 938 H= 234:30 +CPU= 14.458 03:23:24 STEP 939 H= 234:45 +CPU= 14.601 03:23:39 STEP 940 H= 235:00 +CPU= 14.568 03:23:53 STEP 941 H= 235:15 +CPU= 14.133 03:24:07 STEP 942 H= 235:30 +CPU= 14.222 03:24:21 STEP 943 H= 235:45 +CPU= 13.938 03:24:35 STEP 944 H= 236:00 +CPU= 14.220 03:24:50 STEP 945 H= 236:15 +CPU= 14.223 03:25:04 STEP 946 H= 236:30 +CPU= 14.454 03:25:19 STEP 947 H= 236:45 +CPU= 14.464 03:25:36 STEP 948 H= 237:00 +CPU= 17.728 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000948 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000948 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000948 03:25:51 STEP 949 H= 237:15 +CPU= 14.624 03:26:06 STEP 950 H= 237:30 +CPU= 14.685 03:26:20 STEP 951 H= 237:45 +CPU= 14.502 03:26:35 STEP 952 H= 238:00 +CPU= 14.690 03:26:50 STEP 953 H= 238:15 +CPU= 14.507 03:27:04 STEP 954 H= 238:30 +CPU= 14.419 03:27:18 STEP 955 H= 238:45 +CPU= 14.359 03:27:33 STEP 956 H= 239:00 +CPU= 14.440 03:27:47 STEP 957 H= 239:15 +CPU= 14.434 03:28:02 STEP 958 H= 239:30 +CPU= 14.651 03:28:17 STEP 959 H= 239:45 +CPU= 14.726 03:28:37 STEP 960 H= 240:00 +CPU= 20.051 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000960 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000960 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000876 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000876 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000876 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000948 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000948 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000948 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0s7_2016092300_20_1018_12289342_0_r792353749_9.zip Uploading the intermediate file: upload_file_9.zip 03:28:51 STEP 961 H= 240:15 +CPU= 14.296 Uploading trickle at timestep: 864000 03:29:06 STEP 962 H= 240:30 +CPU= 14.780 03:29:21 STEP 963 H= 240:45 +CPU= 14.833 03:29:35 STEP 964 H= 241:00 +CPU= 14.736 03:29:50 STEP 965 H= 241:15 +CPU= 14.211 03:30:04 STEP 966 H= 241:30 +CPU= 14.216 03:30:18 STEP 967 H= 241:45 +CPU= 14.399 03:30:33 STEP 968 H= 242:00 +CPU= 14.321 03:30:47 STEP 969 H= 242:15 +CPU= 14.421 03:31:01 STEP 970 H= 242:30 +CPU= 14.332 03:31:16 STEP 971 H= 242:45 +CPU= 14.386 03:31:34 STEP 972 H= 243:00 +CPU= 18.020 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000972 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000972 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000972 03:31:48 STEP 973 H= 243:15 +CPU= 14.210 03:32:02 STEP 974 H= 243:30 +CPU= 14.337 03:32:17 STEP 975 H= 243:45 +CPU= 14.400 03:32:31 STEP 976 H= 244:00 +CPU= 14.523 03:32:45 STEP 977 H= 244:15 +CPU= 14.064 03:33:00 STEP 978 H= 244:30 +CPU= 14.395 03:33:15 STEP 979 H= 244:45 +CPU= 14.622 03:33:29 STEP 980 H= 245:00 +CPU= 14.826 03:33:44 STEP 981 H= 245:15 +CPU= 14.336 03:33:58 STEP 982 H= 245:30 +CPU= 14.582 03:34:13 STEP 983 H= 245:45 +CPU= 14.512 03:34:31 STEP 984 H= 246:00 +CPU= 17.992 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000984 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000984 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000984 03:34:45 STEP 985 H= 246:15 +CPU= 14.231 03:35:00 STEP 986 H= 246:30 +CPU= 14.433 03:35:14 STEP 987 H= 246:45 +CPU= 14.688 03:35:29 STEP 988 H= 247:00 +CPU= 14.757 03:35:44 STEP 989 H= 247:15 +CPU= 14.537 03:35:58 STEP 990 H= 247:30 +CPU= 14.312 03:36:12 STEP 991 H= 247:45 +CPU= 14.357 03:36:27 STEP 992 H= 248:00 +CPU= 14.773 03:36:41 STEP 993 H= 248:15 +CPU= 14.139 03:36:56 STEP 994 H= 248:30 +CPU= 14.530 03:37:10 STEP 995 H= 248:45 +CPU= 14.293 03:37:28 STEP 996 H= 249:00 +CPU= 18.011 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000996 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000996 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000996 03:37:42 STEP 997 H= 249:15 +CPU= 14.230 03:37:57 STEP 998 H= 249:30 +CPU= 14.318 03:38:11 STEP 999 H= 249:45 +CPU= 14.294 03:38:26 STEP 1000 H= 250:00 +CPU= 14.975 03:38:40 STEP 1001 H= 250:15 +CPU= 14.328 03:38:55 STEP 1002 H= 250:30 +CPU= 14.596 03:39:10 STEP 1003 H= 250:45 +CPU= 14.687 03:39:24 STEP 1004 H= 251:00 +CPU= 14.713 03:39:39 STEP 1005 H= 251:15 +CPU= 14.349 03:39:53 STEP 1006 H= 251:30 +CPU= 14.593 03:40:08 STEP 1007 H= 251:45 +CPU= 14.364 03:40:27 STEP 1008 H= 252:00 +CPU= 19.658 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001008 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001008 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001008 03:40:42 STEP 1009 H= 252:15 +CPU= 14.629 03:40:56 STEP 1010 H= 252:30 +CPU= 14.333 03:41:11 STEP 1011 H= 252:45 +CPU= 14.533 03:41:26 STEP 1012 H= 253:00 +CPU= 14.744 03:41:40 STEP 1013 H= 253:15 +CPU= 14.368 03:41:54 STEP 1014 H= 253:30 +CPU= 14.296 03:42:09 STEP 1015 H= 253:45 +CPU= 14.080 03:42:23 STEP 1016 H= 254:00 +CPU= 14.647 03:42:38 STEP 1017 H= 254:15 +CPU= 14.361 03:42:52 STEP 1018 H= 254:30 +CPU= 14.273 03:43:06 STEP 1019 H= 254:45 +CPU= 14.400 03:43:25 STEP 1020 H= 255:00 +CPU= 18.350 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001020 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001020 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001020 03:43:39 STEP 1021 H= 255:15 +CPU= 14.765 03:43:54 STEP 1022 H= 255:30 +CPU= 14.644 03:44:08 STEP 1023 H= 255:45 +CPU= 14.365 03:44:23 STEP 1024 H= 256:00 +CPU= 14.805 03:44:38 STEP 1025 H= 256:15 +CPU= 14.694 03:44:52 STEP 1026 H= 256:30 +CPU= 14.391 03:45:07 STEP 1027 H= 256:45 +CPU= 14.691 03:45:22 STEP 1028 H= 257:00 +CPU= 14.604 03:45:36 STEP 1029 H= 257:15 +CPU= 14.611 03:45:51 STEP 1030 H= 257:30 +CPU= 14.420 03:46:05 STEP 1031 H= 257:45 +CPU= 14.688 03:46:23 STEP 1032 H= 258:00 +CPU= 17.908 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001032 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001032 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001032 03:46:38 STEP 1033 H= 258:15 +CPU= 14.476 03:46:52 STEP 1034 H= 258:30 +CPU= 13.796 03:47:06 STEP 1035 H= 258:45 +CPU= 14.426 03:47:21 STEP 1036 H= 259:00 +CPU= 14.797 03:47:36 STEP 1037 H= 259:15 +CPU= 14.585 03:47:50 STEP 1038 H= 259:30 +CPU= 14.297 03:48:04 STEP 1039 H= 259:45 +CPU= 14.275 03:48:19 STEP 1040 H= 260:00 +CPU= 14.550 03:48:33 STEP 1041 H= 260:15 +CPU= 14.242 03:48:47 STEP 1042 H= 260:30 +CPU= 14.395 03:49:01 STEP 1043 H= 260:45 +CPU= 14.101 03:49:20 STEP 1044 H= 261:00 +CPU= 18.388 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001044 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001044 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001044 03:49:34 STEP 1045 H= 261:15 +CPU= 14.407 03:49:49 STEP 1046 H= 261:30 +CPU= 14.280 03:50:03 STEP 1047 H= 261:45 +CPU= 14.261 03:50:18 STEP 1048 H= 262:00 +CPU= 14.629 03:50:32 STEP 1049 H= 262:15 +CPU= 14.469 03:50:46 STEP 1050 H= 262:30 +CPU= 14.317 03:51:01 STEP 1051 H= 262:45 +CPU= 14.305 03:51:15 STEP 1052 H= 263:00 +CPU= 14.555 03:51:30 STEP 1053 H= 263:15 +CPU= 14.431 03:51:44 STEP 1054 H= 263:30 +CPU= 14.092 03:51:58 STEP 1055 H= 263:45 +CPU= 14.148 03:52:18 STEP 1056 H= 264:00 +CPU= 19.685 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001056 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001056 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001020 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001020 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001020 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001044 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001044 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001044 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0s7_2016092300_20_1018_12289342_0_r792353749_10.zip Uploading the intermediate file: upload_file_10.zip 03:52:32 STEP 1057 H= 264:15 +CPU= 14.489 03:52:47 STEP 1058 H= 264:30 +CPU= 14.282 03:53:01 STEP 1059 H= 264:45 +CPU= 14.087 03:53:15 STEP 1060 H= 265:00 +CPU= 14.727 03:53:30 STEP 1061 H= 265:15 +CPU= 14.363 03:53:44 STEP 1062 H= 265:30 +CPU= 14.301 03:53:59 STEP 1063 H= 265:45 +CPU= 14.432 03:54:13 STEP 1064 H= 266:00 +CPU= 14.800 03:54:28 STEP 1065 H= 266:15 +CPU= 14.576 03:54:42 STEP 1066 H= 266:30 +CPU= 14.423 03:54:57 STEP 1067 H= 266:45 +CPU= 14.216 03:55:14 STEP 1068 H= 267:00 +CPU= 17.538 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001068 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001068 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001068 03:55:29 STEP 1069 H= 267:15 +CPU= 14.499 03:55:43 STEP 1070 H= 267:30 +CPU= 14.012 03:55:57 STEP 1071 H= 267:45 +CPU= 14.602 03:56:12 STEP 1072 H= 268:00 +CPU= 14.486 03:56:26 STEP 1073 H= 268:15 +CPU= 14.176 03:56:41 STEP 1074 H= 268:30 +CPU= 14.576 03:56:55 STEP 1075 H= 268:45 +CPU= 14.217 03:57:10 STEP 1076 H= 269:00 +CPU= 14.704 03:57:24 STEP 1077 H= 269:15 +CPU= 14.206 03:57:38 STEP 1078 H= 269:30 +CPU= 14.141 03:57:52 STEP 1079 H= 269:45 +CPU= 14.501 03:58:11 STEP 1080 H= 270:00 +CPU= 18.126 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001080 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001080 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001080 03:58:25 STEP 1081 H= 270:15 +CPU= 14.208 03:58:39 STEP 1082 H= 270:30 +CPU= 14.076 03:58:53 STEP 1083 H= 270:45 +CPU= 14.068 03:59:07 STEP 1084 H= 271:00 +CPU= 14.497 03:59:22 STEP 1085 H= 271:15 +CPU= 14.768 03:59:37 STEP 1086 H= 271:30 +CPU= 14.793 03:59:52 STEP 1087 H= 271:45 +CPU= 14.740 04:00:06 STEP 1088 H= 272:00 +CPU= 14.717 04:00:21 STEP 1089 H= 272:15 +CPU= 14.510 04:00:36 STEP 1090 H= 272:30 +CPU= 14.821 04:00:50 STEP 1091 H= 272:45 +CPU= 14.563 04:01:09 STEP 1092 H= 273:00 +CPU= 18.231 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001092 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001092 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001092 04:01:23 STEP 1093 H= 273:15 +CPU= 14.696 04:01:37 STEP 1094 H= 273:30 +CPU= 14.019 04:01:52 STEP 1095 H= 273:45 +CPU= 14.586 04:02:06 STEP 1096 H= 274:00 +CPU= 14.230 04:02:21 STEP 1097 H= 274:15 +CPU= 14.479 04:02:35 STEP 1098 H= 274:30 +CPU= 14.256 04:02:49 STEP 1099 H= 274:45 +CPU= 14.393 04:03:04 STEP 1100 H= 275:00 +CPU= 14.857 04:03:19 STEP 1101 H= 275:15 +CPU= 14.357 04:03:33 STEP 1102 H= 275:30 +CPU= 14.473 04:03:48 STEP 1103 H= 275:45 +CPU= 14.588 04:04:08 STEP 1104 H= 276:00 +CPU= 20.060 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001104 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001104 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001104 04:04:22 STEP 1105 H= 276:15 +CPU= 14.488 04:04:37 STEP 1106 H= 276:30 +CPU= 14.258 04:04:51 STEP 1107 H= 276:45 +CPU= 14.319 04:05:06 STEP 1108 H= 277:00 +CPU= 14.677 04:05:20 STEP 1109 H= 277:15 +CPU= 14.314 04:05:34 STEP 1110 H= 277:30 +CPU= 14.130 04:05:49 STEP 1111 H= 277:45 +CPU= 14.463 04:06:03 STEP 1112 H= 278:00 +CPU= 14.427 04:06:17 STEP 1113 H= 278:15 +CPU= 14.272 04:06:32 STEP 1114 H= 278:30 +CPU= 14.260 04:06:46 STEP 1115 H= 278:45 +CPU= 14.618 04:07:05 STEP 1116 H= 279:00 +CPU= 18.196 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001116 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001116 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001116 04:07:19 STEP 1117 H= 279:15 +CPU= 14.580 04:07:34 STEP 1118 H= 279:30 +CPU= 14.654 04:07:48 STEP 1119 H= 279:45 +CPU= 14.568 04:08:03 STEP 1120 H= 280:00 +CPU= 15.000 04:08:18 STEP 1121 H= 280:15 +CPU= 14.467 04:08:32 STEP 1122 H= 280:30 +CPU= 14.295 04:08:47 STEP 1123 H= 280:45 +CPU= 14.373 04:09:01 STEP 1124 H= 281:00 +CPU= 14.432 04:09:15 STEP 1125 H= 281:15 +CPU= 14.251 04:09:30 STEP 1126 H= 281:30 +CPU= 14.425 04:09:44 STEP 1127 H= 281:45 +CPU= 14.089 04:10:02 STEP 1128 H= 282:00 +CPU= 18.113 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001128 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001128 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001128 04:10:16 STEP 1129 H= 282:15 +CPU= 14.461 04:10:31 STEP 1130 H= 282:30 +CPU= 14.500 04:10:46 STEP 1131 H= 282:45 +CPU= 14.589 04:11:01 STEP 1132 H= 283:00 +CPU= 14.870 04:11:15 STEP 1133 H= 283:15 +CPU= 14.641 04:11:30 STEP 1134 H= 283:30 +CPU= 14.497 04:11:44 STEP 1135 H= 283:45 +CPU= 14.551 04:11:59 STEP 1136 H= 284:00 +CPU= 14.839 04:12:14 STEP 1137 H= 284:15 +CPU= 14.693 04:12:29 STEP 1138 H= 284:30 +CPU= 14.663 04:12:43 STEP 1139 H= 284:45 +CPU= 14.135 04:13:00 STEP 1140 H= 285:00 +CPU= 17.609 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001140 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001140 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001140 04:13:15 STEP 1141 H= 285:15 +CPU= 14.460 04:13:29 STEP 1142 H= 285:30 +CPU= 14.355 04:13:43 STEP 1143 H= 285:45 +CPU= 14.168 04:13:58 STEP 1144 H= 286:00 +CPU= 14.598 04:14:12 STEP 1145 H= 286:15 +CPU= 14.412 04:14:27 STEP 1146 H= 286:30 +CPU= 14.903 04:14:41 STEP 1147 H= 286:45 +CPU= 14.130 04:14:56 STEP 1148 H= 287:00 +CPU= 14.964 04:15:11 STEP 1149 H= 287:15 +CPU= 14.680 04:15:26 STEP 1150 H= 287:30 +CPU= 14.515 04:15:40 STEP 1151 H= 287:45 +CPU= 14.180 04:15:59 STEP 1152 H= 288:00 +CPU= 19.470 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001152 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001152 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001140 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001140 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001140 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0s7_2016092300_20_1018_12289342_0_r792353749_11.zip Uploading the intermediate file: upload_file_11.zip 04:16:13 STEP 1153 H= 288:15 +CPU= 14.141 SMILAG TRAJECTORY UNDERGROUND 1 TIMES. POINT= 17 MAX ETADOT VERTICAL VEL.= -7.406460502417728E-012 LON = 21.3750000000000 degrees LAT = -5.04670442015713 degrees 04:16:28 STEP 1154 H= 288:30 +CPU= 14.190 04:16:42 STEP 1155 H= 288:45 +CPU= 14.515 04:16:57 STEP 1156 H= 289:00 +CPU= 14.451 04:17:11 STEP 1157 H= 289:15 +CPU= 14.343 04:17:26 STEP 1158 H= 289:30 +CPU= 14.500 04:17:40 STEP 1159 H= 289:45 +CPU= 13.916 04:17:54 STEP 1160 H= 290:00 +CPU= 14.389 04:18:09 STEP 1161 H= 290:15 +CPU= 14.652 04:18:23 STEP 1162 H= 290:30 +CPU= 14.564 04:18:38 STEP 1163 H= 290:45 +CPU= 14.455 04:18:55 STEP 1164 H= 291:00 +CPU= 17.753 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001164 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001164 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001164 04:19:10 STEP 1165 H= 291:15 +CPU= 14.400 04:19:24 STEP 1166 H= 291:30 +CPU= 14.168 04:19:38 STEP 1167 H= 291:45 +CPU= 14.245 04:19:53 STEP 1168 H= 292:00 +CPU= 15.103 04:20:08 STEP 1169 H= 292:15 +CPU= 14.270 04:20:22 STEP 1170 H= 292:30 +CPU= 14.390 04:20:36 STEP 1171 H= 292:45 +CPU= 13.917 04:20:51 STEP 1172 H= 293:00 +CPU= 14.749 04:21:05 STEP 1173 H= 293:15 +CPU= 14.414 04:21:20 STEP 1174 H= 293:30 +CPU= 14.474 04:21:34 STEP 1175 H= 293:45 +CPU= 14.367 04:21:52 STEP 1176 H= 294:00 +CPU= 17.992 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001176 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001176 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001176 04:22:07 STEP 1177 H= 294:15 +CPU= 14.783 04:22:21 STEP 1178 H= 294:30 +CPU= 14.409 04:22:36 STEP 1179 H= 294:45 +CPU= 14.369 04:22:50 STEP 1180 H= 295:00 +CPU= 14.730 04:23:05 STEP 1181 H= 295:15 +CPU= 14.428 04:23:19 STEP 1182 H= 295:30 +CPU= 14.260 04:23:33 STEP 1183 H= 295:45 +CPU= 14.089 04:23:48 STEP 1184 H= 296:00 +CPU= 14.985 04:24:03 STEP 1185 H= 296:15 +CPU= 14.412 04:24:17 STEP 1186 H= 296:30 +CPU= 13.978 04:24:31 STEP 1187 H= 296:45 +CPU= 14.117 04:24:48 STEP 1188 H= 297:00 +CPU= 17.500 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001188 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001188 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001188 04:25:03 STEP 1189 H= 297:15 +CPU= 14.513 04:25:17 STEP 1190 H= 297:30 +CPU= 14.579 04:25:32 STEP 1191 H= 297:45 +CPU= 14.444 04:25:46 STEP 1192 H= 298:00 +CPU= 14.201 04:26:01 STEP 1193 H= 298:15 +CPU= 14.371 04:26:15 STEP 1194 H= 298:30 +CPU= 14.895 04:26:30 STEP 1195 H= 298:45 +CPU= 14.294 04:26:45 STEP 1196 H= 299:00 +CPU= 14.994 04:26:59 STEP 1197 H= 299:15 +CPU= 14.691 04:27:14 STEP 1198 H= 299:30 +CPU= 14.240 04:27:28 STEP 1199 H= 299:45 +CPU= 14.493 04:27:48 STEP 1200 H= 300:00 +CPU= 19.566 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001200 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001200 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001200 04:28:02 STEP 1201 H= 300:15 +CPU= 14.275 04:28:17 STEP 1202 H= 300:30 +CPU= 14.455 04:28:31 STEP 1203 H= 300:45 +CPU= 14.508 04:28:45 STEP 1204 H= 301:00 +CPU= 14.138 04:28:59 STEP 1205 H= 301:15 +CPU= 13.934 04:29:14 STEP 1206 H= 301:30 +CPU= 14.566 04:29:28 STEP 1207 H= 301:45 +CPU= 14.013 04:29:43 STEP 1208 H= 302:00 +CPU= 14.718 04:29:58 STEP 1209 H= 302:15 +CPU= 14.990 04:30:12 STEP 1210 H= 302:30 +CPU= 14.617 04:30:27 STEP 1211 H= 302:45 +CPU= 14.736 04:30:45 STEP 1212 H= 303:00 +CPU= 17.750 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001212 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001212 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001212 04:30:59 STEP 1213 H= 303:15 +CPU= 14.286 04:31:14 STEP 1214 H= 303:30 +CPU= 14.687 04:31:28 STEP 1215 H= 303:45 +CPU= 14.412 04:31:42 STEP 1216 H= 304:00 +CPU= 14.227 04:31:57 STEP 1217 H= 304:15 +CPU= 14.658 04:32:12 STEP 1218 H= 304:30 +CPU= 14.520 04:32:26 STEP 1219 H= 304:45 +CPU= 14.392 04:32:41 STEP 1220 H= 305:00 +CPU= 14.583 04:32:55 STEP 1221 H= 305:15 +CPU= 14.195 04:33:09 STEP 1222 H= 305:30 +CPU= 14.614 04:33:24 STEP 1223 H= 305:45 +CPU= 14.686 04:33:42 STEP 1224 H= 306:00 +CPU= 18.113 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001224 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001224 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001224 04:33:57 STEP 1225 H= 306:15 +CPU= 14.440 04:34:11 STEP 1226 H= 306:30 +CPU= 14.585 04:34:26 STEP 1227 H= 306:45 +CPU= 14.545 04:34:40 STEP 1228 H= 307:00 +CPU= 14.523 04:34:55 STEP 1229 H= 307:15 +CPU= 14.849 04:35:10 STEP 1230 H= 307:30 +CPU= 14.449 04:35:24 STEP 1231 H= 307:45 +CPU= 14.524 04:35:39 STEP 1232 H= 308:00 +CPU= 14.328 04:35:53 STEP 1233 H= 308:15 +CPU= 14.398 04:36:08 STEP 1234 H= 308:30 +CPU= 14.609 04:36:22 STEP 1235 H= 308:45 +CPU= 13.795 04:36:40 STEP 1236 H= 309:00 +CPU= 17.980 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001236 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001236 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001236 04:36:54 STEP 1237 H= 309:15 +CPU= 14.360 04:37:08 STEP 1238 H= 309:30 +CPU= 14.384 04:37:23 STEP 1239 H= 309:45 +CPU= 14.412 04:37:37 STEP 1240 H= 310:00 +CPU= 14.693 04:37:52 STEP 1241 H= 310:15 +CPU= 14.312 04:38:06 STEP 1242 H= 310:30 +CPU= 14.230 04:38:20 STEP 1243 H= 310:45 +CPU= 14.385 04:38:35 STEP 1244 H= 311:00 +CPU= 15.009 04:38:50 STEP 1245 H= 311:15 +CPU= 14.284 04:39:04 STEP 1246 H= 311:30 +CPU= 14.405 04:39:19 STEP 1247 H= 311:45 +CPU= 14.501 04:39:38 STEP 1248 H= 312:00 +CPU= 19.371 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001248 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001248 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001248 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001164 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001164 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001164 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001176 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001176 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001176 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001188 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001188 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001188 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001200 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001200 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001200 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001212 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001212 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001212 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001224 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001224 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001224 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001236 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001236 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001236 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0s7_2016092300_20_1018_12289342_0_r792353749_12.zip Uploading the intermediate file: upload_file_12.zip 04:39:52 STEP 1249 H= 312:15 +CPU= 14.364 04:40:07 STEP 1250 H= 312:30 +CPU= 14.353 04:40:21 STEP 1251 H= 312:45 +CPU= 13.953 04:40:35 STEP 1252 H= 313:00 +CPU= 14.511 04:40:50 STEP 1253 H= 313:15 +CPU= 14.678 04:41:04 STEP 1254 H= 313:30 +CPU= 14.327 04:41:19 STEP 1255 H= 313:45 +CPU= 14.225 04:41:33 STEP 1256 H= 314:00 +CPU= 14.574 04:41:48 STEP 1257 H= 314:15 +CPU= 14.366 04:42:02 STEP 1258 H= 314:30 +CPU= 14.629 04:42:17 STEP 1259 H= 314:45 +CPU= 14.397 04:42:34 STEP 1260 H= 315:00 +CPU= 17.225 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001260 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001260 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001260 04:42:48 STEP 1261 H= 315:15 +CPU= 14.282 04:43:02 STEP 1262 H= 315:30 +CPU= 14.314 04:43:17 STEP 1263 H= 315:45 +CPU= 14.705 04:43:32 STEP 1264 H= 316:00 +CPU= 14.598 04:43:46 STEP 1265 H= 316:15 +CPU= 14.553 04:44:01 STEP 1266 H= 316:30 +CPU= 14.460 04:44:16 STEP 1267 H= 316:45 +CPU= 14.743 04:44:31 STEP 1268 H= 317:00 +CPU= 14.886 04:44:45 STEP 1269 H= 317:15 +CPU= 14.532 04:45:00 STEP 1270 H= 317:30 +CPU= 14.723 04:45:14 STEP 1271 H= 317:45 +CPU= 14.324 04:45:32 STEP 1272 H= 318:00 +CPU= 17.660 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001272 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001272 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001272 04:45:46 STEP 1273 H= 318:15 +CPU= 14.196 04:46:00 STEP 1274 H= 318:30 +CPU= 14.294 04:46:15 STEP 1275 H= 318:45 +CPU= 14.499 04:46:30 STEP 1276 H= 319:00 +CPU= 14.640 04:46:44 STEP 1277 H= 319:15 +CPU= 14.334 04:46:58 STEP 1278 H= 319:30 +CPU= 14.360 04:47:13 STEP 1279 H= 319:45 +CPU= 14.341 04:47:27 STEP 1280 H= 320:00 +CPU= 14.795 04:47:42 STEP 1281 H= 320:15 +CPU= 14.202 04:47:56 STEP 1282 H= 320:30 +CPU= 14.349 04:48:11 STEP 1283 H= 320:45 +CPU= 14.599 04:48:28 STEP 1284 H= 321:00 +CPU= 17.808 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001284 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001284 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001284 04:48:43 STEP 1285 H= 321:15 +CPU= 14.163 04:48:57 STEP 1286 H= 321:30 +CPU= 14.392 04:49:11 STEP 1287 H= 321:45 +CPU= 14.272 04:49:26 STEP 1288 H= 322:00 +CPU= 15.079 04:49:41 STEP 1289 H= 322:15 +CPU= 14.289 04:49:55 STEP 1290 H= 322:30 +CPU= 14.097 04:50:09 STEP 1291 H= 322:45 +CPU= 14.326 04:50:24 STEP 1292 H= 323:00 +CPU= 15.071 04:50:39 STEP 1293 H= 323:15 +CPU= 14.618 04:50:54 STEP 1294 H= 323:30 +CPU= 14.970 04:51:08 STEP 1295 H= 323:45 +CPU= 14.252 04:51:28 STEP 1296 H= 324:00 +CPU= 19.461 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001296 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001296 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001296 04:51:42 STEP 1297 H= 324:15 +CPU= 14.021 04:51:56 STEP 1298 H= 324:30 +CPU= 14.197 04:52:10 STEP 1299 H= 324:45 +CPU= 14.322 04:52:25 STEP 1300 H= 325:00 +CPU= 14.622 04:52:39 STEP 1301 H= 325:15 +CPU= 14.245 04:52:54 STEP 1302 H= 325:30 +CPU= 14.519 04:53:08 STEP 1303 H= 325:45 +CPU= 14.613 04:53:23 STEP 1304 H= 326:00 +CPU= 14.695 04:53:38 STEP 1305 H= 326:15 +CPU= 14.567 04:53:52 STEP 1306 H= 326:30 +CPU= 14.333 04:54:06 STEP 1307 H= 326:45 +CPU= 14.184 04:54:24 STEP 1308 H= 327:00 +CPU= 17.940 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001308 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001308 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001308 04:54:38 STEP 1309 H= 327:15 +CPU= 14.185 04:54:53 STEP 1310 H= 327:30 +CPU= 14.963 04:55:08 STEP 1311 H= 327:45 +CPU= 14.527 04:55:23 STEP 1312 H= 328:00 +CPU= 14.882 04:55:37 STEP 1313 H= 328:15 +CPU= 14.482 04:55:52 STEP 1314 H= 328:30 +CPU= 14.805 04:56:06 STEP 1315 H= 328:45 +CPU= 14.318 04:56:21 STEP 1316 H= 329:00 +CPU= 14.419 04:56:35 STEP 1317 H= 329:15 +CPU= 14.224 04:56:50 STEP 1318 H= 329:30 +CPU= 14.517 04:57:04 STEP 1319 H= 329:45 +CPU= 14.511 04:57:22 STEP 1320 H= 330:00 +CPU= 17.751 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001320 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001320 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001320 04:57:36 STEP 1321 H= 330:15 +CPU= 14.172 04:57:50 STEP 1322 H= 330:30 +CPU= 13.920 04:58:04 STEP 1323 H= 330:45 +CPU= 14.263 04:58:19 STEP 1324 H= 331:00 +CPU= 14.256 04:58:33 STEP 1325 H= 331:15 +CPU= 14.303 04:58:47 STEP 1326 H= 331:30 +CPU= 14.522 04:59:01 STEP 1327 H= 331:45 +CPU= 14.096 04:59:17 STEP 1328 H= 332:00 +CPU= 15.134 04:59:31 STEP 1329 H= 332:15 +CPU= 14.522 04:59:45 STEP 1330 H= 332:30 +CPU= 14.211 05:00:00 STEP 1331 H= 332:45 +CPU= 14.411 05:00:18 STEP 1332 H= 333:00 +CPU= 18.207 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001332 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001332 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001332 05:00:32 STEP 1333 H= 333:15 +CPU= 14.088 05:00:47 STEP 1334 H= 333:30 +CPU= 14.463 05:01:01 STEP 1335 H= 333:45 +CPU= 14.165 05:01:15 STEP 1336 H= 334:00 +CPU= 14.528 05:01:30 STEP 1337 H= 334:15 +CPU= 14.797 05:01:45 STEP 1338 H= 334:30 +CPU= 14.363 05:01:59 STEP 1339 H= 334:45 +CPU= 14.315 05:02:14 STEP 1340 H= 335:00 +CPU= 14.690 05:02:28 STEP 1341 H= 335:15 +CPU= 14.362 05:02:42 STEP 1342 H= 335:30 +CPU= 14.383 05:02:57 STEP 1343 H= 335:45 +CPU= 14.579 05:03:16 STEP 1344 H= 336:00 +CPU= 19.459 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001344 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001344 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001344 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001248 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001248 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001248 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001260 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001260 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001260 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001272 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001272 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001272 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001284 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001284 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001284 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001296 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001296 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001296 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001308 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001308 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001308 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001320 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001320 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001320 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMGGh7zg+001332 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMSHh7zg+001332 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342/ICMUAh7zg+001332 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0s7_2016092300_20_1018_12289342_0_r792353749_13.zip Uploading the intermediate file: upload_file_13.zip 05:03:31 STEP 1345 H= 336:15 +CPU= 14.482 05:03:45 STEP 1346 H= 336:30 +CPU= 14.521 05:04:00 STEP 1347 H= 336:45 +CPU= 14.248 05:04:14 STEP 1348 H= 337:00 +CPU= 14.562 05:04:29 STEP 1349 H= 337:15 +CPU= 14.380 05:04:43 STEP 1350 H= 337:30 +CPU= 14.352 05:04:58 STEP 1351 H= 337:45 +CPU= 14.501 05:05:12 STEP 1352 H= 338:00 +CPU= 14.332 05:05:26 STEP 1353 H= 338:15 +CPU= 14.331 05:05:41 STEP 1354 H= 338:30 +CPU= 14.257 05:05:55 STEP 1355 H= 338:45 +CPU= 14.391 05:06:13 STEP 1356 H= 339:00 +CPU= 17.859 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001356 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001356 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001356 05:06:27 STEP 1357 H= 339:15 +CPU= 14.610 05:06:42 STEP 1358 H= 339:30 +CPU= 14.394 05:06:57 STEP 1359 H= 339:45 +CPU= 14.798 05:07:11 STEP 1360 H= 340:00 +CPU= 14.678 05:07:26 STEP 1361 H= 340:15 +CPU= 14.570 05:07:41 STEP 1362 H= 340:30 +CPU= 14.540 05:07:55 STEP 1363 H= 340:45 +CPU= 14.437 05:08:10 STEP 1364 H= 341:00 +CPU= 14.748 05:08:25 STEP 1365 H= 341:15 +CPU= 14.760 05:08:39 STEP 1366 H= 341:30 +CPU= 14.176 05:08:53 STEP 1367 H= 341:45 +CPU= 14.507 05:09:11 STEP 1368 H= 342:00 +CPU= 17.709 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001368 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001368 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001368 05:09:26 STEP 1369 H= 342:15 +CPU= 14.879 05:09:40 STEP 1370 H= 342:30 +CPU= 14.409 05:09:55 STEP 1371 H= 342:45 +CPU= 14.689 05:10:10 STEP 1372 H= 343:00 +CPU= 14.743 05:10:25 STEP 1373 H= 343:15 +CPU= 14.813 05:10:39 STEP 1374 H= 343:30 +CPU= 14.248 05:10:54 STEP 1375 H= 343:45 +CPU= 14.645 05:11:08 STEP 1376 H= 344:00 +CPU= 14.560 05:11:23 STEP 1377 H= 344:15 +CPU= 14.611 05:11:37 STEP 1378 H= 344:30 +CPU= 14.392 05:11:52 STEP 1379 H= 344:45 +CPU= 14.677 05:12:10 STEP 1380 H= 345:00 +CPU= 17.914 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001380 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001380 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001380 05:12:24 STEP 1381 H= 345:15 +CPU= 14.423 05:12:38 STEP 1382 H= 345:30 +CPU= 14.249 05:12:53 STEP 1383 H= 345:45 +CPU= 14.540 05:13:08 STEP 1384 H= 346:00 +CPU= 14.549 05:13:23 STEP 1385 H= 346:15 +CPU= 15.070 05:13:37 STEP 1386 H= 346:30 +CPU= 14.296 05:13:52 STEP 1387 H= 346:45 +CPU= 14.674 05:14:06 STEP 1388 H= 347:00 +CPU= 14.524 05:14:21 STEP 1389 H= 347:15 +CPU= 14.343 05:14:35 STEP 1390 H= 347:30 +CPU= 14.393 05:14:49 STEP 1391 H= 347:45 +CPU= 14.334 (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a0s7 (argv4) batchid: 1018 (argv5) wuid: 12289342 (argv6) fclen: 20 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.13 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289342 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_ffdc14340dbd78f7f77624512a2cddcb to: /var/lib/boinc-client/slots/2/oifs_43r3_bl_a0s7_2016092300_20_1018_12289342.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_bl_a0s7_2016092300_20_1018_12289342.zip ic_ancil_file: ic_ancil_12289342 ifsdata_file: ifsdata_12289342 climate_data_file: clim_data_12289342 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/2/ic_ancil_12289342.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12289342.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_af1c63dfc6daed5a4ded991a91e9d182 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12289342.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12289342.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12289342.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12289342.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12289342.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12289342.xml -- Model is restarting -- Adjusting last_iter, 1392, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12289342.xml last_cpu_time: 20403 upload_file_number: 14 last_iter: 1393 last_upload: 1209600 model_completed: 0 total_length_of_simulation: 1728000 result_base_name: oifs_43r3_bl_a0s7_2016092300_20_1018_12289342_0_r792353749 The child process has been launched with process id: 2979 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil)) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil)) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil)) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil)) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil)) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil)) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060432:1718370272:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> ECCODES ERROR : IO ERROR: Inappropriate ioctl for device (Inappropriate ioctl for device) GRIB_READ_BYTES_INT 50000 4 FAILED -11 GRIB_API ERROR MSG: Input output problem MPL_ABORT: CALLED FROM PROCESSOR 1 THRD 1 MPL_ABORT: THRD 1 GRIB_READ_BYTES_INT FAILED [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 1515 MB (maxheap), 471 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : RERESF [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : IOSTREAM_MIX:IO_GET [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : IOSTREAM_MIX:READ_RECORD [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : >MPL-IOSTREAM READ_RECORD ( 650) [EC_DRHOOK:Rig-05:1:1:2979:2979] [20240614:060434:1718370274:1.280] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : GRIB_API:IGRIB_READ_BYTES_INT SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1 Process 1 thread 1 calling linux_trbk from intel_trbk() [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/2/oifs_43r3_model.exe' : sigcontextptr=0x7ffe7dfdb550 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 17) with addr2line-cmd [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/2/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x9219b7 0x921c09 0xa6ddcd 0xa609fd 0xa54e19 0x4ba112 0x4b55d8 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x9219b7] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x921c09] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xa6ddcd] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0xa609fd] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xa54e19] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x4ba112] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x4b55d8] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x4b4f52] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x49d638] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x406fbd] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x401e7f] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x401e12] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x1dcc34a] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x1dcdbe7] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x401cd5] : ??() at ??:0 [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s) SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1 abort: ..The child process terminated with status: 134 >>> Printing last 70 lines from file: NODE.001_01 == Full-Pos : setup spectral filters == THE POST-PROCESSING RESOLUTION IS NEVER COARSER THAN THE MODEL RESOLUTION == Full-Pos : setup output files handling == == Full-Pos : setup auxilary data == == Full-Pos : setup I/O handling == SUFPIOS PRINTS OUT NFPXFLD = -999 == Full-Pos : setup map factor for LAM == == Full-Pos : fill auxilary data arrays == - Set up GRIB API usage---------------------------------- ---- Set up DDH diags allocation ------------------------------------ ---- Set up dynamical DDH arrays allocation -- ---------------------------------- --- Set up incremental assimilation---------------------------------- --- Set up stochastic physics, SPBS, CABS ------------------------------ convective CA not active! >> Sub SURAND1 ,, --- Set up stochastically perturbed parametrization tendencies --------- SPPT a.k.a. stochastic physics with spectral pattern >> Sub SUSPSDT stochastically perturbed parametrization tendency scheme (SPPT), LSPSDT= F LQPERTLIMIT2=F --- Set up stochastically perturbed parametrization scheme (SPP) ---------------------------------- LSPP=F LPERT_CFM=T, LPERT_TOFDC=T, LPERT_HSDT=T, LPERT_VDEXC_LEN_STABLE_BL=T LPERT_ENTRORG=T, LPERT_ENTSHALP=T, LPERT_DETRPEN=T, LPERT_RPRCON=T, LPERT_RTAU=T LPERT_CUDUDV=T LPERT_RAMID=T, LPERT_RCLDIFF=T, LPERT_RCLCRIT=T, LPERT_RLCRITSNOW=T LPERT_ZDECORR=T, LPERT_ZSIGQCW=T, LPERT_ZRADEFF=T, LPERT_ZRADEFF2=F LPERT_PHR=F, LPERT_ZHS_VDAERO=T, LPERT_DELTA_AERO=T LLNN_MEAN1=F, LRAMIDLIMIT1=F, LUSE_SETRAN=T, LNSEED_OFFS0=F LLNN_MEAN1_CFM=T LLNN_MEAN1_TOFDC=T LLNN_MEAN1_HSDT=T LLNN_MEAN1_VDEXC_LEN_STABLE_BL=T LLNN_MEAN1_ENTRORG=F LLNN_MEAN1_ENTSHALP=F LLNN_MEAN1_DETRPEN=F LLNN_MEAN1_RPRCON=F LLNN_MEAN1_RTAU=F LLNN_MEAN1_CUDUDV=F LLNN_MEAN1_RAMID=F LLNN_MEAN1_RCLDIFF=F LLNN_MEAN1_RCLCRIT=F LLNN_MEAN1_RLCRITSNOW=F LLNN_MEAN1_ZDECORR=F, LLNN_MEAN1_ZSIGQCW=F, LLNN_MEAN1_ZRADEFFLP=F, LLNN_MEAN1_ZRADEFFIP=F LLNN_MEAN1_PHRLW=F, LLNN_MEAN1_PHRSW=F, LLNN_MEAN1_ZHS_VDAERO=F, LLNN_MEAN1_DELTA_AERO=F NSEEDFAC= 10000 CMPERTOC= 0.200, CMPERTLA= 0.500, CMPERTTOFD= 0.600, SDPERTSO= 0.400, CMPERTLENBL= 0.800 CMPERT_ENTRORG= 0.300, CMPERT_ENTSHALP= 0.300, CMPERT_DETRPEN= 0.300, CMPERT_RPRCON= 0.400, CMPERT_RTAU = 0.600 CMPERT_CUDU= 1.000, CMPERT_CUDV= 1.000 CMPERT_RAMID= 0.100, CMPERT_RCLDIFF= 0.800, CMPERT_RCLCRIT_SEA= 0.800, CMPERT_RCLCRIT_LAND= 0.800, CMPERT_RLCRITSNOW= 0.600 CMPERT_ZDECORR= 0.600, CMPERT_ZSIGQCW= 0.400, CMPERT_ZRADEFFLP= 0.600, CMPERT_ZRADEFFIP= 0.600 CMPERT_PHRLW_ST= 0.044, CMPERT_PHRLW_TR= 0.064, CMPERT_PHRSW_ST= 0.064, CMPERT_PHRSW_TR= 0.034 CMPERT_ZHS_VDAERO= 0.800, CMPERT_DELTA_AERO= 0.600 XPRESS_PHR_ST= 0.10000E+05 TAU= 0.25920E+06, SDEV= 0.10000E+01, XLCOR= 0.20000E+07 SETUP_GWDIAG: LGWDIAGS_ON = F LGWDIAGS_FIELD_OUT = F ---- Set up flexible physics-dynamics interface -- ---------------------------------- === Flexible interface setup === LINTFLEX = F LENTHPREC = F LRADFLEX = F ================================ ----------------------------------------------------------------------- ------ END OF SETUPS at level 0 --------------------------------------- ----------------------------------------------------------------------- INTEGRATION JOB 0 1 N1POS = 1 N1HIS = 1 N1GDI = 1 N1SDI = 1 N1DHP = 1 N1DHFG= 0 N1DHFZ= 0 N1DHFD= 0 N1CFU = 1 N1XFU = 1 N1ISP = 0 N1RES = 1 N1SFXHIS = 0 LRFILAF = F LWRSPEC = T START CNT3 READING RESTART FILES - STEP=01392 IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH = ./srf00141200.0001 MODE=r ------------------------------------------------ oifs_get_stat: Error. ifs.stat file is not open CNT0 not found; string returned was: '' >>> Printing last 1 lines from file: ifs.stat 06:04:33 000000000 CNT3 -999 0.259 0.259 0.259 0:00 0:00 0.00000000000000E+00 1GB 0MB ------------------------------------------------ >>> Printing last 11 lines from file: rcf CSTEP = " 1392", CTIME = "00141200 ", NSTEPLPP = 201, 202, 49, 228226, 228227, 2*0, 1380, 2*0, IPRGPNSRES = 1, IPRGPEWRES = 1, IPRTRWRES = 1, IPRTRVRES = 1, GMASS0 = 0.000000000000000E+000, GMASSI = 0.000000000000000E+000 / ------------------------------------------------ ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
14 Jun 2024 10:29:37 | 1541000 | 22442929 | oifs_43r3_bl_a0s7_2016092300_20_1018_12289342_0 | 864,000 | 14,028 | 0.0162 |
©2024 cpdn.org