climateprediction.net home page
Task 22440855

Task 22440855

Name oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1
Workunit 12284044
Created 8 Jun 2024, 10:21:12 UTC
Sent 8 Jun 2024, 10:24:04 UTC
Report deadline 7 Aug 2024, 10:24:04 UTC
Received 9 Jun 2024, 2:40:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1517679
Run time 14 hours 47 min 11 sec
CPU time 13 hours 40 min 44 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 5.26 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.09 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,352.13 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
0)
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:473840:473840] [20240608:150453:1717884293:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:05:26 STEP 1008 H= 252:00 +CPU= 28.785
  15:05:54 STEP 1009 H= 252:15 +CPU= 27.075
  15:06:22 STEP 1010 H= 252:30 +CPU= 27.068
  15:06:49 STEP 1011 H= 252:45 +CPU= 27.343
  15:07:18 STEP 1012 H= 253:00 +CPU= 27.514
  15:07:40 STEP 1013 H= 253:15 +CPU= 22.487
  15:08:02 STEP 1014 H= 253:30 +CPU= 22.000
  15:08:28 STEP 1015 H= 253:45 +CPU= 25.272
  15:08:52 STEP 1016 H= 254:00 +CPU= 23.942
  15:09:24 STEP 1017 H= 254:15 +CPU= 29.353
  15:09:54 STEP 1018 H= 254:30 +CPU= 28.405
  15:10:24 STEP 1019 H= 254:45 +CPU= 27.919
  15:11:02 STEP 1020 H= 255:00 +CPU= 35.072
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001020
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001020
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001020
  15:11:31 STEP 1021 H= 255:15 +CPU= 27.334
  15:12:00 STEP 1022 H= 255:30 +CPU= 27.486
  15:12:29 STEP 1023 H= 255:45 +CPU= 27.443
  15:12:59 STEP 1024 H= 256:00 +CPU= 28.127
  15:13:29 STEP 1025 H= 256:15 +CPU= 27.615
  15:13:58 STEP 1026 H= 256:30 +CPU= 27.230
  15:14:27 STEP 1027 H= 256:45 +CPU= 26.896
  15:14:56 STEP 1028 H= 257:00 +CPU= 27.643
  15:15:24 STEP 1029 H= 257:15 +CPU= 26.929
  15:15:53 STEP 1030 H= 257:30 +CPU= 27.370
  15:16:22 STEP 1031 H= 257:45 +CPU= 26.914
  15:16:58 STEP 1032 H= 258:00 +CPU= 33.927
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001032
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001032
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001032
  15:17:27 STEP 1033 H= 258:15 +CPU= 27.524
  15:17:56 STEP 1034 H= 258:30 +CPU= 27.364
  15:18:25 STEP 1035 H= 258:45 +CPU= 27.337
  15:18:55 STEP 1036 H= 259:00 +CPU= 27.667
  15:19:23 STEP 1037 H= 259:15 +CPU= 27.592
  15:19:52 STEP 1038 H= 259:30 +CPU= 27.544
  15:20:21 STEP 1039 H= 259:45 +CPU= 27.348
  15:20:50 STEP 1040 H= 260:00 +CPU= 27.615
  15:21:18 STEP 1041 H= 260:15 +CPU= 27.242
  15:21:44 STEP 1042 H= 260:30 +CPU= 25.030
  15:22:06 STEP 1043 H= 260:45 +CPU= 22.399
  15:22:34 STEP 1044 H= 261:00 +CPU= 27.410
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001044
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001044
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001044
  15:22:57 STEP 1045 H= 261:15 +CPU= 22.570
  15:23:23 STEP 1046 H= 261:30 +CPU= 25.801
  15:23:50 STEP 1047 H= 261:45 +CPU= 26.492
  15:24:19 STEP 1048 H= 262:00 +CPU= 27.746
  15:24:47 STEP 1049 H= 262:15 +CPU= 27.394
  15:25:15 STEP 1050 H= 262:30 +CPU= 27.393
  15:25:42 STEP 1051 H= 262:45 +CPU= 27.323
  15:26:10 STEP 1052 H= 263:00 +CPU= 27.333
  15:26:37 STEP 1053 H= 263:15 +CPU= 26.380
  15:27:03 STEP 1054 H= 263:30 +CPU= 25.384
  15:27:30 STEP 1055 H= 263:45 +CPU= 25.860
  15:28:05 STEP 1056 H= 264:00 +CPU= 33.903
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001056
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001056
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001056
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+000960
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+000960
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+000960
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+000972
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+000972
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+000972
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+000984
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+000984
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+000984
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+000996
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+000996
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+000996
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001008
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001008
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001008
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001020
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001020
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001020
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001032
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001032
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001032
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001044
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001044
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001044
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_10.zip
Uploading the intermediate file: upload_file_10.zip
  15:28:31 STEP 1057 H= 264:15 +CPU= 25.020
  15:28:56 STEP 1058 H= 264:30 +CPU= 24.561
  15:29:22 STEP 1059 H= 264:45 +CPU= 24.715
  15:29:47 STEP 1060 H= 265:00 +CPU= 25.315
  15:30:14 STEP 1061 H= 265:15 +CPU= 25.478
  15:30:40 STEP 1062 H= 265:30 +CPU= 25.999
  15:31:07 STEP 1063 H= 265:45 +CPU= 25.917
  15:31:34 STEP 1064 H= 266:00 +CPU= 26.349
  15:32:01 STEP 1065 H= 266:15 +CPU= 26.035
  15:32:26 STEP 1066 H= 266:30 +CPU= 24.976
  15:32:53 STEP 1067 H= 266:45 +CPU= 25.521
  15:33:25 STEP 1068 H= 267:00 +CPU= 31.455
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001068
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001068
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001068
  15:33:51 STEP 1069 H= 267:15 +CPU= 25.175
  15:34:17 STEP 1070 H= 267:30 +CPU= 25.224
  15:34:42 STEP 1071 H= 267:45 +CPU= 24.861
  15:35:10 STEP 1072 H= 268:00 +CPU= 26.493
  15:35:35 STEP 1073 H= 268:15 +CPU= 24.779
  15:35:59 STEP 1074 H= 268:30 +CPU= 23.215
  15:36:22 STEP 1075 H= 268:45 +CPU= 22.789
  15:36:45 STEP 1076 H= 269:00 +CPU= 22.936
  15:37:08 STEP 1077 H= 269:15 +CPU= 22.598
  15:37:30 STEP 1078 H= 269:30 +CPU= 22.331
  15:37:59 STEP 1079 H= 269:45 +CPU= 27.630
  15:38:33 STEP 1080 H= 270:00 +CPU= 33.457
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001080
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001080
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001080
  15:39:00 STEP 1081 H= 270:15 +CPU= 26.068
  15:39:27 STEP 1082 H= 270:30 +CPU= 26.284
  15:39:54 STEP 1083 H= 270:45 +CPU= 26.204
  15:40:22 STEP 1084 H= 271:00 +CPU= 27.054
  15:40:49 STEP 1085 H= 271:15 +CPU= 26.106
  15:41:16 STEP 1086 H= 271:30 +CPU= 26.212
  15:41:42 STEP 1087 H= 271:45 +CPU= 25.129
  15:42:09 STEP 1088 H= 272:00 +CPU= 26.339
  15:42:35 STEP 1089 H= 272:15 +CPU= 25.447
  15:43:01 STEP 1090 H= 272:30 +CPU= 25.654
  15:43:26 STEP 1091 H= 272:45 +CPU= 24.566
  15:43:58 STEP 1092 H= 273:00 +CPU= 30.592
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001092
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001092
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001092
  15:44:24 STEP 1093 H= 273:15 +CPU= 24.902
  15:44:50 STEP 1094 H= 273:30 +CPU= 25.510
  15:45:16 STEP 1095 H= 273:45 +CPU= 25.365
  15:45:42 STEP 1096 H= 274:00 +CPU= 25.451
  15:46:08 STEP 1097 H= 274:15 +CPU= 25.080
  15:46:34 STEP 1098 H= 274:30 +CPU= 25.009
  15:46:59 STEP 1099 H= 274:45 +CPU= 24.882
  15:47:25 STEP 1100 H= 275:00 +CPU= 24.938
  15:47:50 STEP 1101 H= 275:15 +CPU= 24.440
  15:48:17 STEP 1102 H= 275:30 +CPU= 25.676
  15:48:43 STEP 1103 H= 275:45 +CPU= 25.331
  15:49:18 STEP 1104 H= 276:00 +CPU= 33.607
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001104
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001104
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001104
  15:49:43 STEP 1105 H= 276:15 +CPU= 24.701
  15:50:09 STEP 1106 H= 276:30 +CPU= 24.677
  15:50:33 STEP 1107 H= 276:45 +CPU= 23.371
  15:50:54 STEP 1108 H= 277:00 +CPU= 21.468
  15:51:17 STEP 1109 H= 277:15 +CPU= 21.955
  15:51:38 STEP 1110 H= 277:30 +CPU= 21.303
  15:52:00 STEP 1111 H= 277:45 +CPU= 21.721
  15:52:29 STEP 1112 H= 278:00 +CPU= 28.140
  15:52:59 STEP 1113 H= 278:15 +CPU= 27.349
  15:53:27 STEP 1114 H= 278:30 +CPU= 26.533
  15:53:56 STEP 1115 H= 278:45 +CPU= 26.863
  15:54:31 STEP 1116 H= 279:00 +CPU= 32.769
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001116
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001116
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001116
  15:54:59 STEP 1117 H= 279:15 +CPU= 26.502
  15:55:26 STEP 1118 H= 279:30 +CPU= 26.409
  15:55:54 STEP 1119 H= 279:45 +CPU= 26.070
  15:56:22 STEP 1120 H= 280:00 +CPU= 26.392
  15:56:49 STEP 1121 H= 280:15 +CPU= 26.189
  15:57:18 STEP 1122 H= 280:30 +CPU= 26.428
  15:57:45 STEP 1123 H= 280:45 +CPU= 26.193
  15:58:15 STEP 1124 H= 281:00 +CPU= 27.239
  15:58:42 STEP 1125 H= 281:15 +CPU= 25.587
  15:59:10 STEP 1126 H= 281:30 +CPU= 25.986
  15:59:36 STEP 1127 H= 281:45 +CPU= 25.296
  16:00:09 STEP 1128 H= 282:00 +CPU= 31.169
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001128
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001128
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001128
  16:00:36 STEP 1129 H= 282:15 +CPU= 25.691
  16:01:03 STEP 1130 H= 282:30 +CPU= 25.736
  16:01:30 STEP 1131 H= 282:45 +CPU= 25.756
  16:01:56 STEP 1132 H= 283:00 +CPU= 25.564
  16:02:20 STEP 1133 H= 283:15 +CPU= 23.341
  16:02:42 STEP 1134 H= 283:30 +CPU= 21.568
  16:03:05 STEP 1135 H= 283:45 +CPU= 22.275
  16:03:27 STEP 1136 H= 284:00 +CPU= 21.996
  16:03:53 STEP 1137 H= 284:15 +CPU= 24.740
  16:04:22 STEP 1138 H= 284:30 +CPU= 27.316
  16:04:52 STEP 1139 H= 284:45 +CPU= 27.593
  16:05:28 STEP 1140 H= 285:00 +CPU= 33.092
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001140
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001140
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001140
  16:05:56 STEP 1141 H= 285:15 +CPU= 26.937
  16:06:25 STEP 1142 H= 285:30 +CPU= 27.110
  16:06:54 STEP 1143 H= 285:45 +CPU= 26.598
  16:07:24 STEP 1144 H= 286:00 +CPU= 27.495
  16:07:52 STEP 1145 H= 286:15 +CPU= 26.752
  16:08:21 STEP 1146 H= 286:30 +CPU= 27.049
  16:08:49 STEP 1147 H= 286:45 +CPU= 26.261
  16:09:19 STEP 1148 H= 287:00 +CPU= 27.105
  16:09:47 STEP 1149 H= 287:15 +CPU= 26.584
  16:10:15 STEP 1150 H= 287:30 +CPU= 26.409
  16:10:43 STEP 1151 H= 287:45 +CPU= 26.171
  16:11:21 STEP 1152 H= 288:00 +CPU= 35.532
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001152
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001152
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001152
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001056
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001056
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001056
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001068
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001068
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001068
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001080
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001080
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001080
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001092
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001092
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001092
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001104
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001104
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001104
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001116
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001116
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001116
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001128
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001128
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001128
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001140
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001140
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001140
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_11.zip
Uploading the intermediate file: upload_file_11.zip
  16:11:49 STEP 1153 H= 288:15 +CPU= 26.112
  16:12:17 STEP 1154 H= 288:30 +CPU= 26.201
  16:12:46 STEP 1155 H= 288:45 +CPU= 27.136
  16:13:15 STEP 1156 H= 289:00 +CPU= 27.434
  16:13:44 STEP 1157 H= 289:15 +CPU= 26.136
  16:14:12 STEP 1158 H= 289:30 +CPU= 26.204
  16:14:40 STEP 1159 H= 289:45 +CPU= 25.790
  16:15:08 STEP 1160 H= 290:00 +CPU= 26.713
  16:15:35 STEP 1161 H= 290:15 +CPU= 26.102
  16:16:01 STEP 1162 H= 290:30 +CPU= 25.563
  16:16:23 STEP 1163 H= 290:45 +CPU= 21.909
  16:16:49 STEP 1164 H= 291:00 +CPU= 25.551
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001164
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001164
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001164
  16:17:10 STEP 1165 H= 291:15 +CPU= 21.091
  16:17:31 STEP 1166 H= 291:30 +CPU= 20.658
  16:17:54 STEP 1167 H= 291:45 +CPU= 22.415
  16:18:21 STEP 1168 H= 292:00 +CPU= 27.061
  16:18:49 STEP 1169 H= 292:15 +CPU= 26.864
  16:19:16 STEP 1170 H= 292:30 +CPU= 25.858
  16:19:44 STEP 1171 H= 292:45 +CPU= 26.537
  16:20:11 STEP 1172 H= 293:00 +CPU= 26.683
  16:20:39 STEP 1173 H= 293:15 +CPU= 26.539
  16:21:06 STEP 1174 H= 293:30 +CPU= 25.747
  16:21:33 STEP 1175 H= 293:45 +CPU= 25.939
  16:22:06 STEP 1176 H= 294:00 +CPU= 32.333
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001176
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001176
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001176
  16:22:33 STEP 1177 H= 294:15 +CPU= 25.805
  16:22:59 STEP 1178 H= 294:30 +CPU= 25.416
  16:23:25 STEP 1179 H= 294:45 +CPU= 25.134
  16:23:51 STEP 1180 H= 295:00 +CPU= 25.652
  16:24:17 STEP 1181 H= 295:15 +CPU= 25.683
  16:24:44 STEP 1182 H= 295:30 +CPU= 25.649
  16:25:10 STEP 1183 H= 295:45 +CPU= 25.611
  16:25:36 STEP 1184 H= 296:00 +CPU= 25.715
  16:26:03 STEP 1185 H= 296:15 +CPU= 25.903
  16:26:29 STEP 1186 H= 296:30 +CPU= 25.776
  16:26:55 STEP 1187 H= 296:45 +CPU= 25.349
  16:27:27 STEP 1188 H= 297:00 +CPU= 31.208
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001188
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001188
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001188
  16:27:54 STEP 1189 H= 297:15 +CPU= 25.530
  16:28:20 STEP 1190 H= 297:30 +CPU= 26.108
  16:28:48 STEP 1191 H= 297:45 +CPU= 26.413
  16:29:14 STEP 1192 H= 298:00 +CPU= 25.868
  16:29:40 STEP 1193 H= 298:15 +CPU= 25.422
  16:30:07 STEP 1194 H= 298:30 +CPU= 25.922
  16:30:30 STEP 1195 H= 298:45 +CPU= 23.063
  16:30:53 STEP 1196 H= 299:00 +CPU= 22.602
  16:31:16 STEP 1197 H= 299:15 +CPU= 22.925
  16:31:40 STEP 1198 H= 299:30 +CPU= 23.266
  16:32:04 STEP 1199 H= 299:45 +CPU= 23.709
  16:32:44 STEP 1200 H= 300:00 +CPU= 37.198
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001200
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001200
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001200
  16:33:13 STEP 1201 H= 300:15 +CPU= 27.426
  16:33:42 STEP 1202 H= 300:30 +CPU= 27.194
  16:34:11 STEP 1203 H= 300:45 +CPU= 27.190
  16:34:40 STEP 1204 H= 301:00 +CPU= 27.713
  16:35:09 STEP 1205 H= 301:15 +CPU= 26.691
  16:35:38 STEP 1206 H= 301:30 +CPU= 27.055
  16:36:05 STEP 1207 H= 301:45 +CPU= 26.117
  16:36:33 STEP 1208 H= 302:00 +CPU= 27.128
  16:37:01 STEP 1209 H= 302:15 +CPU= 26.231
  16:37:29 STEP 1210 H= 302:30 +CPU= 26.565
  16:37:58 STEP 1211 H= 302:45 +CPU= 26.574
  16:38:32 STEP 1212 H= 303:00 +CPU= 32.533
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001212
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001212
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001212
  16:39:01 STEP 1213 H= 303:15 +CPU= 26.317
  16:39:29 STEP 1214 H= 303:30 +CPU= 26.269
  16:39:57 STEP 1215 H= 303:45 +CPU= 26.355
  16:40:26 STEP 1216 H= 304:00 +CPU= 26.997
  16:40:55 STEP 1217 H= 304:15 +CPU= 26.453
  16:41:22 STEP 1218 H= 304:30 +CPU= 25.947
  16:41:49 STEP 1219 H= 304:45 +CPU= 25.378
  16:42:16 STEP 1220 H= 305:00 +CPU= 26.308
  16:42:44 STEP 1221 H= 305:15 +CPU= 26.410
  16:43:11 STEP 1222 H= 305:30 +CPU= 26.162
  16:43:38 STEP 1223 H= 305:45 +CPU= 25.739
  16:44:12 STEP 1224 H= 306:00 +CPU= 32.064
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001224
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001224
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001224
  16:44:36 STEP 1225 H= 306:15 +CPU= 23.350
  16:44:58 STEP 1226 H= 306:30 +CPU= 22.321
  16:45:21 STEP 1227 H= 306:45 +CPU= 22.420
  16:45:44 STEP 1228 H= 307:00 +CPU= 23.131
  16:46:09 STEP 1229 H= 307:15 +CPU= 24.050
  16:46:39 STEP 1230 H= 307:30 +CPU= 27.509
  16:47:09 STEP 1231 H= 307:45 +CPU= 27.710
  16:47:39 STEP 1232 H= 308:00 +CPU= 27.845
  16:48:08 STEP 1233 H= 308:15 +CPU= 27.565
  16:48:36 STEP 1234 H= 308:30 +CPU= 26.924
  16:49:04 STEP 1235 H= 308:45 +CPU= 26.335
  16:49:38 STEP 1236 H= 309:00 +CPU= 32.751
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001236
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001236
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001236
  16:50:05 STEP 1237 H= 309:15 +CPU= 26.331
  16:50:32 STEP 1238 H= 309:30 +CPU= 26.146
  16:50:59 STEP 1239 H= 309:45 +CPU= 26.319
  16:51:27 STEP 1240 H= 310:00 +CPU= 26.655
  16:51:54 STEP 1241 H= 310:15 +CPU= 26.453
  16:52:21 STEP 1242 H= 310:30 +CPU= 25.852
  16:52:47 STEP 1243 H= 310:45 +CPU= 25.628
  16:53:15 STEP 1244 H= 311:00 +CPU= 26.796
  16:53:42 STEP 1245 H= 311:15 +CPU= 25.862
  16:54:10 STEP 1246 H= 311:30 +CPU= 26.455
  16:54:38 STEP 1247 H= 311:45 +CPU= 26.090
  16:55:17 STEP 1248 H= 312:00 +CPU= 35.565
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001248
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001248
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001248
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001152
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001152
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001152
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001164
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001164
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001164
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001176
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001176
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001176
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001188
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001188
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001188
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001200
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001200
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001200
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001212
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001212
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001212
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001224
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001224
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001224
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001236
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001236
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001236
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_12.zip
Uploading the intermediate file: upload_file_12.zip
  16:55:45 STEP 1249 H= 312:15 +CPU= 26.211
  16:56:14 STEP 1250 H= 312:30 +CPU= 27.404
  16:56:43 STEP 1251 H= 312:45 +CPU= 27.242
  16:57:12 STEP 1252 H= 313:00 +CPU= 27.479
  16:57:39 STEP 1253 H= 313:15 +CPU= 26.085
  16:58:05 STEP 1254 H= 313:30 +CPU= 24.505
  16:58:27 STEP 1255 H= 313:45 +CPU= 21.770
  16:58:50 STEP 1256 H= 314:00 +CPU= 22.952
  16:59:12 STEP 1257 H= 314:15 +CPU= 21.971
  16:59:34 STEP 1258 H= 314:30 +CPU= 21.260
  17:00:01 STEP 1259 H= 314:45 +CPU= 26.560
  17:00:34 STEP 1260 H= 315:00 +CPU= 32.458
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001260
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001260
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001260
  17:01:01 STEP 1261 H= 315:15 +CPU= 25.662
  17:01:28 STEP 1262 H= 315:30 +CPU= 26.121
  17:01:55 STEP 1263 H= 315:45 +CPU= 26.723
  17:02:24 STEP 1264 H= 316:00 +CPU= 26.730
  17:02:53 STEP 1265 H= 316:15 +CPU= 26.553
  17:03:20 STEP 1266 H= 316:30 +CPU= 26.006
  17:03:50 STEP 1267 H= 316:45 +CPU= 26.430
  17:04:18 STEP 1268 H= 317:00 +CPU= 26.896
  17:04:45 STEP 1269 H= 317:15 +CPU= 25.997
  17:05:13 STEP 1270 H= 317:30 +CPU= 26.413
  17:05:41 STEP 1271 H= 317:45 +CPU= 25.914
  17:06:16 STEP 1272 H= 318:00 +CPU= 32.014
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001272
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001272
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001272
  17:06:43 STEP 1273 H= 318:15 +CPU= 25.661
  17:07:11 STEP 1274 H= 318:30 +CPU= 25.989
  17:07:39 STEP 1275 H= 318:45 +CPU= 26.069
  17:08:07 STEP 1276 H= 319:00 +CPU= 26.461
  17:08:35 STEP 1277 H= 319:15 +CPU= 26.315
  17:09:02 STEP 1278 H= 319:30 +CPU= 25.802
  17:09:30 STEP 1279 H= 319:45 +CPU= 26.132
  17:09:58 STEP 1280 H= 320:00 +CPU= 26.778
  17:10:26 STEP 1281 H= 320:15 +CPU= 26.586
  17:10:53 STEP 1282 H= 320:30 +CPU= 26.134
  17:11:21 STEP 1283 H= 320:45 +CPU= 26.056
  17:11:54 STEP 1284 H= 321:00 +CPU= 31.702
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001284
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001284
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001284
  17:12:32 STEP 1285 H= 321:15 +CPU= 34.026
  17:13:11 STEP 1286 H= 321:30 +CPU= 34.858
  17:13:48 STEP 1287 H= 321:45 +CPU= 34.771
  17:14:27 STEP 1288 H= 322:00 +CPU= 36.550
  17:15:06 STEP 1289 H= 322:15 +CPU= 36.665
  17:15:43 STEP 1290 H= 322:30 +CPU= 35.011
  17:16:21 STEP 1291 H= 322:45 +CPU= 36.485
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1pl
(argv4) batchid: 1017
(argv5) wuid: 12284044
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /mydisks/b/boinc-client/slots/8
Project directory is: /mydisks/b/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044
..mkdir for temp folder for results failed
Copying: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /mydisks/b/boinc-client/slots/8/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mydisks/b/boinc-client/slots/8/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_dbc6751920f04623a823ae2a7aa0f11b to: /mydisks/b/boinc-client/slots/8/oifs_43r3_bl_a1pl_2016092300_20_1017_12284044.zip
Unzipping the namelist zip file: /mydisks/b/boinc-client/slots/8/oifs_43r3_bl_a1pl_2016092300_20_1017_12284044.zip
ic_ancil_file: ic_ancil_12284044
ifsdata_file: ifsdata_12284044
climate_data_file: clim_data_12284044
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /mydisks/b/boinc-client/slots/8/ic_ancil_12284044.zip
Unzipping the IC ancils zip file: /mydisks/b/boinc-client/slots/8/ic_ancil_12284044.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f645686348761a8677b936ff2a445050 to: /mydisks/b/boinc-client/slots/8/ifsdata/ifsdata_12284044.zip
Unzipping the ifsdata_zip file: /mydisks/b/boinc-client/slots/8/ifsdata/ifsdata_12284044.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mydisks/b/boinc-client/slots/8/159l_2/clim_data_12284044.zip
Unzipping the climate data zip file: /mydisks/b/boinc-client/slots/8/159l_2/clim_data_12284044.zip
Checking for progress XML file: /mydisks/b/boinc-client/slots/8/progress_file_12284044.xml
Opened progress file ok : /mydisks/b/boinc-client/slots/8/progress_file_12284044.xml
-- Model is restarting --
Adjusting last_iter, 1292, to previous model restart step.
Creating progress file: /mydisks/b/boinc-client/slots/8/progress_file_12284044.xml
last_cpu_time: 43552
upload_file_number: 13
last_iter: 1249
last_upload: 1123200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678
The child process has been launched with process id: 3229050
Executing the command: /mydisks/b/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:centurion:1:1:3229050:3229050] [20240608:174435:1717893875:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:45:02 STEP 1248 H= 312:00 +CPU= 23.077
  17:45:23 STEP 1249 H= 312:15 +CPU= 21.388
  17:45:45 STEP 1250 H= 312:30 +CPU= 21.250
  17:46:12 STEP 1251 H= 312:45 +CPU= 25.918
  17:46:43 STEP 1252 H= 313:00 +CPU= 28.460
  17:47:13 STEP 1253 H= 313:15 +CPU= 28.074
  17:47:43 STEP 1254 H= 313:30 +CPU= 27.760
  17:48:13 STEP 1255 H= 313:45 +CPU= 27.199
  17:48:44 STEP 1256 H= 314:00 +CPU= 28.037
  17:49:12 STEP 1257 H= 314:15 +CPU= 26.512
  17:49:40 STEP 1258 H= 314:30 +CPU= 27.089
  17:50:08 STEP 1259 H= 314:45 +CPU= 26.472
  17:50:43 STEP 1260 H= 315:00 +CPU= 33.049
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001260
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001260
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001260
  17:51:11 STEP 1261 H= 315:15 +CPU= 26.982
  17:51:38 STEP 1262 H= 315:30 +CPU= 25.879
  17:52:06 STEP 1263 H= 315:45 +CPU= 26.210
  17:52:34 STEP 1264 H= 316:00 +CPU= 27.580
  17:53:02 STEP 1265 H= 316:15 +CPU= 26.520
  17:53:31 STEP 1266 H= 316:30 +CPU= 26.783
  17:53:59 STEP 1267 H= 316:45 +CPU= 26.784
  17:54:27 STEP 1268 H= 317:00 +CPU= 27.123
  17:54:55 STEP 1269 H= 317:15 +CPU= 26.909
  17:55:23 STEP 1270 H= 317:30 +CPU= 26.909
  17:55:51 STEP 1271 H= 317:45 +CPU= 26.320
  17:56:25 STEP 1272 H= 318:00 +CPU= 33.071
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001272
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001272
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001272
  17:56:53 STEP 1273 H= 318:15 +CPU= 27.246
  17:57:20 STEP 1274 H= 318:30 +CPU= 26.324
  17:57:47 STEP 1275 H= 318:45 +CPU= 26.067
  17:58:14 STEP 1276 H= 319:00 +CPU= 25.779
  17:58:52 STEP 1277 H= 319:15 +CPU= 35.466
  17:59:30 STEP 1278 H= 319:30 +CPU= 35.236
  18:00:09 STEP 1279 H= 319:45 +CPU= 36.238
  18:00:51 STEP 1280 H= 320:00 +CPU= 38.735
  18:01:32 STEP 1281 H= 320:15 +CPU= 36.868
  18:02:11 STEP 1282 H= 320:30 +CPU= 37.259
  18:02:51 STEP 1283 H= 320:45 +CPU= 36.670
  18:03:37 STEP 1284 H= 321:00 +CPU= 42.894
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001284
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001284
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001284
  18:04:13 STEP 1285 H= 321:15 +CPU= 33.261
  18:04:50 STEP 1286 H= 321:30 +CPU= 33.304
  18:05:27 STEP 1287 H= 321:45 +CPU= 34.252
  18:06:05 STEP 1288 H= 322:00 +CPU= 36.274
  18:06:43 STEP 1289 H= 322:15 +CPU= 34.773
  18:07:20 STEP 1290 H= 322:30 +CPU= 34.124
  18:07:57 STEP 1291 H= 322:45 +CPU= 34.573
  18:08:35 STEP 1292 H= 323:00 +CPU= 36.180
  18:09:15 STEP 1293 H= 323:15 +CPU= 37.047
  18:09:54 STEP 1294 H= 323:30 +CPU= 36.801
  18:10:32 STEP 1295 H= 323:45 +CPU= 34.542
  18:11:24 STEP 1296 H= 324:00 +CPU= 47.304
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001296
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001296
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001296
  18:12:05 STEP 1297 H= 324:15 +CPU= 37.418
  18:12:44 STEP 1298 H= 324:30 +CPU= 36.867
  18:13:24 STEP 1299 H= 324:45 +CPU= 36.553
  18:14:05 STEP 1300 H= 325:00 +CPU= 37.611
  18:14:47 STEP 1301 H= 325:15 +CPU= 38.163
  18:15:28 STEP 1302 H= 325:30 +CPU= 36.211
  18:16:06 STEP 1303 H= 325:45 +CPU= 34.160
  18:16:47 STEP 1304 H= 326:00 +CPU= 36.831
  18:17:28 STEP 1305 H= 326:15 +CPU= 36.787
  18:18:08 STEP 1306 H= 326:30 +CPU= 35.768
  18:18:44 STEP 1307 H= 326:45 +CPU= 33.573
  18:19:28 STEP 1308 H= 327:00 +CPU= 41.606
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001308
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001308
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001308
  18:20:04 STEP 1309 H= 327:15 +CPU= 33.669
  18:20:40 STEP 1310 H= 327:30 +CPU= 33.282
  18:21:15 STEP 1311 H= 327:45 +CPU= 31.805
  18:21:49 STEP 1312 H= 328:00 +CPU= 32.155
  18:22:24 STEP 1313 H= 328:15 +CPU= 33.182
  18:23:00 STEP 1314 H= 328:30 +CPU= 33.929
  18:23:34 STEP 1315 H= 328:45 +CPU= 32.990
  18:24:08 STEP 1316 H= 329:00 +CPU= 31.965
  18:24:42 STEP 1317 H= 329:15 +CPU= 32.018
  18:25:15 STEP 1318 H= 329:30 +CPU= 31.763
  18:25:52 STEP 1319 H= 329:45 +CPU= 33.941
  18:26:39 STEP 1320 H= 330:00 +CPU= 43.961
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001320
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001320
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001320
  18:27:17 STEP 1321 H= 330:15 +CPU= 34.412
  18:27:52 STEP 1322 H= 330:30 +CPU= 31.727
  18:28:25 STEP 1323 H= 330:45 +CPU= 30.703
  18:28:59 STEP 1324 H= 331:00 +CPU= 31.055
  18:29:34 STEP 1325 H= 331:15 +CPU= 31.291
  18:30:08 STEP 1326 H= 331:30 +CPU= 31.013
  18:30:33 STEP 1327 H= 331:45 +CPU= 23.627
  18:30:54 STEP 1328 H= 332:00 +CPU= 20.946
  18:31:15 STEP 1329 H= 332:15 +CPU= 20.971
  18:31:36 STEP 1330 H= 332:30 +CPU= 20.902
  18:31:56 STEP 1331 H= 332:45 +CPU= 20.514
  18:32:31 STEP 1332 H= 333:00 +CPU= 33.704
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001332
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001332
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001332
  18:32:58 STEP 1333 H= 333:15 +CPU= 26.777
  18:33:25 STEP 1334 H= 333:30 +CPU= 26.855
  18:33:52 STEP 1335 H= 333:45 +CPU= 26.929
  18:34:20 STEP 1336 H= 334:00 +CPU= 27.112
  18:34:48 STEP 1337 H= 334:15 +CPU= 26.989
  18:35:15 STEP 1338 H= 334:30 +CPU= 26.845
  18:35:42 STEP 1339 H= 334:45 +CPU= 26.493
  18:36:10 STEP 1340 H= 335:00 +CPU= 26.994
  18:36:37 STEP 1341 H= 335:15 +CPU= 26.482
  18:36:59 STEP 1342 H= 335:30 +CPU= 21.951
  18:37:19 STEP 1343 H= 335:45 +CPU= 20.554
  18:37:47 STEP 1344 H= 336:00 +CPU= 27.788
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001344
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001344
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001344
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001248
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001248
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001248
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001260
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001260
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001260
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001272
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001272
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001272
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001284
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001284
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001284
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001296
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001296
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001296
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001308
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001308
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001308
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001320
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001320
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001320
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001332
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001332
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001332
Zipping up the intermediate file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_13.zip
Uploading the intermediate file: upload_file_13.zip
  18:38:08 STEP 1345 H= 336:15 +CPU= 20.632
  18:38:35 STEP 1346 H= 336:30 +CPU= 25.972
  18:39:05 STEP 1347 H= 336:45 +CPU= 29.079
  18:39:36 STEP 1348 H= 337:00 +CPU= 29.574
  18:40:07 STEP 1349 H= 337:15 +CPU= 29.061
  18:40:37 STEP 1350 H= 337:30 +CPU= 29.169
  18:41:07 STEP 1351 H= 337:45 +CPU= 28.116
  18:41:37 STEP 1352 H= 338:00 +CPU= 28.585
  18:42:06 STEP 1353 H= 338:15 +CPU= 27.868
  18:42:36 STEP 1354 H= 338:30 +CPU= 28.471
  18:43:05 STEP 1355 H= 338:45 +CPU= 28.356
  18:43:43 STEP 1356 H= 339:00 +CPU= 35.367
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001356
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001356
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001356
  18:44:14 STEP 1357 H= 339:15 +CPU= 28.443
  18:44:44 STEP 1358 H= 339:30 +CPU= 28.413
  18:45:13 STEP 1359 H= 339:45 +CPU= 27.814
  18:45:43 STEP 1360 H= 340:00 +CPU= 27.986
  18:46:11 STEP 1361 H= 340:15 +CPU= 27.317
  18:46:40 STEP 1362 H= 340:30 +CPU= 27.288
  18:47:08 STEP 1363 H= 340:45 +CPU= 27.263
  18:47:38 STEP 1364 H= 341:00 +CPU= 27.940
  18:48:06 STEP 1365 H= 341:15 +CPU= 27.038
  18:48:34 STEP 1366 H= 341:30 +CPU= 27.050
  18:49:03 STEP 1367 H= 341:45 +CPU= 27.202
  18:49:39 STEP 1368 H= 342:00 +CPU= 33.603
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001368
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001368
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001368
  18:50:08 STEP 1369 H= 342:15 +CPU= 27.258
  18:50:37 STEP 1370 H= 342:30 +CPU= 27.553
  18:51:07 STEP 1371 H= 342:45 +CPU= 27.404
  18:51:38 STEP 1372 H= 343:00 +CPU= 28.144
  18:52:07 STEP 1373 H= 343:15 +CPU= 27.434
  18:52:37 STEP 1374 H= 343:30 +CPU= 27.348
  18:53:07 STEP 1375 H= 343:45 +CPU= 27.382
  18:53:38 STEP 1376 H= 344:00 +CPU= 28.309
  18:54:07 STEP 1377 H= 344:15 +CPU= 27.826
  18:54:35 STEP 1378 H= 344:30 +CPU= 27.121
  18:55:04 STEP 1379 H= 344:45 +CPU= 27.527
  18:55:40 STEP 1380 H= 345:00 +CPU= 34.388
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001380
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001380
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001380
  18:56:09 STEP 1381 H= 345:15 +CPU= 27.979
  18:56:38 STEP 1382 H= 345:30 +CPU= 27.940
  18:57:08 STEP 1383 H= 345:45 +CPU= 27.747
  18:57:38 STEP 1384 H= 346:00 +CPU= 28.502
  18:58:08 STEP 1385 H= 346:15 +CPU= 27.632
  18:58:38 STEP 1386 H= 346:30 +CPU= 27.715
  18:59:08 STEP 1387 H= 346:45 +CPU= 27.884
  18:59:39 STEP 1388 H= 347:00 +CPU= 28.376
  19:00:09 STEP 1389 H= 347:15 +CPU= 28.159
  19:00:39 STEP 1390 H= 347:30 +CPU= 27.604
  19:01:08 STEP 1391 H= 347:45 +CPU= 27.608
  19:01:48 STEP 1392 H= 348:00 +CPU= 37.439
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001392
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001392
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001392
  19:02:18 STEP 1393 H= 348:15 +CPU= 27.908
  19:02:48 STEP 1394 H= 348:30 +CPU= 27.875
  19:03:19 STEP 1395 H= 348:45 +CPU= 28.273
  19:03:50 STEP 1396 H= 349:00 +CPU= 28.539
  19:04:20 STEP 1397 H= 349:15 +CPU= 27.450
  19:04:49 STEP 1398 H= 349:30 +CPU= 27.494
  19:05:18 STEP 1399 H= 349:45 +CPU= 27.512
  19:05:49 STEP 1400 H= 350:00 +CPU= 28.256
  19:06:18 STEP 1401 H= 350:15 +CPU= 27.516
  19:06:48 STEP 1402 H= 350:30 +CPU= 27.683
  19:07:17 STEP 1403 H= 350:45 +CPU= 27.431
  19:07:55 STEP 1404 H= 351:00 +CPU= 34.697
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001404
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001404
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001404
  19:08:24 STEP 1405 H= 351:15 +CPU= 27.537
  19:08:53 STEP 1406 H= 351:30 +CPU= 27.291
  19:09:21 STEP 1407 H= 351:45 +CPU= 27.265
  19:09:50 STEP 1408 H= 352:00 +CPU= 27.530
  19:10:19 STEP 1409 H= 352:15 +CPU= 27.110
  19:10:48 STEP 1410 H= 352:30 +CPU= 26.992
  19:11:17 STEP 1411 H= 352:45 +CPU= 27.067
  19:11:46 STEP 1412 H= 353:00 +CPU= 27.518
  19:12:15 STEP 1413 H= 353:15 +CPU= 27.117
  19:12:44 STEP 1414 H= 353:30 +CPU= 27.167
  19:13:13 STEP 1415 H= 353:45 +CPU= 26.809
  19:13:49 STEP 1416 H= 354:00 +CPU= 33.388
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001416
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001416
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001416
  19:14:19 STEP 1417 H= 354:15 +CPU= 27.019
  19:14:48 STEP 1418 H= 354:30 +CPU= 26.809
  19:15:18 STEP 1419 H= 354:45 +CPU= 26.860
  19:15:47 STEP 1420 H= 355:00 +CPU= 27.173
  19:16:16 STEP 1421 H= 355:15 +CPU= 26.622
  19:16:44 STEP 1422 H= 355:30 +CPU= 26.545
  19:17:12 STEP 1423 H= 355:45 +CPU= 26.480
  19:17:41 STEP 1424 H= 356:00 +CPU= 27.405
  19:18:10 STEP 1425 H= 356:15 +CPU= 26.764
  19:18:40 STEP 1426 H= 356:30 +CPU= 27.431
  19:19:09 STEP 1427 H= 356:45 +CPU= 27.003
  19:19:45 STEP 1428 H= 357:00 +CPU= 33.509
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001428
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001428
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001428
  19:20:14 STEP 1429 H= 357:15 +CPU= 26.906
  19:20:43 STEP 1430 H= 357:30 +CPU= 27.238
  19:21:11 STEP 1431 H= 357:45 +CPU= 26.744
  19:21:40 STEP 1432 H= 358:00 +CPU= 27.212
  19:22:08 STEP 1433 H= 358:15 +CPU= 26.218
  19:22:36 STEP 1434 H= 358:30 +CPU= 26.750
  19:23:04 STEP 1435 H= 358:45 +CPU= 26.647
  19:23:33 STEP 1436 H= 359:00 +CPU= 27.186
  19:24:02 STEP 1437 H= 359:15 +CPU= 26.767
  19:24:31 STEP 1438 H= 359:30 +CPU= 26.856
  19:25:01 STEP 1439 H= 359:45 +CPU= 26.649
  19:25:37 STEP 1440 H= 360:00 +CPU= 33.663
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.5770402909E-21      I       0.2311558117E-05      I
ICloud fractionI      0.0000000000E+00      I       0.4046565716E-01      I
I  Relat. Hum. I      0.1142506337E-02      I       0.1345472719E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3027988716E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    300441636 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0117( 2.6%)
 19:25:37 STEP 1440 H= 360:00 +CPU= 33.663
 END CNT3
 NSTEP =  1440 CNT0   000000000  24.934  24.934  26.376   0.945               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         5720/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    103    0       46      1044068     5720 ************    10.3     1.2     9.1     1.8    20.3
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 6061402.9 6061402.9    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 6058332.8 6058332.8     99.95      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  209    4643.6    4643.6     16.01      0.00
   9 SPCM     - SPECTRAL COMP.                     192     876.4     876.4      2.78      0.00
  10 SCAN2M - PHYSICS                              193   15349.7   15349.7     48.87      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 16     804.7     804.7      0.21      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  9      53.1      53.1      0.01      0.00
  17 GRIDFPOS IN CNT4                               16      36.2      36.2      0.01      0.00
  27 RESTART FILES - WRITING                         4    3218.9    3218.9      0.21      0.00
  28 RESTART FILES - READING                         1    2554.7    2554.7      0.04      0.00
  29 SU4FPOS IN CNT4                                16       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                16    6356.2    6356.2      1.68      0.00
  31 POSDDH IN STEPO                                 1       0.6       0.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    193    6274.9    6274.9     19.98      0.00
  39 SU0YOMB                                         1     321.8     321.8      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    193      44.1      44.1      0.14      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              192     328.4     328.4      1.04      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              192      77.2      77.2      0.24      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  9       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      420     529.9     529.9      3.67      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       400     308.2     308.2      2.03      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        400     102.4     102.4      0.68      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       420     167.7     167.7      1.16      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      31.6      31.6      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            420      63.2      63.2      0.44      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            400      32.9      32.9      0.22      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            420      87.6      87.6      0.61      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            400      64.5      64.5      0.43      0.00
 400 GSTATS                                      23311       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 21635       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     6061.4 CPU TIME    5692.2 VECTOR TIME     5692.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      213.9
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMGGh7zg+001440
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMSHh7zg+001440
Moving to projects directory: /mydisks/b/boinc-client/slots/8/ICMUAh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/slots/8/NODE.001_01
Adding to the zip: /mydisks/b/boinc-client/slots/8/ifs.stat
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001344
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001344
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001344
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001356
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001356
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001356
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001368
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001368
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001368
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001380
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001380
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001380
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001392
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001392
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001392
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001404
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001404
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001404
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001416
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001416
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001416
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001428
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001428
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001428
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMGGh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMSHh7zg+001440
Adding to the zip: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284044/ICMUAh7zg+001440
Zipping up the final file: /mydisks/b/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
19:29:07 (3228229): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1_r130899678_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 02:41:57 1517679 22440855 oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1 1,295,100 49,244 0.0380
08 Jun 2024 21:27:07 1517679 22440855 oifs_43r3_bl_a1pl_2016092300_20_1017_12284044_1 864,000 33,786 0.0391


©2024 climateprediction.net