climateprediction.net home page
Task 22261869

Task 22261869

Name oifs_43r3_ps_0449_1991050100_123_960_12177093_0
Workunit 12177093
Created 21 Dec 2022, 21:50:27 UTC
Sent 17 Jan 2023, 16:27:01 UTC
Report deadline 18 Mar 2023, 16:27:01 UTC
Received 22 Jan 2023, 10:17:32 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1530639
Run time 1 days 6 hours 52 min 51 sec
CPU time 23 hours 46 min 36 sec
Validate state Invalid
Credit 7,136.64
Device peak FLOPS 3.82 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.20 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,973.70 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
093/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_36.zip
Uploading the intermediate file: upload_file_36.zip
  17:24:15 STEP 1177 H=1177:00 +CPU= 50.576
  17:25:05 STEP 1178 H=1178:00 +CPU= 48.922
  17:26:22 STEP 1179 H=1179:00 +CPU= 74.303
  17:27:16 STEP 1180 H=1180:00 +CPU= 51.449
  17:28:06 STEP 1181 H=1181:00 +CPU= 48.469
  17:29:25 STEP 1182 H=1182:00 +CPU= 76.707
  17:30:17 STEP 1183 H=1183:00 +CPU= 49.548
  17:31:10 STEP 1184 H=1184:00 +CPU= 51.076
  17:32:26 STEP 1185 H=1185:00 +CPU= 72.881
  17:33:19 STEP 1186 H=1186:00 +CPU= 51.436
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0449
(argv4) batchid: 960
(argv5) wuid: 12177093
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_35cbed6bdadc6839af9ec4ddddf06694 to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0449_1991050100_123_960_12177093.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0449_1991050100_123_960_12177093.zip
ic_ancil_file: ic_ancil_12177093
ifsdata_file: ifsdata_12177093
climate_data_file: clim_data_12177093
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f22c73b90bf2406481020dc18b59def6 to: /var/lib/boinc-client/slots/7/ic_ancil_12177093.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12177093.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_75bf3669b28928bc686ff10111c07527 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12177093.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12177093.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12177093.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12177093.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12177093.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12177093.xml
-- Model is restarting --
Adjusting last_iter, 1187, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12177093.xml
last_cpu_time: 61687
upload_file_number: 37
last_iter: 1175
last_upload: 4233600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112
The child process has been launched with process id: 1449586
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:1449586:1449586] [20230121:203825:1674326305:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:39:53 STEP 1176 H=1176:00 +CPU= 81.166
  20:40:54 STEP 1177 H=1177:00 +CPU= 58.439
  20:41:57 STEP 1178 H=1178:00 +CPU= 60.749
  20:43:27 STEP 1179 H=1179:00 +CPU= 86.831
  20:44:26 STEP 1180 H=1180:00 +CPU= 57.016
  20:45:26 STEP 1181 H=1181:00 +CPU= 57.943
  20:47:00 STEP 1182 H=1182:00 +CPU= 91.620
  20:47:58 STEP 1183 H=1183:00 +CPU= 55.965
  20:48:58 STEP 1184 H=1184:00 +CPU= 58.535
  20:50:29 STEP 1185 H=1185:00 +CPU= 87.975
  20:51:27 STEP 1186 H=1186:00 +CPU= 56.690
  20:52:26 STEP 1187 H=1187:00 +CPU= 57.601
  20:54:03 STEP 1188 H=1188:00 +CPU= 92.974
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001188
  20:55:05 STEP 1189 H=1189:00 +CPU= 60.261
  20:56:09 STEP 1190 H=1190:00 +CPU= 62.060
  20:57:41 STEP 1191 H=1191:00 +CPU= 87.480
  20:58:39 STEP 1192 H=1192:00 +CPU= 56.586
  20:59:36 STEP 1193 H=1193:00 +CPU= 55.403
  21:01:07 STEP 1194 H=1194:00 +CPU= 88.108
  21:02:07 STEP 1195 H=1195:00 +CPU= 57.881
  21:03:06 STEP 1196 H=1196:00 +CPU= 56.646
  21:04:32 STEP 1197 H=1197:00 +CPU= 83.786
  21:05:32 STEP 1198 H=1198:00 +CPU= 58.506
  21:06:33 STEP 1199 H=1199:00 +CPU= 58.597
  21:08:14 STEP 1200 H=1200:00 +CPU= 97.146
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_37.zip
Uploading the intermediate file: upload_file_37.zip
  21:09:15 STEP 1201 H=1201:00 +CPU= 57.734
  21:10:19 STEP 1202 H=1202:00 +CPU= 61.556
  21:11:45 STEP 1203 H=1203:00 +CPU= 83.413
  21:12:46 STEP 1204 H=1204:00 +CPU= 57.765
  21:13:44 STEP 1205 H=1205:00 +CPU= 55.369
  21:15:09 STEP 1206 H=1206:00 +CPU= 82.573
  21:16:10 STEP 1207 H=1207:00 +CPU= 58.680
  21:17:11 STEP 1208 H=1208:00 +CPU= 58.786
  21:18:37 STEP 1209 H=1209:00 +CPU= 82.809
  21:19:35 STEP 1210 H=1210:00 +CPU= 56.356
  21:20:33 STEP 1211 H=1211:00 +CPU= 55.840
  21:22:07 STEP 1212 H=1212:00 +CPU= 90.268
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001212
  21:23:07 STEP 1213 H=1213:00 +CPU= 57.737
  21:24:07 STEP 1214 H=1214:00 +CPU= 57.998
  21:25:33 STEP 1215 H=1215:00 +CPU= 83.146
  21:26:31 STEP 1216 H=1216:00 +CPU= 56.644
  21:27:28 STEP 1217 H=1217:00 +CPU= 54.920
  21:28:56 STEP 1218 H=1218:00 +CPU= 85.866
  21:29:54 STEP 1219 H=1219:00 +CPU= 55.773
  21:30:51 STEP 1220 H=1220:00 +CPU= 54.867
  21:32:18 STEP 1221 H=1221:00 +CPU= 84.054
  21:33:21 STEP 1222 H=1222:00 +CPU= 60.017
  21:34:17 STEP 1223 H=1223:00 +CPU= 54.391
  21:36:01 STEP 1224 H=1224:00 +CPU=100.727
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_38.zip
Uploading the intermediate file: upload_file_38.zip
  21:37:02 STEP 1225 H=1225:00 +CPU= 57.223
  21:38:03 STEP 1226 H=1226:00 +CPU= 58.396
  21:39:29 STEP 1227 H=1227:00 +CPU= 83.078
  21:40:30 STEP 1228 H=1228:00 +CPU= 58.490
  21:41:31 STEP 1229 H=1229:00 +CPU= 57.991
  21:42:55 STEP 1230 H=1230:00 +CPU= 81.439
  21:43:55 STEP 1231 H=1231:00 +CPU= 58.427
  21:44:57 STEP 1232 H=1232:00 +CPU= 59.096
  21:46:19 STEP 1233 H=1233:00 +CPU= 78.783
  21:47:18 STEP 1234 H=1234:00 +CPU= 56.877
  21:48:19 STEP 1235 H=1235:00 +CPU= 58.754
  21:49:54 STEP 1236 H=1236:00 +CPU= 92.198
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001236
  21:50:55 STEP 1237 H=1237:00 +CPU= 59.042
  21:51:59 STEP 1238 H=1238:00 +CPU= 61.006
  21:53:28 STEP 1239 H=1239:00 +CPU= 86.215
  21:54:27 STEP 1240 H=1240:00 +CPU= 57.680
  21:55:29 STEP 1241 H=1241:00 +CPU= 59.886
  21:57:00 STEP 1242 H=1242:00 +CPU= 87.990
  21:57:56 STEP 1243 H=1243:00 +CPU= 54.066
  21:58:57 STEP 1244 H=1244:00 +CPU= 58.670
  22:00:29 STEP 1245 H=1245:00 +CPU= 88.836
  22:01:31 STEP 1246 H=1246:00 +CPU= 59.766
  22:02:29 STEP 1247 H=1247:00 +CPU= 56.660
  22:04:17 STEP 1248 H=1248:00 +CPU=103.317
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_39.zip
Uploading the intermediate file: upload_file_39.zip
  22:05:18 STEP 1249 H=1249:00 +CPU= 58.579
  22:06:19 STEP 1250 H=1250:00 +CPU= 59.098
  22:07:44 STEP 1251 H=1251:00 +CPU= 81.724
  22:08:46 STEP 1252 H=1252:00 +CPU= 59.660
  22:09:44 STEP 1253 H=1253:00 +CPU= 56.107
  22:11:14 STEP 1254 H=1254:00 +CPU= 87.240
  22:12:13 STEP 1255 H=1255:00 +CPU= 57.483
  22:13:15 STEP 1256 H=1256:00 +CPU= 59.340
  22:14:43 STEP 1257 H=1257:00 +CPU= 85.284
  22:15:48 STEP 1258 H=1258:00 +CPU= 62.482
  22:16:46 STEP 1259 H=1259:00 +CPU= 56.279
  22:18:22 STEP 1260 H=1260:00 +CPU= 92.131
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001260
  22:19:21 STEP 1261 H=1261:00 +CPU= 56.999
  22:20:22 STEP 1262 H=1262:00 +CPU= 58.448
  22:21:49 STEP 1263 H=1263:00 +CPU= 84.892
  22:22:53 STEP 1264 H=1264:00 +CPU= 60.929
  22:23:51 STEP 1265 H=1265:00 +CPU= 55.827
  22:25:17 STEP 1266 H=1266:00 +CPU= 82.679
  22:26:14 STEP 1267 H=1267:00 +CPU= 54.808
  22:27:15 STEP 1268 H=1268:00 +CPU= 58.574
  22:28:42 STEP 1269 H=1269:00 +CPU= 83.678
  22:29:38 STEP 1270 H=1270:00 +CPU= 54.861
  22:30:39 STEP 1271 H=1271:00 +CPU= 58.300
  22:32:27 STEP 1272 H=1272:00 +CPU=102.857
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_40.zip
Uploading the intermediate file: upload_file_40.zip
  22:33:23 STEP 1273 H=1273:00 +CPU= 53.707
  22:34:26 STEP 1274 H=1274:00 +CPU= 59.726
  22:35:53 STEP 1275 H=1275:00 +CPU= 82.404
  22:36:52 STEP 1276 H=1276:00 +CPU= 56.988
  22:37:47 STEP 1277 H=1277:00 +CPU= 53.185
  22:39:18 STEP 1278 H=1278:00 +CPU= 86.838
  22:40:19 STEP 1279 H=1279:00 +CPU= 57.770
  22:41:19 STEP 1280 H=1280:00 +CPU= 57.682
  22:42:48 STEP 1281 H=1281:00 +CPU= 84.859
  22:43:50 STEP 1282 H=1282:00 +CPU= 59.703
  22:44:49 STEP 1283 H=1283:00 +CPU= 56.001
  22:46:24 STEP 1284 H=1284:00 +CPU= 92.154
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001284
  22:47:26 STEP 1285 H=1285:00 +CPU= 59.067
  22:48:27 STEP 1286 H=1286:00 +CPU= 58.392
  22:49:52 STEP 1287 H=1287:00 +CPU= 80.820
  22:50:52 STEP 1288 H=1288:00 +CPU= 57.914
  22:51:52 STEP 1289 H=1289:00 +CPU= 57.839
  22:53:14 STEP 1290 H=1290:00 +CPU= 79.752
  22:54:15 STEP 1291 H=1291:00 +CPU= 58.446
  22:55:14 STEP 1292 H=1292:00 +CPU= 56.481
  22:56:39 STEP 1293 H=1293:00 +CPU= 81.774
  22:57:38 STEP 1294 H=1294:00 +CPU= 57.595
  22:58:40 STEP 1295 H=1295:00 +CPU= 59.263
  23:00:23 STEP 1296 H=1296:00 +CPU= 99.074
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_41.zip
Uploading the intermediate file: upload_file_41.zip
  23:01:20 STEP 1297 H=1297:00 +CPU= 55.315
  23:02:20 STEP 1298 H=1298:00 +CPU= 56.690
  23:03:49 STEP 1299 H=1299:00 +CPU= 85.248
  23:04:49 STEP 1300 H=1300:00 +CPU= 57.217
  23:05:51 STEP 1301 H=1301:00 +CPU= 59.974
  23:07:19 STEP 1302 H=1302:00 +CPU= 84.332
  23:08:17 STEP 1303 H=1303:00 +CPU= 55.921
  23:09:13 STEP 1304 H=1304:00 +CPU= 54.492
  23:10:41 STEP 1305 H=1305:00 +CPU= 84.579
  23:11:43 STEP 1306 H=1306:00 +CPU= 59.288
  23:12:40 STEP 1307 H=1307:00 +CPU= 55.980
  23:14:15 STEP 1308 H=1308:00 +CPU= 91.072
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001308
  23:15:14 STEP 1309 H=1309:00 +CPU= 56.302
  23:16:14 STEP 1310 H=1310:00 +CPU= 58.047
  23:17:36 STEP 1311 H=1311:00 +CPU= 79.169
  23:18:37 STEP 1312 H=1312:00 +CPU= 58.729
  23:19:35 STEP 1313 H=1313:00 +CPU= 56.238
  23:21:01 STEP 1314 H=1314:00 +CPU= 82.638
  23:22:01 STEP 1315 H=1315:00 +CPU= 58.541
  23:23:05 STEP 1316 H=1316:00 +CPU= 60.917
  23:24:32 STEP 1317 H=1317:00 +CPU= 84.231
  23:25:33 STEP 1318 H=1318:00 +CPU= 59.124
  23:26:35 STEP 1319 H=1319:00 +CPU= 59.175
  23:28:17 STEP 1320 H=1320:00 +CPU= 98.094
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_42.zip
Uploading the intermediate file: upload_file_42.zip
  23:29:14 STEP 1321 H=1321:00 +CPU= 55.273
  23:30:13 STEP 1322 H=1322:00 +CPU= 57.268
  23:31:38 STEP 1323 H=1323:00 +CPU= 81.468
  23:32:35 STEP 1324 H=1324:00 +CPU= 55.182
  23:33:33 STEP 1325 H=1325:00 +CPU= 56.302
  23:35:03 STEP 1326 H=1326:00 +CPU= 86.362
  23:36:02 STEP 1327 H=1327:00 +CPU= 56.115
  23:37:02 STEP 1328 H=1328:00 +CPU= 58.766
  23:38:30 STEP 1329 H=1329:00 +CPU= 84.841
  23:39:30 STEP 1330 H=1330:00 +CPU= 56.786
  23:40:27 STEP 1331 H=1331:00 +CPU= 54.662
  23:42:03 STEP 1332 H=1332:00 +CPU= 92.956
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001332
  23:43:02 STEP 1333 H=1333:00 +CPU= 57.069
  23:44:00 STEP 1334 H=1334:00 +CPU= 56.213
  23:45:28 STEP 1335 H=1335:00 +CPU= 84.901
  23:46:32 STEP 1336 H=1336:00 +CPU= 61.871
  23:47:31 STEP 1337 H=1337:00 +CPU= 56.613
  23:49:01 STEP 1338 H=1338:00 +CPU= 87.160
  23:50:00 STEP 1339 H=1339:00 +CPU= 57.123
  23:51:00 STEP 1340 H=1340:00 +CPU= 58.110
  23:52:25 STEP 1341 H=1341:00 +CPU= 81.888
  23:53:28 STEP 1342 H=1342:00 +CPU= 61.180
  23:54:30 STEP 1343 H=1343:00 +CPU= 60.054
  23:56:12 STEP 1344 H=1344:00 +CPU= 99.511
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_43.zip
Uploading the intermediate file: upload_file_43.zip
  23:57:15 STEP 1345 H=1345:00 +CPU= 60.651
  23:58:17 STEP 1346 H=1346:00 +CPU= 60.076
  23:59:43 STEP 1347 H=1347:00 +CPU= 83.182
  00:00:48 STEP 1348 H=1348:00 +CPU= 60.735
  00:01:47 STEP 1349 H=1349:00 +CPU= 57.866
  00:03:16 STEP 1350 H=1350:00 +CPU= 86.430
  00:04:12 STEP 1351 H=1351:00 +CPU= 53.718
  00:05:15 STEP 1352 H=1352:00 +CPU= 61.188
  00:06:40 STEP 1353 H=1353:00 +CPU= 82.287
  00:07:37 STEP 1354 H=1354:00 +CPU= 55.372
  00:08:36 STEP 1355 H=1355:00 +CPU= 56.726
  00:10:11 STEP 1356 H=1356:00 +CPU= 91.558
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001356
  00:11:11 STEP 1357 H=1357:00 +CPU= 57.662
  00:12:10 STEP 1358 H=1358:00 +CPU= 57.517
  00:13:39 STEP 1359 H=1359:00 +CPU= 85.465
  00:14:38 STEP 1360 H=1360:00 +CPU= 57.038
  00:15:34 STEP 1361 H=1361:00 +CPU= 54.693
  00:17:03 STEP 1362 H=1362:00 +CPU= 86.395
  00:18:08 STEP 1363 H=1363:00 +CPU= 62.837
  00:19:06 STEP 1364 H=1364:00 +CPU= 56.500
  00:20:34 STEP 1365 H=1365:00 +CPU= 85.823
  00:21:36 STEP 1366 H=1366:00 +CPU= 60.225
  00:22:35 STEP 1367 H=1367:00 +CPU= 57.501
  00:24:18 STEP 1368 H=1368:00 +CPU= 99.273
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_44.zip
Uploading the intermediate file: upload_file_44.zip
  00:25:17 STEP 1369 H=1369:00 +CPU= 56.353
  00:26:15 STEP 1370 H=1370:00 +CPU= 56.901
  00:27:44 STEP 1371 H=1371:00 +CPU= 85.347
  00:28:44 STEP 1372 H=1372:00 +CPU= 58.428
  00:29:46 STEP 1373 H=1373:00 +CPU= 60.321
  00:31:10 STEP 1374 H=1374:00 +CPU= 81.607
  00:32:08 STEP 1375 H=1375:00 +CPU= 55.470
  00:33:11 STEP 1376 H=1376:00 +CPU= 60.686
  00:34:35 STEP 1377 H=1377:00 +CPU= 80.911
  00:35:36 STEP 1378 H=1378:00 +CPU= 59.069
  00:36:38 STEP 1379 H=1379:00 +CPU= 59.922
  00:38:17 STEP 1380 H=1380:00 +CPU= 94.589
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001380
  00:39:14 STEP 1381 H=1381:00 +CPU= 55.014
  00:40:16 STEP 1382 H=1382:00 +CPU= 60.755
  00:41:41 STEP 1383 H=1383:00 +CPU= 81.908
  00:42:41 STEP 1384 H=1384:00 +CPU= 57.954
  00:43:40 STEP 1385 H=1385:00 +CPU= 57.302
  00:45:07 STEP 1386 H=1386:00 +CPU= 83.118
  00:46:03 STEP 1387 H=1387:00 +CPU= 55.118
  00:47:00 STEP 1388 H=1388:00 +CPU= 54.689
  00:48:27 STEP 1389 H=1389:00 +CPU= 83.783
  00:49:26 STEP 1390 H=1390:00 +CPU= 57.441
  00:50:26 STEP 1391 H=1391:00 +CPU= 57.667
  00:52:13 STEP 1392 H=1392:00 +CPU=103.401
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_45.zip
Uploading the intermediate file: upload_file_45.zip
  00:53:13 STEP 1393 H=1393:00 +CPU= 57.476
  00:54:11 STEP 1394 H=1394:00 +CPU= 55.500
  00:55:42 STEP 1395 H=1395:00 +CPU= 88.490
  00:56:43 STEP 1396 H=1396:00 +CPU= 59.161
  00:57:43 STEP 1397 H=1397:00 +CPU= 57.708
  00:59:09 STEP 1398 H=1398:00 +CPU= 83.701
  01:00:11 STEP 1399 H=1399:00 +CPU= 59.634
  01:01:12 STEP 1400 H=1400:00 +CPU= 59.040
  01:02:37 STEP 1401 H=1401:00 +CPU= 82.599
  01:03:39 STEP 1402 H=1402:00 +CPU= 59.631
  01:04:41 STEP 1403 H=1403:00 +CPU= 59.510
  01:06:16 STEP 1404 H=1404:00 +CPU= 92.973
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001404
  01:07:20 STEP 1405 H=1405:00 +CPU= 61.416
  01:08:19 STEP 1406 H=1406:00 +CPU= 57.251
  01:09:46 STEP 1407 H=1407:00 +CPU= 83.405
  01:10:48 STEP 1408 H=1408:00 +CPU= 59.953
  01:11:46 STEP 1409 H=1409:00 +CPU= 56.596
  01:13:16 STEP 1410 H=1410:00 +CPU= 87.300
  01:14:12 STEP 1411 H=1411:00 +CPU= 53.587
  01:15:15 STEP 1412 H=1412:00 +CPU= 61.256
  01:16:40 STEP 1413 H=1413:00 +CPU= 82.259
  01:17:40 STEP 1414 H=1414:00 +CPU= 58.083
  01:18:39 STEP 1415 H=1415:00 +CPU= 57.378
  01:20:21 STEP 1416 H=1416:00 +CPU= 98.613
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_46.zip
Uploading the intermediate file: upload_file_46.zip
Uploading trickle at timestep: 5097600
  01:21:23 STEP 1417 H=1417:00 +CPU= 59.324
  01:22:23 STEP 1418 H=1418:00 +CPU= 58.770
  01:23:53 STEP 1419 H=1419:00 +CPU= 86.483
  01:24:53 STEP 1420 H=1420:00 +CPU= 58.629
  01:25:50 STEP 1421 H=1421:00 +CPU= 54.942
  01:27:21 STEP 1422 H=1422:00 +CPU= 87.480
  01:28:19 STEP 1423 H=1423:00 +CPU= 56.231
  01:29:18 STEP 1424 H=1424:00 +CPU= 57.147
  01:30:45 STEP 1425 H=1425:00 +CPU= 84.968
  01:31:47 STEP 1426 H=1426:00 +CPU= 59.559
  01:32:45 STEP 1427 H=1427:00 +CPU= 56.112
  01:34:17 STEP 1428 H=1428:00 +CPU= 90.008
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001428
  01:35:16 STEP 1429 H=1429:00 +CPU= 56.853
  01:36:19 STEP 1430 H=1430:00 +CPU= 60.961
  01:37:45 STEP 1431 H=1431:00 +CPU= 83.382
  01:38:46 STEP 1432 H=1432:00 +CPU= 58.777
  01:39:47 STEP 1433 H=1433:00 +CPU= 59.258
  01:41:15 STEP 1434 H=1434:00 +CPU= 85.485
  01:42:13 STEP 1435 H=1435:00 +CPU= 55.853
  01:43:17 STEP 1436 H=1436:00 +CPU= 61.299
  01:44:44 STEP 1437 H=1437:00 +CPU= 83.970
  01:45:43 STEP 1438 H=1438:00 +CPU= 57.279
  01:46:45 STEP 1439 H=1439:00 +CPU= 60.262
  01:48:27 STEP 1440 H=1440:00 +CPU= 98.749
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_47.zip
Uploading the intermediate file: upload_file_47.zip
  01:49:21 STEP 1441 H=1441:00 +CPU= 52.207
  01:50:25 STEP 1442 H=1442:00 +CPU= 61.598
  01:51:52 STEP 1443 H=1443:00 +CPU= 84.613
  01:52:50 STEP 1444 H=1444:00 +CPU= 55.327
  01:53:49 STEP 1445 H=1445:00 +CPU= 57.270
  01:55:19 STEP 1446 H=1446:00 +CPU= 87.266
  01:56:18 STEP 1447 H=1447:00 +CPU= 56.833
  01:57:16 STEP 1448 H=1448:00 +CPU= 56.561
  01:58:45 STEP 1449 H=1449:00 +CPU= 85.588
  01:59:48 STEP 1450 H=1450:00 +CPU= 60.956
  02:00:44 STEP 1451 H=1451:00 +CPU= 54.281
  02:02:23 STEP 1452 H=1452:00 +CPU= 95.908
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001452
  02:03:19 STEP 1453 H=1453:00 +CPU= 54.334
  02:04:20 STEP 1454 H=1454:00 +CPU= 57.831
  02:05:45 STEP 1455 H=1455:00 +CPU= 82.374
  02:06:46 STEP 1456 H=1456:00 +CPU= 59.333
  02:07:46 STEP 1457 H=1457:00 +CPU= 57.934
  02:09:11 STEP 1458 H=1458:00 +CPU= 82.392
  02:10:10 STEP 1459 H=1459:00 +CPU= 57.541
  02:11:13 STEP 1460 H=1460:00 +CPU= 61.504
  02:12:39 STEP 1461 H=1461:00 +CPU= 83.116
  02:13:39 STEP 1462 H=1462:00 +CPU= 57.521
  02:14:39 STEP 1463 H=1463:00 +CPU= 57.526
  02:16:18 STEP 1464 H=1464:00 +CPU= 96.554
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_48.zip
Uploading the intermediate file: upload_file_48.zip
  02:17:15 STEP 1465 H=1465:00 +CPU= 54.540
  02:18:18 STEP 1466 H=1466:00 +CPU= 61.144
  02:19:46 STEP 1467 H=1467:00 +CPU= 84.665
  02:20:49 STEP 1468 H=1468:00 +CPU= 61.047
  02:21:47 STEP 1469 H=1469:00 +CPU= 56.657
  02:23:14 STEP 1470 H=1470:00 +CPU= 83.751
  02:24:14 STEP 1471 H=1471:00 +CPU= 57.508
  02:25:16 STEP 1472 H=1472:00 +CPU= 59.507
  02:26:40 STEP 1473 H=1473:00 +CPU= 81.141
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0449
(argv4) batchid: 960
(argv5) wuid: 12177093
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_35cbed6bdadc6839af9ec4ddddf06694 to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0449_1991050100_123_960_12177093.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0449_1991050100_123_960_12177093.zip
ic_ancil_file: ic_ancil_12177093
ifsdata_file: ifsdata_12177093
climate_data_file: clim_data_12177093
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f22c73b90bf2406481020dc18b59def6 to: /var/lib/boinc-client/slots/7/ic_ancil_12177093.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12177093.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_75bf3669b28928bc686ff10111c07527 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12177093.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12177093.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12177093.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12177093.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12177093.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12177093.xml
-- Model is restarting --
Adjusting last_iter, 1474, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12177093.xml
last_cpu_time: 81884
upload_file_number: 49
last_iter: 1463
last_upload: 5270400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112
The child process has been launched with process id: 2236682
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2236682:2236682] [20230122:080007:1674367207:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:01:34 STEP 1464 H=1464:00 +CPU= 80.061
  08:02:36 STEP 1465 H=1465:00 +CPU= 59.980
  08:03:40 STEP 1466 H=1466:00 +CPU= 61.772
  08:05:08 STEP 1467 H=1467:00 +CPU= 84.581
  08:06:09 STEP 1468 H=1468:00 +CPU= 58.929
  08:07:10 STEP 1469 H=1469:00 +CPU= 59.362
  08:08:38 STEP 1470 H=1470:00 +CPU= 85.412
  08:09:32 STEP 1471 H=1471:00 +CPU= 52.648
  08:10:36 STEP 1472 H=1472:00 +CPU= 61.935
  08:12:00 STEP 1473 H=1473:00 +CPU= 81.333
  08:13:03 STEP 1474 H=1474:00 +CPU= 61.042
  08:14:03 STEP 1475 H=1475:00 +CPU= 57.728
  08:15:42 STEP 1476 H=1476:00 +CPU= 96.718
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001476
  08:16:43 STEP 1477 H=1477:00 +CPU= 59.231
  08:17:42 STEP 1478 H=1478:00 +CPU= 56.704
  08:19:09 STEP 1479 H=1479:00 +CPU= 83.147
  08:20:10 STEP 1480 H=1480:00 +CPU= 58.852
  08:21:06 STEP 1481 H=1481:00 +CPU= 53.740
  08:22:34 STEP 1482 H=1482:00 +CPU= 86.090
  08:23:33 STEP 1483 H=1483:00 +CPU= 57.457
  08:24:33 STEP 1484 H=1484:00 +CPU= 57.114
  08:25:57 STEP 1485 H=1485:00 +CPU= 81.941
  08:26:58 STEP 1486 H=1486:00 +CPU= 59.154
  08:28:00 STEP 1487 H=1487:00 +CPU= 59.339
  08:29:40 STEP 1488 H=1488:00 +CPU= 96.734
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_49.zip
Uploading the intermediate file: upload_file_49.zip
  08:30:41 STEP 1489 H=1489:00 +CPU= 57.774
  08:31:43 STEP 1490 H=1490:00 +CPU= 59.906
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0449
(argv4) batchid: 960
(argv5) wuid: 12177093
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_35cbed6bdadc6839af9ec4ddddf06694 to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0449_1991050100_123_960_12177093.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0449_1991050100_123_960_12177093.zip
ic_ancil_file: ic_ancil_12177093
ifsdata_file: ifsdata_12177093
climate_data_file: clim_data_12177093
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f22c73b90bf2406481020dc18b59def6 to: /var/lib/boinc-client/slots/7/ic_ancil_12177093.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12177093.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_75bf3669b28928bc686ff10111c07527 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12177093.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12177093.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12177093.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12177093.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12177093.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12177093.xml
-- Model is restarting --
Adjusting last_iter, 1491, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12177093.xml
last_cpu_time: 83717
upload_file_number: 50
last_iter: 1487
last_upload: 5356800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112
The child process has been launched with process id: 2469177
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:LMP:1:1:2469177:2469177] [20230122:104434:1674377074:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:45:56 STEP 1488 H=1488:00 +CPU= 75.851
  10:46:57 STEP 1489 H=1489:00 +CPU= 58.530
  10:47:59 STEP 1490 H=1490:00 +CPU= 60.062
  10:49:31 STEP 1491 H=1491:00 +CPU= 89.063
  10:50:32 STEP 1492 H=1492:00 +CPU= 58.141
  10:51:34 STEP 1493 H=1493:00 +CPU= 60.458
  10:53:07 STEP 1494 H=1494:00 +CPU= 89.721
  10:54:05 STEP 1495 H=1495:00 +CPU= 56.216
  10:55:06 STEP 1496 H=1496:00 +CPU= 58.300
  10:56:33 STEP 1497 H=1497:00 +CPU= 83.506
  10:57:39 STEP 1498 H=1498:00 +CPU= 63.121
  10:58:38 STEP 1499 H=1499:00 +CPU= 57.331
  11:00:21 STEP 1500 H=1500:00 +CPU= 99.687
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001500
  11:01:23 STEP 1501 H=1501:00 +CPU= 59.480
  11:02:24 STEP 1502 H=1502:00 +CPU= 59.885
  11:03:54 STEP 1503 H=1503:00 +CPU= 86.532
  11:04:57 STEP 1504 H=1504:00 +CPU= 61.829
  11:06:01 STEP 1505 H=1505:00 +CPU= 60.363
  11:07:28 STEP 1506 H=1506:00 +CPU= 83.358
  11:08:31 STEP 1507 H=1507:00 +CPU= 61.189
  11:09:34 STEP 1508 H=1508:00 +CPU= 60.293
  11:11:02 STEP 1509 H=1509:00 +CPU= 84.874
  11:12:03 STEP 1510 H=1510:00 +CPU= 59.615
  11:13:05 STEP 1511 H=1511:00 +CPU= 59.582
  11:14:51 STEP 1512 H=1512:00 +CPU=102.002
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177093/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0449_1991050100_123_960_12177093_0_r992297112_50.zip
Uploading the intermediate file: upload_file_50.zip
  11:15:53 STEP 1513 H=1513:00 +CPU= 59.166
  11:16:55 STEP 1514 H=1514:00 +CPU= 60.358
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:34:12 1530639 22261869 oifs_43r3_ps_0449_1991050100_123_960_12177093_0 5,097,600 78,009 0.0153
09 May 2023 16:32:32 1530639 22261869 oifs_43r3_ps_0449_1991050100_123_960_12177093_0 3,369,600 45,414 0.0135
09 May 2023 16:24:55 1530639 22261869 oifs_43r3_ps_0449_1991050100_123_960_12177093_0 864,000 14,081 0.0163


©2024 cpdn.org