climateprediction.net home page
Task 22279844

Task 22279844

Name oifs_43r3_ps_0380_2008050100_123_977_12194024_0
Workunit 12194024
Created 23 Dec 2022, 18:17:05 UTC
Sent 22 Jan 2023, 13:30:20 UTC
Report deadline 23 Mar 2023, 13:30:20 UTC
Received 25 Jan 2023, 22:03:27 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1537562
Run time 15 hours 21 min 35 sec
CPU time 14 hours 36 min 57 sec
Validate state Invalid
Credit 3,870.72
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,787.23 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,753.24 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ojects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_49.zip
Uploading the intermediate file: upload_file_49.zip
  17:23:51 STEP 1201 H=1201:00 +CPU= 43.520
  17:24:29 STEP 1202 H=1202:00 +CPU= 38.090
  17:25:32 STEP 1203 H=1203:00 +CPU= 61.223
  17:26:09 STEP 1204 H=1204:00 +CPU= 36.953
  17:26:44 STEP 1205 H=1205:00 +CPU= 34.181
  17:27:48 STEP 1206 H=1206:00 +CPU= 63.561
  17:28:25 STEP 1207 H=1207:00 +CPU= 35.645
  17:29:10 STEP 1208 H=1208:00 +CPU= 44.696
  17:30:06 STEP 1209 H=1209:00 +CPU= 55.343
  17:30:51 STEP 1210 H=1210:00 +CPU= 44.613
  17:31:25 STEP 1211 H=1211:00 +CPU= 33.437
  17:32:37 STEP 1212 H=1212:00 +CPU= 71.216
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001212
  17:33:10 STEP 1213 H=1213:00 +CPU= 32.237
  17:33:52 STEP 1214 H=1214:00 +CPU= 40.767
  17:34:52 STEP 1215 H=1215:00 +CPU= 59.403
  17:35:34 STEP 1216 H=1216:00 +CPU= 42.002
  17:36:14 STEP 1217 H=1217:00 +CPU= 39.132
  17:37:19 STEP 1218 H=1218:00 +CPU= 63.449
  17:37:58 STEP 1219 H=1219:00 +CPU= 39.021
  17:38:36 STEP 1220 H=1220:00 +CPU= 37.653
  17:39:47 STEP 1221 H=1221:00 +CPU= 65.321
  17:40:27 STEP 1222 H=1222:00 +CPU= 38.473
  17:41:09 STEP 1223 H=1223:00 +CPU= 40.716
  17:42:09 STEP 1224 H=1224:00 +CPU= 58.309
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_50.zip
Uploading the intermediate file: upload_file_50.zip
  17:42:29 STEP 1225 H=1225:00 +CPU= 19.541
  17:42:50 STEP 1226 H=1226:00 +CPU= 20.281
  17:43:25 STEP 1227 H=1227:00 +CPU= 34.277
  17:44:00 STEP 1228 H=1228:00 +CPU= 33.890
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0380
(argv4) batchid: 977
(argv5) wuid: 12194024
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8a51e7655091d36b938f7fcb852794fe to: /var/lib/boinc/slots/16/oifs_43r3_ps_0380_2008050100_123_977_12194024.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0380_2008050100_123_977_12194024.zip
ic_ancil_file: ic_ancil_12194024
ifsdata_file: ifsdata_12194024
climate_data_file: clim_data_12194024
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_81fc5873f3b31049d3bf56e2ed46238a to: /var/lib/boinc/slots/16/ic_ancil_12194024.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12194024.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_653502126c7b806ca15d8d53907f15ea to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12194024.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12194024.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12194024.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12194024.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12194024.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12194024.xml
-- Model is restarting --
Adjusting last_iter, 1229, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12194024.xml
last_cpu_time: 39364
upload_file_number: 51
last_iter: 1223
last_upload: 4406400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728
The child process has been launched with process id: 9144
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:9144:9144] [20230125:174523:1674668723:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:47:34 STEP 1224 H=1224:00 +CPU= 91.951
  17:48:42 STEP 1225 H=1225:00 +CPU= 53.385
  17:49:30 STEP 1226 H=1226:00 +CPU= 45.274
  17:50:41 STEP 1227 H=1227:00 +CPU= 67.249
  17:51:21 STEP 1228 H=1228:00 +CPU= 39.036
  17:52:01 STEP 1229 H=1229:00 +CPU= 38.055
  17:53:08 STEP 1230 H=1230:00 +CPU= 64.209
  17:54:02 STEP 1231 H=1231:00 +CPU= 42.290
  17:54:42 STEP 1232 H=1232:00 +CPU= 38.710
  17:55:51 STEP 1233 H=1233:00 +CPU= 65.283
  17:56:34 STEP 1234 H=1234:00 +CPU= 40.146
  17:57:14 STEP 1235 H=1235:00 +CPU= 37.950
  17:58:33 STEP 1236 H=1236:00 +CPU= 73.208
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001236
  17:59:18 STEP 1237 H=1237:00 +CPU= 41.182
  18:00:07 STEP 1238 H=1238:00 +CPU= 40.996
  18:01:55 STEP 1239 H=1239:00 +CPU= 67.922
  18:02:36 STEP 1240 H=1240:00 +CPU= 39.961
  18:03:15 STEP 1241 H=1241:00 +CPU= 37.527
  18:04:24 STEP 1242 H=1242:00 +CPU= 66.121
  18:05:07 STEP 1243 H=1243:00 +CPU= 36.634
  18:05:47 STEP 1244 H=1244:00 +CPU= 38.365
  18:06:52 STEP 1245 H=1245:00 +CPU= 61.946
  18:07:33 STEP 1246 H=1246:00 +CPU= 39.040
  18:08:15 STEP 1247 H=1247:00 +CPU= 40.039
  18:10:13 STEP 1248 H=1248:00 +CPU= 69.803
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_51.zip
Uploading the intermediate file: upload_file_51.zip
  18:10:45 STEP 1249 H=1249:00 +CPU= 31.639
  18:11:30 STEP 1250 H=1250:00 +CPU= 37.209
  18:12:38 STEP 1251 H=1251:00 +CPU= 58.909
  18:13:22 STEP 1252 H=1252:00 +CPU= 41.607
  18:13:59 STEP 1253 H=1253:00 +CPU= 35.054
  18:15:04 STEP 1254 H=1254:00 +CPU= 63.038
  18:15:41 STEP 1255 H=1255:00 +CPU= 34.896
  18:16:23 STEP 1256 H=1256:00 +CPU= 39.151
  18:17:28 STEP 1257 H=1257:00 +CPU= 58.616
  18:18:16 STEP 1258 H=1258:00 +CPU= 37.899
  18:20:13 STEP 1259 H=1259:00 +CPU= 43.330
  18:24:59 STEP 1260 H=1260:00 +CPU= 67.886
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001260
  18:26:15 STEP 1261 H=1261:00 +CPU= 38.006
  18:26:58 STEP 1262 H=1262:00 +CPU= 41.681
  18:27:57 STEP 1263 H=1263:00 +CPU= 56.741
  18:28:50 STEP 1264 H=1264:00 +CPU= 41.012
  18:29:33 STEP 1265 H=1265:00 +CPU= 36.567
  18:32:13 STEP 1266 H=1266:00 +CPU= 62.785
  18:33:28 STEP 1267 H=1267:00 +CPU= 40.759
  18:35:20 STEP 1268 H=1268:00 +CPU= 41.430
  18:36:30 STEP 1269 H=1269:00 +CPU= 62.618
  18:38:00 STEP 1270 H=1270:00 +CPU= 38.660
  18:38:42 STEP 1271 H=1271:00 +CPU= 38.701
  18:39:56 STEP 1272 H=1272:00 +CPU= 69.292
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_52.zip
Uploading the intermediate file: upload_file_52.zip
  18:40:33 STEP 1273 H=1273:00 +CPU= 35.541
  18:41:15 STEP 1274 H=1274:00 +CPU= 38.976
  18:42:20 STEP 1275 H=1275:00 +CPU= 59.379
  18:43:03 STEP 1276 H=1276:00 +CPU= 40.525
  18:43:45 STEP 1277 H=1277:00 +CPU= 39.613
  18:45:34 STEP 1278 H=1278:00 +CPU= 63.763
  18:46:25 STEP 1279 H=1279:00 +CPU= 37.991
  18:47:09 STEP 1280 H=1280:00 +CPU= 42.552
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0380
(argv4) batchid: 977
(argv5) wuid: 12194024
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8a51e7655091d36b938f7fcb852794fe to: /var/lib/boinc/slots/16/oifs_43r3_ps_0380_2008050100_123_977_12194024.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0380_2008050100_123_977_12194024.zip
ic_ancil_file: ic_ancil_12194024
ifsdata_file: ifsdata_12194024
climate_data_file: clim_data_12194024
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_81fc5873f3b31049d3bf56e2ed46238a to: /var/lib/boinc/slots/16/ic_ancil_12194024.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12194024.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_653502126c7b806ca15d8d53907f15ea to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12194024.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12194024.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12194024.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12194024.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12194024.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12194024.xml
-- Model is restarting --
Adjusting last_iter, 1281, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12194024.xml
last_cpu_time: 42131
upload_file_number: 53
last_iter: 1271
last_upload: 4579200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728
The child process has been launched with process id: 12424
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:12424:12424] [20230125:184838:1674672518:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:51:04 STEP 1272 H=1272:00 +CPU= 81.686
  18:52:25 STEP 1273 H=1273:00 +CPU= 47.604
  18:53:07 STEP 1274 H=1274:00 +CPU= 39.898
  18:54:38 STEP 1275 H=1275:00 +CPU= 63.224
  18:55:19 STEP 1276 H=1276:00 +CPU= 38.205
  18:56:02 STEP 1277 H=1277:00 +CPU= 40.747
  18:58:47 STEP 1278 H=1278:00 +CPU= 55.873
  18:59:19 STEP 1279 H=1279:00 +CPU= 31.177
  18:59:53 STEP 1280 H=1280:00 +CPU= 33.337
  19:00:46 STEP 1281 H=1281:00 +CPU= 50.878
  19:01:21 STEP 1282 H=1282:00 +CPU= 34.878
  19:02:06 STEP 1283 H=1283:00 +CPU= 32.609
  19:04:08 STEP 1284 H=1284:00 +CPU= 56.859
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001284
  19:04:41 STEP 1285 H=1285:00 +CPU= 32.298
  19:05:15 STEP 1286 H=1286:00 +CPU= 33.298
  19:06:10 STEP 1287 H=1287:00 +CPU= 53.322
  19:06:44 STEP 1288 H=1288:00 +CPU= 32.548
  19:07:20 STEP 1289 H=1289:00 +CPU= 34.841
  19:08:11 STEP 1290 H=1290:00 +CPU= 50.066
  19:08:49 STEP 1291 H=1291:00 +CPU= 36.006
  19:09:23 STEP 1292 H=1292:00 +CPU= 33.059
  19:10:17 STEP 1293 H=1293:00 +CPU= 52.553
  19:10:51 STEP 1294 H=1294:00 +CPU= 32.840
  19:11:25 STEP 1295 H=1295:00 +CPU= 33.459
  19:12:29 STEP 1296 H=1296:00 +CPU= 61.395
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_53.zip
Uploading the intermediate file: upload_file_53.zip
  19:13:01 STEP 1297 H=1297:00 +CPU= 31.069
  19:13:38 STEP 1298 H=1298:00 +CPU= 35.657
  19:14:29 STEP 1299 H=1299:00 +CPU= 49.274
  19:15:06 STEP 1300 H=1300:00 +CPU= 36.417
  19:15:37 STEP 1301 H=1301:00 +CPU= 29.758
  19:16:34 STEP 1302 H=1302:00 +CPU= 55.311
  19:17:05 STEP 1303 H=1303:00 +CPU= 30.095
  19:17:40 STEP 1304 H=1304:00 +CPU= 33.048
  19:18:36 STEP 1305 H=1305:00 +CPU= 54.240
  19:19:09 STEP 1306 H=1306:00 +CPU= 32.020
  19:19:46 STEP 1307 H=1307:00 +CPU= 36.700
  19:20:43 STEP 1308 H=1308:00 +CPU= 54.933
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001308
  19:21:21 STEP 1309 H=1309:00 +CPU= 36.180
  19:21:50 STEP 1310 H=1310:00 +CPU= 28.634
  19:22:50 STEP 1311 H=1311:00 +CPU= 57.758
  19:23:20 STEP 1312 H=1312:00 +CPU= 28.868
  19:23:56 STEP 1313 H=1313:00 +CPU= 34.826
  19:24:50 STEP 1314 H=1314:00 +CPU= 52.470
  19:25:24 STEP 1315 H=1315:00 +CPU= 32.838
  19:26:01 STEP 1316 H=1316:00 +CPU= 35.946
  19:26:52 STEP 1317 H=1317:00 +CPU= 49.351
  19:27:29 STEP 1318 H=1318:00 +CPU= 35.841
  19:28:02 STEP 1319 H=1319:00 +CPU= 31.950
  19:29:07 STEP 1320 H=1320:00 +CPU= 62.873
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_54.zip
Uploading the intermediate file: upload_file_54.zip
  19:29:39 STEP 1321 H=1321:00 +CPU= 31.206
  19:30:15 STEP 1322 H=1322:00 +CPU= 34.969
  19:31:12 STEP 1323 H=1323:00 +CPU= 54.683
  19:31:50 STEP 1324 H=1324:00 +CPU= 36.732
  19:32:29 STEP 1325 H=1325:00 +CPU= 36.779
  19:33:23 STEP 1326 H=1326:00 +CPU= 52.567
  19:34:00 STEP 1327 H=1327:00 +CPU= 35.727
  19:34:36 STEP 1328 H=1328:00 +CPU= 34.690
  19:35:44 STEP 1329 H=1329:00 +CPU= 62.589
  19:36:17 STEP 1330 H=1330:00 +CPU= 32.634
  19:36:55 STEP 1331 H=1331:00 +CPU= 36.561
  19:37:58 STEP 1332 H=1332:00 +CPU= 61.718
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001332
  19:38:37 STEP 1333 H=1333:00 +CPU= 37.555
  19:39:27 STEP 1334 H=1334:00 +CPU= 37.663
  19:40:27 STEP 1335 H=1335:00 +CPU= 59.099
  19:41:01 STEP 1336 H=1336:00 +CPU= 32.696
  19:41:42 STEP 1337 H=1337:00 +CPU= 39.903
  19:42:45 STEP 1338 H=1338:00 +CPU= 61.662
  19:43:23 STEP 1339 H=1339:00 +CPU= 37.108
  19:43:59 STEP 1340 H=1340:00 +CPU= 35.595
  19:44:54 STEP 1341 H=1341:00 +CPU= 53.776
  19:45:33 STEP 1342 H=1342:00 +CPU= 38.096
  19:46:14 STEP 1343 H=1343:00 +CPU= 39.174
  19:47:25 STEP 1344 H=1344:00 +CPU= 69.119
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_55.zip
Uploading the intermediate file: upload_file_55.zip
  19:48:10 STEP 1345 H=1345:00 +CPU= 42.442
  19:48:49 STEP 1346 H=1346:00 +CPU= 38.085
  19:49:50 STEP 1347 H=1347:00 +CPU= 59.799
  19:50:23 STEP 1348 H=1348:00 +CPU= 32.149
  19:50:58 STEP 1349 H=1349:00 +CPU= 33.686
  19:51:56 STEP 1350 H=1350:00 +CPU= 57.004
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0380
(argv4) batchid: 977
(argv5) wuid: 12194024
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8a51e7655091d36b938f7fcb852794fe to: /var/lib/boinc/slots/16/oifs_43r3_ps_0380_2008050100_123_977_12194024.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0380_2008050100_123_977_12194024.zip
ic_ancil_file: ic_ancil_12194024
ifsdata_file: ifsdata_12194024
climate_data_file: clim_data_12194024
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_81fc5873f3b31049d3bf56e2ed46238a to: /var/lib/boinc/slots/16/ic_ancil_12194024.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12194024.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_653502126c7b806ca15d8d53907f15ea to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12194024.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12194024.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12194024.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12194024.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12194024.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12194024.xml
-- Model is restarting --
Adjusting last_iter, 1350, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12194024.xml
last_cpu_time: 45507
upload_file_number: 56
last_iter: 1343
last_upload: 4838400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728
The child process has been launched with process id: 15154
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15154:15154] [20230125:195310:1674676390:0.003] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:54:50 STEP 1344 H=1344:00 +CPU= 85.819
  19:55:50 STEP 1345 H=1345:00 +CPU= 46.158
  19:56:31 STEP 1346 H=1346:00 +CPU= 39.032
  19:57:42 STEP 1347 H=1347:00 +CPU= 68.252
  19:58:23 STEP 1348 H=1348:00 +CPU= 39.608
  19:59:09 STEP 1349 H=1349:00 +CPU= 44.206
  20:00:04 STEP 1350 H=1350:00 +CPU= 53.228
  20:00:42 STEP 1351 H=1351:00 +CPU= 36.979
  20:01:15 STEP 1352 H=1352:00 +CPU= 31.721
  20:02:27 STEP 1353 H=1353:00 +CPU= 69.211
  20:03:06 STEP 1354 H=1354:00 +CPU= 38.346
  20:03:53 STEP 1355 H=1355:00 +CPU= 45.085
  20:05:03 STEP 1356 H=1356:00 +CPU= 68.764
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001356
  20:05:46 STEP 1357 H=1357:00 +CPU= 41.476
  20:06:25 STEP 1358 H=1358:00 +CPU= 38.234
  20:07:35 STEP 1359 H=1359:00 +CPU= 64.786
  20:08:13 STEP 1360 H=1360:00 +CPU= 37.269
  20:08:57 STEP 1361 H=1361:00 +CPU= 42.386
  20:10:01 STEP 1362 H=1362:00 +CPU= 63.149
  20:10:43 STEP 1363 H=1363:00 +CPU= 40.265
  20:11:24 STEP 1364 H=1364:00 +CPU= 39.791
  20:12:27 STEP 1365 H=1365:00 +CPU= 61.807
  20:13:13 STEP 1366 H=1366:00 +CPU= 44.705
  20:13:54 STEP 1367 H=1367:00 +CPU= 40.530
  20:15:14 STEP 1368 H=1368:00 +CPU= 77.630
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_56.zip
Uploading the intermediate file: upload_file_56.zip
  20:15:48 STEP 1369 H=1369:00 +CPU= 33.361
  20:16:30 STEP 1370 H=1370:00 +CPU= 40.398
  20:17:32 STEP 1371 H=1371:00 +CPU= 61.087
  20:18:19 STEP 1372 H=1372:00 +CPU= 45.915
  20:19:01 STEP 1373 H=1373:00 +CPU= 40.193
  20:20:05 STEP 1374 H=1374:00 +CPU= 62.528
  20:20:44 STEP 1375 H=1375:00 +CPU= 37.151
  20:21:35 STEP 1376 H=1376:00 +CPU= 48.220
  20:22:55 STEP 1377 H=1377:00 +CPU= 77.466
  20:23:42 STEP 1378 H=1378:00 +CPU= 46.057
  20:24:38 STEP 1379 H=1379:00 +CPU= 54.493
  20:26:03 STEP 1380 H=1380:00 +CPU= 83.616
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001380
  20:26:53 STEP 1381 H=1381:00 +CPU= 49.245
  20:27:40 STEP 1382 H=1382:00 +CPU= 46.345
  20:28:58 STEP 1383 H=1383:00 +CPU= 77.773
  20:29:48 STEP 1384 H=1384:00 +CPU= 48.667
  20:30:38 STEP 1385 H=1385:00 +CPU= 49.674
  20:31:58 STEP 1386 H=1386:00 +CPU= 78.253
  20:32:49 STEP 1387 H=1387:00 +CPU= 50.846
  20:33:42 STEP 1388 H=1388:00 +CPU= 52.150
  20:35:00 STEP 1389 H=1389:00 +CPU= 77.093
  20:35:52 STEP 1390 H=1390:00 +CPU= 51.330
  20:36:43 STEP 1391 H=1391:00 +CPU= 49.587
  20:38:15 STEP 1392 H=1392:00 +CPU= 90.915
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_57.zip
Uploading the intermediate file: upload_file_57.zip
  20:39:05 STEP 1393 H=1393:00 +CPU= 49.178
  20:39:59 STEP 1394 H=1394:00 +CPU= 53.129
  20:41:19 STEP 1395 H=1395:00 +CPU= 78.659
  20:42:10 STEP 1396 H=1396:00 +CPU= 50.831
  20:42:56 STEP 1397 H=1397:00 +CPU= 45.174
  20:44:15 STEP 1398 H=1398:00 +CPU= 77.253
  20:45:03 STEP 1399 H=1399:00 +CPU= 47.535
  20:45:57 STEP 1400 H=1400:00 +CPU= 53.491
  20:47:14 STEP 1401 H=1401:00 +CPU= 75.357
  20:48:06 STEP 1402 H=1402:00 +CPU= 51.816
  20:49:00 STEP 1403 H=1403:00 +CPU= 53.086
  20:50:23 STEP 1404 H=1404:00 +CPU= 81.336
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001404
  20:51:10 STEP 1405 H=1405:00 +CPU= 45.758
  20:51:56 STEP 1406 H=1406:00 +CPU= 45.212
  20:53:09 STEP 1407 H=1407:00 +CPU= 71.887
  20:53:54 STEP 1408 H=1408:00 +CPU= 44.060
  20:54:44 STEP 1409 H=1409:00 +CPU= 49.117
  20:55:54 STEP 1410 H=1410:00 +CPU= 68.571
  20:56:43 STEP 1411 H=1411:00 +CPU= 48.231
  20:57:29 STEP 1412 H=1412:00 +CPU= 44.504
  20:58:43 STEP 1413 H=1413:00 +CPU= 72.475
  20:59:28 STEP 1414 H=1414:00 +CPU= 44.854
  21:00:19 STEP 1415 H=1415:00 +CPU= 49.532
  21:01:44 STEP 1416 H=1416:00 +CPU= 82.869
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_58.zip
Uploading the intermediate file: upload_file_58.zip
  21:02:33 STEP 1417 H=1417:00 +CPU= 48.029
  21:03:20 STEP 1418 H=1418:00 +CPU= 46.760
  21:04:35 STEP 1419 H=1419:00 +CPU= 71.768
  21:05:23 STEP 1420 H=1420:00 +CPU= 45.654
  21:06:12 STEP 1421 H=1421:00 +CPU= 45.337
  21:07:27 STEP 1422 H=1422:00 +CPU= 71.073
  21:08:15 STEP 1423 H=1423:00 +CPU= 46.381
  21:09:05 STEP 1424 H=1424:00 +CPU= 47.636
  21:10:19 STEP 1425 H=1425:00 +CPU= 70.853
  21:11:08 STEP 1426 H=1426:00 +CPU= 46.701
  21:11:57 STEP 1427 H=1427:00 +CPU= 47.700
  21:13:19 STEP 1428 H=1428:00 +CPU= 79.410
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001428
  21:14:07 STEP 1429 H=1429:00 +CPU= 46.113
  21:14:55 STEP 1430 H=1430:00 +CPU= 45.653
  21:16:11 STEP 1431 H=1431:00 +CPU= 73.490
  21:16:58 STEP 1432 H=1432:00 +CPU= 45.942
  21:17:50 STEP 1433 H=1433:00 +CPU= 49.518
  21:19:03 STEP 1434 H=1434:00 +CPU= 70.630
  21:19:53 STEP 1435 H=1435:00 +CPU= 48.497
  21:20:40 STEP 1436 H=1436:00 +CPU= 44.869
  21:21:57 STEP 1437 H=1437:00 +CPU= 73.989
  21:22:45 STEP 1438 H=1438:00 +CPU= 45.680
  21:23:41 STEP 1439 H=1439:00 +CPU= 51.980
  21:25:06 STEP 1440 H=1440:00 +CPU= 81.995
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728_59.zip
Uploading the intermediate file: upload_file_59.zip
  21:25:55 STEP 1441 H=1441:00 +CPU= 46.510
  21:26:41 STEP 1442 H=1442:00 +CPU= 39.027
  21:28:00 STEP 1443 H=1443:00 +CPU= 72.493
  21:28:43 STEP 1444 H=1444:00 +CPU= 41.902
  21:29:17 STEP 1445 H=1445:00 +CPU= 33.007
  21:29:50 STEP 1446 H=1446:00 +CPU= 33.154
  21:30:12 STEP 1447 H=1447:00 +CPU= 21.339
  21:30:40 STEP 1448 H=1448:00 +CPU= 27.090
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0380
(argv4) batchid: 977
(argv5) wuid: 12194024
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194024
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8a51e7655091d36b938f7fcb852794fe to: /var/lib/boinc/slots/16/oifs_43r3_ps_0380_2008050100_123_977_12194024.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0380_2008050100_123_977_12194024.zip
ic_ancil_file: ic_ancil_12194024
ifsdata_file: ifsdata_12194024
climate_data_file: clim_data_12194024
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_81fc5873f3b31049d3bf56e2ed46238a to: /var/lib/boinc/slots/16/ic_ancil_12194024.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12194024.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_653502126c7b806ca15d8d53907f15ea to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12194024.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12194024.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12194024.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12194024.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12194024.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12194024.xml
-- Model is restarting --
Adjusting last_iter, 1449, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12194024.xml
last_cpu_time: 51201
upload_file_number: 60
last_iter: 1439
last_upload: 5184000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0380_2008050100_123_977_12194024_0_r800914728
The child process has been launched with process id: 4205
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:4205:4205] [20230125:213705:1674682625:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:37:48 STEP 1440 H=1440:00 +CPU= 30.816
  21:38:30 STEP 1441 H=1441:00 +CPU= 38.346
  21:39:22 STEP 1442 H=1442:00 +CPU= 49.361
  21:40:36 STEP 1443 H=1443:00 +CPU= 72.028
  21:41:26 STEP 1444 H=1444:00 +CPU= 48.564
  21:42:11 STEP 1445 H=1445:00 +CPU= 43.514
  21:43:32 STEP 1446 H=1446:00 +CPU= 78.992
  21:44:18 STEP 1447 H=1447:00 +CPU= 44.757
  21:45:09 STEP 1448 H=1448:00 +CPU= 48.440
  21:46:28 STEP 1449 H=1449:00 +CPU= 72.864
  21:47:19 STEP 1450 H=1450:00 +CPU= 49.280
  21:48:03 STEP 1451 H=1451:00 +CPU= 42.458
  21:49:29 STEP 1452 H=1452:00 +CPU= 83.878
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001452
  21:50:15 STEP 1453 H=1453:00 +CPU= 44.665
  21:51:13 STEP 1454 H=1454:00 +CPU= 55.350
  21:52:27 STEP 1455 H=1455:00 +CPU= 71.842
  21:53:26 STEP 1456 H=1456:00 +CPU= 55.838
  21:54:10 STEP 1457 H=1457:00 +CPU= 42.320
  21:55:39 STEP 1458 H=1458:00 +CPU= 83.498
  21:56:20 STEP 1459 H=1459:00 +CPU= 39.291
  21:57:11 STEP 1460 H=1460:00 +CPU= 48.943
  21:58:26 STEP 1461 H=1461:00 +CPU= 70.574
  21:59:22 STEP 1462 H=1462:00 +CPU= 52.625
  22:00:18 STEP 1463 H=1463:00 +CPU= 51.611
  22:01:54 STEP 1464 H=1464:00 +CPU= 89.579
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:39:43 1537562 22279844 oifs_43r3_ps_0380_2008050100_123_977_12194024_0 2,764,800 18,442 0.0067
09 May 2023 16:38:09 1537562 22279844 oifs_43r3_ps_0380_2008050100_123_977_12194024_0 1,728,000 10,389 0.0060
09 May 2023 16:35:56 1537562 22279844 oifs_43r3_ps_0380_2008050100_123_977_12194024_0 864,000 5,125 0.0059


©2024 cpdn.org