climateprediction.net home page
Task 22293783

Task 22293783

Name oifs_43r3_ps_0878_2020050100_123_989_12206522_1
Workunit 12206522
Created 27 Dec 2022, 0:33:04 UTC
Sent 16 Jan 2023, 15:20:24 UTC
Report deadline 17 Mar 2023, 15:20:24 UTC
Received 18 Jan 2023, 21:26:05 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1516274
Run time 19 hours 42 min 46 sec
CPU time 15 hours 12 min 28 sec
Validate state Invalid
Credit 7,741.44
Device peak FLOPS 2.32 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,788.64 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.26 MB
Stderr 
<core_client_version>7.21.0</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 1286 H=1286:00 +CPU= 25.562
  21:52:20 STEP 1287 H=1287:00 +CPU= 39.430
  21:52:52 STEP 1288 H=1288:00 +CPU= 25.703
  21:53:23 STEP 1289 H=1289:00 +CPU= 25.185
  21:54:12 STEP 1290 H=1290:00 +CPU= 39.784
  21:54:43 STEP 1291 H=1291:00 +CPU= 24.536
  21:55:19 STEP 1292 H=1292:00 +CPU= 26.372
  21:56:09 STEP 1293 H=1293:00 +CPU= 39.390
  21:56:40 STEP 1294 H=1294:00 +CPU= 25.508
  21:57:12 STEP 1295 H=1295:00 +CPU= 26.025
  21:58:10 STEP 1296 H=1296:00 +CPU= 45.887
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001296
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001296
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001296
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001272
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001272
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001272
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001284
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001284
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001284
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_53.zip
Uploading the intermediate file: upload_file_53.zip
  21:58:43 STEP 1297 H=1297:00 +CPU= 24.462
  21:59:25 STEP 1298 H=1298:00 +CPU= 26.020
  22:00:18 STEP 1299 H=1299:00 +CPU= 40.026
  22:00:50 STEP 1300 H=1300:00 +CPU= 26.015
  22:01:22 STEP 1301 H=1301:00 +CPU= 25.335
  22:02:11 STEP 1302 H=1302:00 +CPU= 40.523
  22:02:44 STEP 1303 H=1303:00 +CPU= 25.065
  22:03:22 STEP 1304 H=1304:00 +CPU= 26.357
  22:04:14 STEP 1305 H=1305:00 +CPU= 39.408
  22:04:46 STEP 1306 H=1306:00 +CPU= 25.583
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0878
(argv4) batchid: 989
(argv5) wuid: 12206522
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/dan/BOINC/slots/0
Project directory is: /home/dan/BOINC/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522
..mkdir for temp folder for results failed
Copying: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/dan/BOINC/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/dan/BOINC/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ccaef69f7085177ec9c5d74221da607c to: /home/dan/BOINC/slots/0/oifs_43r3_ps_0878_2020050100_123_989_12206522.zip
Unzipping the namelist zip file: /home/dan/BOINC/slots/0/oifs_43r3_ps_0878_2020050100_123_989_12206522.zip
ic_ancil_file: ic_ancil_12206522
ifsdata_file: ifsdata_12206522
climate_data_file: clim_data_12206522
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1cdafa83e982a6f6ec0bb2b89cd3bbd6 to: /home/dan/BOINC/slots/0/ic_ancil_12206522.zip
Unzipping the IC ancils zip file: /home/dan/BOINC/slots/0/ic_ancil_12206522.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ee3163b6468ddd19ed08128da1343ba6 to: /home/dan/BOINC/slots/0/ifsdata/ifsdata_12206522.zip
Unzipping the ifsdata_zip file: /home/dan/BOINC/slots/0/ifsdata/ifsdata_12206522.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/dan/BOINC/slots/0/159l_2/clim_data_12206522.zip
Unzipping the climate data zip file: /home/dan/BOINC/slots/0/159l_2/clim_data_12206522.zip
Checking for progress XML file: /home/dan/BOINC/slots/0/progress_file_12206522.xml
Opened progress file ok : /home/dan/BOINC/slots/0/progress_file_12206522.xml
-- Model is restarting --
Adjusting last_iter, 1307, to previous model restart step.
Creating progress file: /home/dan/BOINC/slots/0/progress_file_12206522.xml
last_cpu_time: 42379
upload_file_number: 54
last_iter: 1295
last_upload: 4665600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524
The child process has been launched with process id: 77900
Executing the command: /home/dan/BOINC/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:77900:77900] [20230117:223517:1673991317:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:36:24 STEP 1296 H=1296:00 +CPU= 49.292
  22:37:00 STEP 1297 H=1297:00 +CPU= 26.786
  22:37:38 STEP 1298 H=1298:00 +CPU= 27.613
  22:38:32 STEP 1299 H=1299:00 +CPU= 43.107
  22:39:06 STEP 1300 H=1300:00 +CPU= 27.900
  22:39:42 STEP 1301 H=1301:00 +CPU= 27.010
  22:40:55 STEP 1302 H=1302:00 +CPU= 44.307
  22:41:30 STEP 1303 H=1303:00 +CPU= 27.185
  22:42:07 STEP 1304 H=1304:00 +CPU= 28.393
  22:43:02 STEP 1305 H=1305:00 +CPU= 43.485
  22:43:39 STEP 1306 H=1306:00 +CPU= 28.982
  22:44:14 STEP 1307 H=1307:00 +CPU= 27.650
  22:45:20 STEP 1308 H=1308:00 +CPU= 48.136
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001308
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001308
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001308
  22:45:57 STEP 1309 H=1309:00 +CPU= 27.342
  22:46:33 STEP 1310 H=1310:00 +CPU= 28.331
  22:47:26 STEP 1311 H=1311:00 +CPU= 42.894
  22:48:02 STEP 1312 H=1312:00 +CPU= 28.306
  22:48:37 STEP 1313 H=1313:00 +CPU= 27.352
  22:49:33 STEP 1314 H=1314:00 +CPU= 44.016
  22:50:08 STEP 1315 H=1315:00 +CPU= 27.583
  22:50:44 STEP 1316 H=1316:00 +CPU= 28.580
  22:51:38 STEP 1317 H=1317:00 +CPU= 43.236
  22:52:12 STEP 1318 H=1318:00 +CPU= 27.788
  22:52:46 STEP 1319 H=1319:00 +CPU= 28.058
  22:53:51 STEP 1320 H=1320:00 +CPU= 50.468
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001320
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001320
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001320
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001296
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001296
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001296
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001308
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001308
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001308
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_54.zip
Uploading the intermediate file: upload_file_54.zip
  22:54:29 STEP 1321 H=1321:00 +CPU= 27.430
  22:55:08 STEP 1322 H=1322:00 +CPU= 28.375
  22:56:09 STEP 1323 H=1323:00 +CPU= 45.820
  22:56:49 STEP 1324 H=1324:00 +CPU= 30.977
  22:57:24 STEP 1325 H=1325:00 +CPU= 27.573
  22:58:40 STEP 1326 H=1326:00 +CPU= 44.646
  22:59:13 STEP 1327 H=1327:00 +CPU= 26.928
  22:59:50 STEP 1328 H=1328:00 +CPU= 28.413
  23:00:52 STEP 1329 H=1329:00 +CPU= 44.507
  23:01:29 STEP 1330 H=1330:00 +CPU= 28.995
  23:02:04 STEP 1331 H=1331:00 +CPU= 27.681
  23:03:07 STEP 1332 H=1332:00 +CPU= 48.380
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001332
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001332
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001332
  23:03:44 STEP 1333 H=1333:00 +CPU= 27.498
  23:04:20 STEP 1334 H=1334:00 +CPU= 28.572
  23:05:17 STEP 1335 H=1335:00 +CPU= 43.098
  23:05:54 STEP 1336 H=1336:00 +CPU= 28.196
  23:06:28 STEP 1337 H=1337:00 +CPU= 27.009
  23:07:21 STEP 1338 H=1338:00 +CPU= 43.405
  23:07:56 STEP 1339 H=1339:00 +CPU= 27.377
  23:08:36 STEP 1340 H=1340:00 +CPU= 28.860
  23:09:29 STEP 1341 H=1341:00 +CPU= 42.595
  23:10:19 STEP 1342 H=1342:00 +CPU= 28.521
  23:11:00 STEP 1343 H=1343:00 +CPU= 28.399
  23:12:02 STEP 1344 H=1344:00 +CPU= 50.250
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001344
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001344
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001344
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001320
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001320
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001320
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001332
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001332
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001332
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_55.zip
Uploading the intermediate file: upload_file_55.zip
  23:12:37 STEP 1345 H=1345:00 +CPU= 27.124
  23:13:12 STEP 1346 H=1346:00 +CPU= 28.090
  23:14:06 STEP 1347 H=1347:00 +CPU= 42.837
  23:14:42 STEP 1348 H=1348:00 +CPU= 28.273
  23:15:17 STEP 1349 H=1349:00 +CPU= 27.252
  23:16:11 STEP 1350 H=1350:00 +CPU= 43.689
  23:16:45 STEP 1351 H=1351:00 +CPU= 27.012
  23:17:20 STEP 1352 H=1352:00 +CPU= 27.907
  23:18:14 STEP 1353 H=1353:00 +CPU= 43.048
  23:18:51 STEP 1354 H=1354:00 +CPU= 28.661
  23:19:25 STEP 1355 H=1355:00 +CPU= 27.744
  23:20:24 STEP 1356 H=1356:00 +CPU= 47.624
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001356
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001356
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001356
  23:20:58 STEP 1357 H=1357:00 +CPU= 27.301
  23:21:33 STEP 1358 H=1358:00 +CPU= 28.517
  23:22:28 STEP 1359 H=1359:00 +CPU= 43.094
  23:23:03 STEP 1360 H=1360:00 +CPU= 28.218
  23:23:37 STEP 1361 H=1361:00 +CPU= 27.134
  23:24:42 STEP 1362 H=1362:00 +CPU= 45.505
  23:25:16 STEP 1363 H=1363:00 +CPU= 27.404
  23:25:55 STEP 1364 H=1364:00 +CPU= 30.597
  23:26:59 STEP 1365 H=1365:00 +CPU= 48.163
  23:27:35 STEP 1366 H=1366:00 +CPU= 28.054
  23:28:12 STEP 1367 H=1367:00 +CPU= 28.330
  23:29:17 STEP 1368 H=1368:00 +CPU= 50.820
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001368
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001368
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001368
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001344
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001344
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001344
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001356
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001356
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001356
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_56.zip
Uploading the intermediate file: upload_file_56.zip
  23:29:52 STEP 1369 H=1369:00 +CPU= 27.577
  23:30:28 STEP 1370 H=1370:00 +CPU= 28.514
  23:31:23 STEP 1371 H=1371:00 +CPU= 43.343
  23:31:59 STEP 1372 H=1372:00 +CPU= 28.960
  23:32:35 STEP 1373 H=1373:00 +CPU= 28.345
  23:33:40 STEP 1374 H=1374:00 +CPU= 49.918
  23:34:16 STEP 1375 H=1375:00 +CPU= 27.654
  23:34:52 STEP 1376 H=1376:00 +CPU= 28.489
  23:35:47 STEP 1377 H=1377:00 +CPU= 42.705
  23:36:25 STEP 1378 H=1378:00 +CPU= 27.531
  23:37:02 STEP 1379 H=1379:00 +CPU= 27.055
  23:38:05 STEP 1380 H=1380:00 +CPU= 48.224
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001380
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001380
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001380
  23:38:40 STEP 1381 H=1381:00 +CPU= 27.673
  23:39:15 STEP 1382 H=1382:00 +CPU= 28.304
  23:40:08 STEP 1383 H=1383:00 +CPU= 42.641
  23:40:43 STEP 1384 H=1384:00 +CPU= 28.053
  23:41:18 STEP 1385 H=1385:00 +CPU= 27.613
  23:42:12 STEP 1386 H=1386:00 +CPU= 43.843
  23:42:46 STEP 1387 H=1387:00 +CPU= 27.097
  23:43:22 STEP 1388 H=1388:00 +CPU= 28.647
  23:44:17 STEP 1389 H=1389:00 +CPU= 42.710
  23:44:57 STEP 1390 H=1390:00 +CPU= 27.961
  23:45:35 STEP 1391 H=1391:00 +CPU= 28.248
  23:46:42 STEP 1392 H=1392:00 +CPU= 50.508
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001392
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001392
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001392
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001368
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001368
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001368
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001380
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001380
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001380
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_57.zip
Uploading the intermediate file: upload_file_57.zip
  23:47:22 STEP 1393 H=1393:00 +CPU= 27.128
  23:48:00 STEP 1394 H=1394:00 +CPU= 28.164
  23:48:54 STEP 1395 H=1395:00 +CPU= 43.038
  23:49:29 STEP 1396 H=1396:00 +CPU= 28.375
  23:50:03 STEP 1397 H=1397:00 +CPU= 27.109
  23:50:58 STEP 1398 H=1398:00 +CPU= 43.639
  23:51:31 STEP 1399 H=1399:00 +CPU= 27.023
  23:52:12 STEP 1400 H=1400:00 +CPU= 29.044
  23:53:10 STEP 1401 H=1401:00 +CPU= 43.670
  23:53:45 STEP 1402 H=1402:00 +CPU= 28.228
  23:54:22 STEP 1403 H=1403:00 +CPU= 26.981
  23:55:22 STEP 1404 H=1404:00 +CPU= 47.447
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001404
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001404
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001404
  23:55:56 STEP 1405 H=1405:00 +CPU= 27.105
  23:56:32 STEP 1406 H=1406:00 +CPU= 28.243
  23:57:25 STEP 1407 H=1407:00 +CPU= 42.486
  23:57:59 STEP 1408 H=1408:00 +CPU= 27.865
  23:58:32 STEP 1409 H=1409:00 +CPU= 26.901
  23:59:27 STEP 1410 H=1410:00 +CPU= 43.904
  00:00:01 STEP 1411 H=1411:00 +CPU= 27.446
  00:00:37 STEP 1412 H=1412:00 +CPU= 28.510
  00:01:31 STEP 1413 H=1413:00 +CPU= 43.269
  00:02:08 STEP 1414 H=1414:00 +CPU= 28.597
  00:02:46 STEP 1415 H=1415:00 +CPU= 28.851
  00:03:51 STEP 1416 H=1416:00 +CPU= 51.655
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001416
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001416
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001416
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001392
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001392
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001392
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001404
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001404
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001404
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_58.zip
Uploading the intermediate file: upload_file_58.zip
  00:04:25 STEP 1417 H=1417:00 +CPU= 27.048
  00:05:00 STEP 1418 H=1418:00 +CPU= 28.061
  00:05:58 STEP 1419 H=1419:00 +CPU= 43.022
  00:06:43 STEP 1420 H=1420:00 +CPU= 29.035
  00:07:18 STEP 1421 H=1421:00 +CPU= 27.497
  00:08:15 STEP 1422 H=1422:00 +CPU= 44.108
  00:08:51 STEP 1423 H=1423:00 +CPU= 27.381
  00:09:27 STEP 1424 H=1424:00 +CPU= 28.638
  00:10:24 STEP 1425 H=1425:00 +CPU= 44.105
  00:11:00 STEP 1426 H=1426:00 +CPU= 28.649
  00:11:35 STEP 1427 H=1427:00 +CPU= 27.446
  00:12:35 STEP 1428 H=1428:00 +CPU= 48.321
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001428
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001428
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001428
  00:13:11 STEP 1429 H=1429:00 +CPU= 27.448
  00:13:47 STEP 1430 H=1430:00 +CPU= 28.730
  00:14:42 STEP 1431 H=1431:00 +CPU= 43.074
  00:15:18 STEP 1432 H=1432:00 +CPU= 28.319
  00:15:52 STEP 1433 H=1433:00 +CPU= 27.354
  00:16:47 STEP 1434 H=1434:00 +CPU= 44.043
  00:17:20 STEP 1435 H=1435:00 +CPU= 26.875
  00:17:54 STEP 1436 H=1436:00 +CPU= 27.855
  00:18:49 STEP 1437 H=1437:00 +CPU= 42.600
  00:19:31 STEP 1438 H=1438:00 +CPU= 28.344
  00:20:20 STEP 1439 H=1439:00 +CPU= 28.521
  00:21:28 STEP 1440 H=1440:00 +CPU= 50.540
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001440
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001440
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001440
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001416
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001416
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001416
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001428
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001428
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001428
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_59.zip
Uploading the intermediate file: upload_file_59.zip
  00:22:06 STEP 1441 H=1441:00 +CPU= 29.482
  00:22:46 STEP 1442 H=1442:00 +CPU= 31.348
  00:23:40 STEP 1443 H=1443:00 +CPU= 42.860
  00:24:15 STEP 1444 H=1444:00 +CPU= 27.972
  00:24:49 STEP 1445 H=1445:00 +CPU= 27.150
  00:25:44 STEP 1446 H=1446:00 +CPU= 44.354
  00:26:18 STEP 1447 H=1447:00 +CPU= 27.269
  00:26:55 STEP 1448 H=1448:00 +CPU= 28.657
  00:27:54 STEP 1449 H=1449:00 +CPU= 44.038
  00:28:31 STEP 1450 H=1450:00 +CPU= 28.679
  00:29:08 STEP 1451 H=1451:00 +CPU= 27.633
  00:30:11 STEP 1452 H=1452:00 +CPU= 49.579
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001452
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001452
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001452
  00:30:47 STEP 1453 H=1453:00 +CPU= 28.317
  00:31:22 STEP 1454 H=1454:00 +CPU= 27.992
  00:32:15 STEP 1455 H=1455:00 +CPU= 42.734
  00:32:52 STEP 1456 H=1456:00 +CPU= 28.425
  00:33:27 STEP 1457 H=1457:00 +CPU= 27.767
  00:34:32 STEP 1458 H=1458:00 +CPU= 44.379
  00:35:10 STEP 1459 H=1459:00 +CPU= 27.401
  00:35:47 STEP 1460 H=1460:00 +CPU= 28.392
  00:36:42 STEP 1461 H=1461:00 +CPU= 43.443
  00:37:16 STEP 1462 H=1462:00 +CPU= 27.343
  00:37:51 STEP 1463 H=1463:00 +CPU= 27.610
  00:38:58 STEP 1464 H=1464:00 +CPU= 51.213
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001464
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001464
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001464
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001440
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001440
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001440
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001452
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001452
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001452
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_60.zip
Uploading the intermediate file: upload_file_60.zip
  00:39:33 STEP 1465 H=1465:00 +CPU= 27.580
  00:40:08 STEP 1466 H=1466:00 +CPU= 28.076
  00:41:03 STEP 1467 H=1467:00 +CPU= 42.658
  00:41:39 STEP 1468 H=1468:00 +CPU= 27.987
  00:42:14 STEP 1469 H=1469:00 +CPU= 27.277
  00:43:10 STEP 1470 H=1470:00 +CPU= 44.353
  00:43:45 STEP 1471 H=1471:00 +CPU= 27.694
  00:44:20 STEP 1472 H=1472:00 +CPU= 28.285
  00:45:17 STEP 1473 H=1473:00 +CPU= 43.990
  00:45:54 STEP 1474 H=1474:00 +CPU= 29.229
  00:46:29 STEP 1475 H=1475:00 +CPU= 27.731
  00:47:29 STEP 1476 H=1476:00 +CPU= 48.198
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001476
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001476
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001476
  00:48:05 STEP 1477 H=1477:00 +CPU= 28.191
  00:48:43 STEP 1478 H=1478:00 +CPU= 29.143
  00:49:38 STEP 1479 H=1479:00 +CPU= 43.686
  00:50:14 STEP 1480 H=1480:00 +CPU= 28.748
  00:50:48 STEP 1481 H=1481:00 +CPU= 27.266
  00:51:46 STEP 1482 H=1482:00 +CPU= 44.067
  00:52:20 STEP 1483 H=1483:00 +CPU= 27.454
  00:52:56 STEP 1484 H=1484:00 +CPU= 28.454
  00:53:51 STEP 1485 H=1485:00 +CPU= 43.565
  00:54:25 STEP 1486 H=1486:00 +CPU= 27.733
  00:54:59 STEP 1487 H=1487:00 +CPU= 27.954
  00:56:01 STEP 1488 H=1488:00 +CPU= 50.089
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001488
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001488
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001488
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001464
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001464
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001464
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001476
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001476
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001476
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_61.zip
Uploading the intermediate file: upload_file_61.zip
  00:56:35 STEP 1489 H=1489:00 +CPU= 26.669
  00:57:11 STEP 1490 H=1490:00 +CPU= 28.520
  00:58:06 STEP 1491 H=1491:00 +CPU= 43.269
  00:58:41 STEP 1492 H=1492:00 +CPU= 28.194
  00:59:14 STEP 1493 H=1493:00 +CPU= 27.029
  01:00:09 STEP 1494 H=1494:00 +CPU= 44.113
  01:00:44 STEP 1495 H=1495:00 +CPU= 27.334
  01:01:22 STEP 1496 H=1496:00 +CPU= 30.064
  01:02:17 STEP 1497 H=1497:00 +CPU= 43.710
  01:02:51 STEP 1498 H=1498:00 +CPU= 27.951
  01:03:26 STEP 1499 H=1499:00 +CPU= 27.762
  01:04:25 STEP 1500 H=1500:00 +CPU= 47.561
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001500
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001500
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001500
  01:05:03 STEP 1501 H=1501:00 +CPU= 26.993
  01:05:47 STEP 1502 H=1502:00 +CPU= 28.184
  01:06:41 STEP 1503 H=1503:00 +CPU= 42.491
  01:07:32 STEP 1504 H=1504:00 +CPU= 28.643
  01:08:10 STEP 1505 H=1505:00 +CPU= 27.790
  01:09:06 STEP 1506 H=1506:00 +CPU= 44.422
  01:09:42 STEP 1507 H=1507:00 +CPU= 27.766
  01:10:18 STEP 1508 H=1508:00 +CPU= 28.809
  01:11:12 STEP 1509 H=1509:00 +CPU= 43.189
  01:11:48 STEP 1510 H=1510:00 +CPU= 28.781
  01:12:25 STEP 1511 H=1511:00 +CPU= 28.997
  01:13:28 STEP 1512 H=1512:00 +CPU= 50.693
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001512
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001512
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001512
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001488
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001488
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001488
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001500
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001500
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001500
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_62.zip
Uploading the intermediate file: upload_file_62.zip
  01:14:04 STEP 1513 H=1513:00 +CPU= 27.731
  01:14:40 STEP 1514 H=1514:00 +CPU= 28.784
  01:15:37 STEP 1515 H=1515:00 +CPU= 41.755
  01:16:14 STEP 1516 H=1516:00 +CPU= 28.596
  01:16:51 STEP 1517 H=1517:00 +CPU= 29.049
  01:17:55 STEP 1518 H=1518:00 +CPU= 49.689
  01:18:29 STEP 1519 H=1519:00 +CPU= 27.586
  01:19:05 STEP 1520 H=1520:00 +CPU= 28.671
  01:20:00 STEP 1521 H=1521:00 +CPU= 43.025
  01:20:38 STEP 1522 H=1522:00 +CPU= 28.488
  01:21:13 STEP 1523 H=1523:00 +CPU= 27.686
  01:22:15 STEP 1524 H=1524:00 +CPU= 48.172
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001524
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001524
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001524
  01:22:50 STEP 1525 H=1525:00 +CPU= 27.726
  01:23:25 STEP 1526 H=1526:00 +CPU= 28.306
  01:24:17 STEP 1527 H=1527:00 +CPU= 42.757
  01:24:52 STEP 1528 H=1528:00 +CPU= 28.171
  01:25:25 STEP 1529 H=1529:00 +CPU= 26.543
  01:26:15 STEP 1530 H=1530:00 +CPU= 40.987
  01:26:48 STEP 1531 H=1531:00 +CPU= 25.873
  01:27:41 STEP 1532 H=1532:00 +CPU= 27.601
  01:28:33 STEP 1533 H=1533:00 +CPU= 41.060
  01:29:05 STEP 1534 H=1534:00 +CPU= 26.543
  01:29:37 STEP 1535 H=1535:00 +CPU= 26.517
  01:30:39 STEP 1536 H=1536:00 +CPU= 48.804
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001536
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001536
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001536
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001512
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001512
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001512
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001524
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001524
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001524
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_63.zip
Uploading the intermediate file: upload_file_63.zip
Uploading trickle at timestep: 5529600
  01:31:13 STEP 1537 H=1537:00 +CPU= 26.746
  01:31:48 STEP 1538 H=1538:00 +CPU= 28.273
  01:32:46 STEP 1539 H=1539:00 +CPU= 43.473
  01:33:26 STEP 1540 H=1540:00 +CPU= 28.995
  01:34:02 STEP 1541 H=1541:00 +CPU= 28.051
  01:35:00 STEP 1542 H=1542:00 +CPU= 44.592
  01:35:40 STEP 1543 H=1543:00 +CPU= 27.838
  01:36:16 STEP 1544 H=1544:00 +CPU= 28.653
  01:37:09 STEP 1545 H=1545:00 +CPU= 43.115
  01:37:45 STEP 1546 H=1546:00 +CPU= 28.668
  01:38:21 STEP 1547 H=1547:00 +CPU= 28.224
  01:39:21 STEP 1548 H=1548:00 +CPU= 48.063
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001548
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001548
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001548
  01:39:55 STEP 1549 H=1549:00 +CPU= 27.511
  01:40:31 STEP 1550 H=1550:00 +CPU= 28.858
  01:41:26 STEP 1551 H=1551:00 +CPU= 43.647
  01:42:04 STEP 1552 H=1552:00 +CPU= 29.053
  01:42:41 STEP 1553 H=1553:00 +CPU= 27.999
  01:43:43 STEP 1554 H=1554:00 +CPU= 45.306
  01:44:17 STEP 1555 H=1555:00 +CPU= 27.203
  01:44:53 STEP 1556 H=1556:00 +CPU= 28.370
  01:45:51 STEP 1557 H=1557:00 +CPU= 43.644
  01:46:35 STEP 1558 H=1558:00 +CPU= 28.439
  01:47:14 STEP 1559 H=1559:00 +CPU= 28.350
  01:48:21 STEP 1560 H=1560:00 +CPU= 49.765
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001560
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001560
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001560
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001536
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001536
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001536
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001548
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001548
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001548
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_64.zip
Uploading the intermediate file: upload_file_64.zip
  01:49:05 STEP 1561 H=1561:00 +CPU= 25.972
  01:49:53 STEP 1562 H=1562:00 +CPU= 27.110
  01:51:09 STEP 1563 H=1563:00 +CPU= 41.854
  01:51:59 STEP 1564 H=1564:00 +CPU= 27.691
  01:52:44 STEP 1565 H=1565:00 +CPU= 26.149
  01:53:59 STEP 1566 H=1566:00 +CPU= 42.347
  01:54:45 STEP 1567 H=1567:00 +CPU= 27.295
  01:55:30 STEP 1568 H=1568:00 +CPU= 34.691
  01:56:30 STEP 1569 H=1569:00 +CPU= 44.801
  01:57:05 STEP 1570 H=1570:00 +CPU= 28.298
  01:57:39 STEP 1571 H=1571:00 +CPU= 26.970
  01:58:49 STEP 1572 H=1572:00 +CPU= 47.376
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001572
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001572
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001572
  01:59:24 STEP 1573 H=1573:00 +CPU= 27.691
  02:00:00 STEP 1574 H=1574:00 +CPU= 28.002
  02:00:56 STEP 1575 H=1575:00 +CPU= 42.680
  02:01:30 STEP 1576 H=1576:00 +CPU= 28.120
  02:02:04 STEP 1577 H=1577:00 +CPU= 27.049
  02:03:00 STEP 1578 H=1578:00 +CPU= 44.162
  02:03:33 STEP 1579 H=1579:00 +CPU= 27.211
  02:04:09 STEP 1580 H=1580:00 +CPU= 27.933
  02:05:04 STEP 1581 H=1581:00 +CPU= 43.056
  02:05:40 STEP 1582 H=1582:00 +CPU= 28.420
  02:06:18 STEP 1583 H=1583:00 +CPU= 28.984
  02:07:43 STEP 1584 H=1584:00 +CPU= 51.734
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001584
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001584
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001584
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001560
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001560
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001560
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001572
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001572
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001572
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_65.zip
Uploading the intermediate file: upload_file_65.zip
  02:08:20 STEP 1585 H=1585:00 +CPU= 26.943
  02:08:55 STEP 1586 H=1586:00 +CPU= 27.865
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0878
(argv4) batchid: 989
(argv5) wuid: 12206522
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/dan/BOINC/slots/0
Project directory is: /home/dan/BOINC/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522
..mkdir for temp folder for results failed
Copying: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/dan/BOINC/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/dan/BOINC/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ccaef69f7085177ec9c5d74221da607c to: /home/dan/BOINC/slots/0/oifs_43r3_ps_0878_2020050100_123_989_12206522.zip
Unzipping the namelist zip file: /home/dan/BOINC/slots/0/oifs_43r3_ps_0878_2020050100_123_989_12206522.zip
ic_ancil_file: ic_ancil_12206522
ifsdata_file: ifsdata_12206522
climate_data_file: clim_data_12206522
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1cdafa83e982a6f6ec0bb2b89cd3bbd6 to: /home/dan/BOINC/slots/0/ic_ancil_12206522.zip
Unzipping the IC ancils zip file: /home/dan/BOINC/slots/0/ic_ancil_12206522.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ee3163b6468ddd19ed08128da1343ba6 to: /home/dan/BOINC/slots/0/ifsdata/ifsdata_12206522.zip
Unzipping the ifsdata_zip file: /home/dan/BOINC/slots/0/ifsdata/ifsdata_12206522.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/dan/BOINC/slots/0/159l_2/clim_data_12206522.zip
Unzipping the climate data zip file: /home/dan/BOINC/slots/0/159l_2/clim_data_12206522.zip
Checking for progress XML file: /home/dan/BOINC/slots/0/progress_file_12206522.xml
Opened progress file ok : /home/dan/BOINC/slots/0/progress_file_12206522.xml
-- Model is restarting --
Adjusting last_iter, 1587, to previous model restart step.
Creating progress file: /home/dan/BOINC/slots/0/progress_file_12206522.xml
last_cpu_time: 52222
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524
The child process has been launched with process id: 13142
Executing the command: /home/dan/BOINC/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:13142:13142] [20230118:163037:1674055837:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:31:57 STEP 1584 H=1584:00 +CPU= 48.425
  16:32:29 STEP 1585 H=1585:00 +CPU= 26.133
  16:33:06 STEP 1586 H=1586:00 +CPU= 26.896
  16:34:03 STEP 1587 H=1587:00 +CPU= 40.871
  16:34:37 STEP 1588 H=1588:00 +CPU= 27.100
  16:35:09 STEP 1589 H=1589:00 +CPU= 26.257
  16:36:01 STEP 1590 H=1590:00 +CPU= 42.475
  16:36:33 STEP 1591 H=1591:00 +CPU= 26.267
  16:37:08 STEP 1592 H=1592:00 +CPU= 27.073
  16:38:00 STEP 1593 H=1593:00 +CPU= 41.095
  16:38:34 STEP 1594 H=1594:00 +CPU= 27.288
  16:39:06 STEP 1595 H=1595:00 +CPU= 26.247
  16:40:02 STEP 1596 H=1596:00 +CPU= 45.829
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001596
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001596
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001596
  16:40:33 STEP 1597 H=1597:00 +CPU= 25.707
  16:41:07 STEP 1598 H=1598:00 +CPU= 26.829
  16:41:58 STEP 1599 H=1599:00 +CPU= 41.024
  16:42:32 STEP 1600 H=1600:00 +CPU= 27.410
  16:43:04 STEP 1601 H=1601:00 +CPU= 26.177
  16:43:56 STEP 1602 H=1602:00 +CPU= 42.301
  16:44:30 STEP 1603 H=1603:00 +CPU= 26.540
  16:45:03 STEP 1604 H=1604:00 +CPU= 27.180
  16:45:54 STEP 1605 H=1605:00 +CPU= 41.092
  16:46:27 STEP 1606 H=1606:00 +CPU= 27.053
  16:47:00 STEP 1607 H=1607:00 +CPU= 27.052
  16:48:02 STEP 1608 H=1608:00 +CPU= 48.590
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001608
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001608
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001608
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001584
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001584
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001584
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001596
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001596
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001596
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_66.zip
Uploading the intermediate file: upload_file_66.zip
  16:48:41 STEP 1609 H=1609:00 +CPU= 26.147
  16:49:15 STEP 1610 H=1610:00 +CPU= 26.853
  16:50:06 STEP 1611 H=1611:00 +CPU= 41.196
  16:50:40 STEP 1612 H=1612:00 +CPU= 27.367
  16:51:18 STEP 1613 H=1613:00 +CPU= 30.288
  16:52:15 STEP 1614 H=1614:00 +CPU= 45.395
  16:52:47 STEP 1615 H=1615:00 +CPU= 26.153
  16:53:21 STEP 1616 H=1616:00 +CPU= 27.441
  16:54:12 STEP 1617 H=1617:00 +CPU= 40.823
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0878
(argv4) batchid: 989
(argv5) wuid: 12206522
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/dan/BOINC/slots/0
Project directory is: /home/dan/BOINC/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522
..mkdir for temp folder for results failed
Copying: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/dan/BOINC/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/dan/BOINC/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ccaef69f7085177ec9c5d74221da607c to: /home/dan/BOINC/slots/0/oifs_43r3_ps_0878_2020050100_123_989_12206522.zip
Unzipping the namelist zip file: /home/dan/BOINC/slots/0/oifs_43r3_ps_0878_2020050100_123_989_12206522.zip
ic_ancil_file: ic_ancil_12206522
ifsdata_file: ifsdata_12206522
climate_data_file: clim_data_12206522
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1cdafa83e982a6f6ec0bb2b89cd3bbd6 to: /home/dan/BOINC/slots/0/ic_ancil_12206522.zip
Unzipping the IC ancils zip file: /home/dan/BOINC/slots/0/ic_ancil_12206522.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ee3163b6468ddd19ed08128da1343ba6 to: /home/dan/BOINC/slots/0/ifsdata/ifsdata_12206522.zip
Unzipping the ifsdata_zip file: /home/dan/BOINC/slots/0/ifsdata/ifsdata_12206522.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/dan/BOINC/slots/0/159l_2/clim_data_12206522.zip
Unzipping the climate data zip file: /home/dan/BOINC/slots/0/159l_2/clim_data_12206522.zip
Checking for progress XML file: /home/dan/BOINC/slots/0/progress_file_12206522.xml
Opened progress file ok : /home/dan/BOINC/slots/0/progress_file_12206522.xml
-- Model is restarting --
Adjusting last_iter, 1618, to previous model restart step.
Creating progress file: /home/dan/BOINC/slots/0/progress_file_12206522.xml
last_cpu_time: 53355
upload_file_number: 67
last_iter: 1607
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524
The child process has been launched with process id: 19947
Executing the command: /home/dan/BOINC/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rigel:1:1:19947:19947] [20230118:205540:1674071740:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:57:04 STEP 1608 H=1608:00 +CPU= 48.538
  20:57:37 STEP 1609 H=1609:00 +CPU= 26.575
  20:58:10 STEP 1610 H=1610:00 +CPU= 27.423
  20:58:59 STEP 1611 H=1611:00 +CPU= 40.853
  20:59:32 STEP 1612 H=1612:00 +CPU= 27.104
  21:00:04 STEP 1613 H=1613:00 +CPU= 26.401
  21:00:56 STEP 1614 H=1614:00 +CPU= 42.272
  21:01:28 STEP 1615 H=1615:00 +CPU= 26.465
  21:02:01 STEP 1616 H=1616:00 +CPU= 27.249
  21:03:06 STEP 1617 H=1617:00 +CPU= 41.481
  21:03:44 STEP 1618 H=1618:00 +CPU= 27.739
  21:04:17 STEP 1619 H=1619:00 +CPU= 26.681
  21:05:21 STEP 1620 H=1620:00 +CPU= 46.000
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001620
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001620
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001620
  21:05:58 STEP 1621 H=1621:00 +CPU= 27.028
  21:06:34 STEP 1622 H=1622:00 +CPU= 27.849
  21:07:25 STEP 1623 H=1623:00 +CPU= 41.814
  21:07:57 STEP 1624 H=1624:00 +CPU= 26.724
  21:08:28 STEP 1625 H=1625:00 +CPU= 25.197
  21:09:17 STEP 1626 H=1626:00 +CPU= 40.215
  21:09:49 STEP 1627 H=1627:00 +CPU= 24.744
  21:10:22 STEP 1628 H=1628:00 +CPU= 25.777
  21:11:14 STEP 1629 H=1629:00 +CPU= 39.324
  21:11:48 STEP 1630 H=1630:00 +CPU= 25.781
  21:12:24 STEP 1631 H=1631:00 +CPU= 26.069
  21:13:36 STEP 1632 H=1632:00 +CPU= 46.714
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001632
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001632
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001632
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001608
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001608
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001608
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMGGhq0f+001620
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMSHhq0f+001620
Adding to the zip: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_12206522/ICMUAhq0f+001620
Zipping up the intermediate file: /home/dan/BOINC/projects/climateprediction.net/oifs_43r3_ps_0878_2020050100_123_989_12206522_1_r377736524_67.zip
Uploading the intermediate file: upload_file_67.zip
  21:14:06 STEP 1633 H=1633:00 +CPU= 24.938
  21:14:39 STEP 1634 H=1634:00 +CPU= 26.057
  21:15:28 STEP 1635 H=1635:00 +CPU= 39.661
  21:15:59 STEP 1636 H=1636:00 +CPU= 26.030
  21:16:29 STEP 1637 H=1637:00 +CPU= 24.832
  21:17:17 STEP 1638 H=1638:00 +CPU= 39.796
  21:17:47 STEP 1639 H=1639:00 +CPU= 24.831
  21:18:19 STEP 1640 H=1640:00 +CPU= 25.482
  21:19:10 STEP 1641 H=1641:00 +CPU= 40.039
  21:19:42 STEP 1642 H=1642:00 +CPU= 26.142
  21:20:12 STEP 1643 H=1643:00 +CPU= 24.980
  21:21:17 STEP 1644 H=1644:00 +CPU= 51.438
Moving to projects directory: /home/dan/BOINC/slots/0/ICMGGhq0f+001644
Moving to projects directory: /home/dan/BOINC/slots/0/ICMSHhq0f+001644
Moving to projects directory: /home/dan/BOINC/slots/0/ICMUAhq0f+001644
  21:21:48 STEP 1645 H=1645:00 +CPU= 25.301
  21:22:19 STEP 1646 H=1646:00 +CPU= 25.701
  21:23:08 STEP 1647 H=1647:00 +CPU= 39.306
  21:23:39 STEP 1648 H=1648:00 +CPU= 25.851
  21:24:08 STEP 1649 H=1649:00 +CPU= 24.544
  21:24:59 STEP 1650 H=1650:00 +CPU= 40.554
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:25:11 1516274 22293783 oifs_43r3_ps_0878_2020050100_123_989_12206522_1 5,529,600 50,520 0.0091
09 May 2023 16:24:52 1516274 22293783 oifs_43r3_ps_0878_2020050100_123_989_12206522_1 4,579,200 41,346 0.0090
09 May 2023 16:24:38 1516274 22293783 oifs_43r3_ps_0878_2020050100_123_989_12206522_1 3,715,200 33,978 0.0091
09 May 2023 16:24:24 1516274 22293783 oifs_43r3_ps_0878_2020050100_123_989_12206522_1 2,851,200 26,297 0.0092
09 May 2023 16:22:43 1516274 22293783 oifs_43r3_ps_0878_2020050100_123_989_12206522_1 864,000 7,270 0.0084


©2024 cpdn.org