<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_960d830ed1079cdda12516796a75c269 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0088_2009050100_123_978_12194732.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0088_2009050100_123_978_12194732.zip
ic_ancil_file: ic_ancil_12194732
ifsdata_file: ifsdata_12194732
climate_data_file: clim_data_12194732
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9e4885bb6912a692627d53e72648dac to: /var/lib/boinc-client/slots/3/ic_ancil_12194732.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12194732.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_37671b8b8a934fc17a0052f4686ae980 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12194732.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12194732.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12194732.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12194732.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12194732.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12194732.xml
-- Model is restarting --
Adjusting last_iter, 354, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12194732.xml
last_cpu_time: 18630
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611
The child process has been launched with process id: 19533
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:19533:19533] [20230114:023659:1673660219:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
02:38:14 STEP 336 H= 336:00 +CPU= 50.921
02:38:46 STEP 337 H= 337:00 +CPU= 31.727
02:39:18 STEP 338 H= 338:00 +CPU= 31.381
02:40:07 STEP 339 H= 339:00 +CPU= 48.735
02:40:35 STEP 340 H= 340:00 +CPU= 27.968
02:41:05 STEP 341 H= 341:00 +CPU= 29.691
02:41:56 STEP 342 H= 342:00 +CPU= 50.141
02:42:27 STEP 343 H= 343:00 +CPU= 31.022
02:43:01 STEP 344 H= 344:00 +CPU= 32.728
02:43:50 STEP 345 H= 345:00 +CPU= 49.425
02:44:23 STEP 346 H= 346:00 +CPU= 32.390
02:44:48 STEP 347 H= 347:00 +CPU= 25.270
02:45:45 STEP 348 H= 348:00 +CPU= 54.476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000348
02:46:11 STEP 349 H= 349:00 +CPU= 25.257
02:46:43 STEP 350 H= 350:00 +CPU= 31.424
02:47:33 STEP 351 H= 351:00 +CPU= 49.646
02:48:09 STEP 352 H= 352:00 +CPU= 35.109
02:48:39 STEP 353 H= 353:00 +CPU= 29.360
02:49:22 STEP 354 H= 354:00 +CPU= 43.168
02:49:54 STEP 355 H= 355:00 +CPU= 31.225
02:50:23 STEP 356 H= 356:00 +CPU= 29.264
02:51:12 STEP 357 H= 357:00 +CPU= 48.971
02:51:45 STEP 358 H= 358:00 +CPU= 32.115
02:52:15 STEP 359 H= 359:00 +CPU= 29.989
02:53:14 STEP 360 H= 360:00 +CPU= 55.916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611_14.zip
Uploading the intermediate file: upload_file_14.zip
02:53:46 STEP 361 H= 361:00 +CPU= 31.271
02:54:18 STEP 362 H= 362:00 +CPU= 32.065
02:55:07 STEP 363 H= 363:00 +CPU= 49.138
02:55:39 STEP 364 H= 364:00 +CPU= 31.594
02:56:06 STEP 365 H= 365:00 +CPU= 26.844
02:56:58 STEP 366 H= 366:00 +CPU= 50.012
02:57:29 STEP 367 H= 367:00 +CPU= 31.073
02:58:02 STEP 368 H= 368:00 +CPU= 32.598
02:58:49 STEP 369 H= 369:00 +CPU= 46.178
02:59:19 STEP 370 H= 370:00 +CPU= 29.957
02:59:52 STEP 371 H= 371:00 +CPU= 31.423
03:00:43 STEP 372 H= 372:00 +CPU= 50.607
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000372
03:01:15 STEP 373 H= 373:00 +CPU= 31.426
03:01:42 STEP 374 H= 374:00 +CPU= 26.719
03:02:26 STEP 375 H= 375:00 +CPU= 44.064
03:03:01 STEP 376 H= 376:00 +CPU= 32.871
03:03:29 STEP 377 H= 377:00 +CPU= 28.446
03:04:19 STEP 378 H= 378:00 +CPU= 48.620
03:04:49 STEP 379 H= 379:00 +CPU= 30.525
03:05:22 STEP 380 H= 380:00 +CPU= 32.704
03:06:06 STEP 381 H= 381:00 +CPU= 42.929
03:06:37 STEP 382 H= 382:00 +CPU= 31.207
03:07:11 STEP 383 H= 383:00 +CPU= 32.973
03:08:06 STEP 384 H= 384:00 +CPU= 53.155
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611_15.zip
Uploading the intermediate file: upload_file_15.zip
03:08:37 STEP 385 H= 385:00 +CPU= 30.957
03:09:08 STEP 386 H= 386:00 +CPU= 31.121
03:09:58 STEP 387 H= 387:00 +CPU= 49.283
03:10:30 STEP 388 H= 388:00 +CPU= 31.826
03:10:58 STEP 389 H= 389:00 +CPU= 26.519
03:11:47 STEP 390 H= 390:00 +CPU= 49.298
03:12:20 STEP 391 H= 391:00 +CPU= 32.044
03:12:47 STEP 392 H= 392:00 +CPU= 27.457
03:13:38 STEP 393 H= 393:00 +CPU= 49.889
03:14:11 STEP 394 H= 394:00 +CPU= 32.870
03:14:41 STEP 395 H= 395:00 +CPU= 29.724
03:15:34 STEP 396 H= 396:00 +CPU= 52.010
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000396
03:16:04 STEP 397 H= 397:00 +CPU= 29.147
03:16:34 STEP 398 H= 398:00 +CPU= 29.917
03:17:24 STEP 399 H= 399:00 +CPU= 49.529
03:17:56 STEP 400 H= 400:00 +CPU= 31.824
03:18:22 STEP 401 H= 401:00 +CPU= 26.270
03:19:10 STEP 402 H= 402:00 +CPU= 47.490
03:19:44 STEP 403 H= 403:00 +CPU= 31.702
03:20:37 STEP 404 H= 404:00 +CPU= 31.580
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0088
(argv4) batchid: 978
(argv5) wuid: 12194732
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_960d830ed1079cdda12516796a75c269 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0088_2009050100_123_978_12194732.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0088_2009050100_123_978_12194732.zip
ic_ancil_file: ic_ancil_12194732
ifsdata_file: ifsdata_12194732
climate_data_file: clim_data_12194732
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9e4885bb6912a692627d53e72648dac to: /var/lib/boinc-client/slots/3/ic_ancil_12194732.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12194732.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_37671b8b8a934fc17a0052f4686ae980 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12194732.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12194732.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12194732.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12194732.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12194732.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12194732.xml
-- Model is restarting --
Adjusting last_iter, 405, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12194732.xml
last_cpu_time: 21211
upload_file_number: 16
last_iter: 383
last_upload: 1382400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611
The child process has been launched with process id: 20908
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:20908:20908] [20230114:041708:1673666228:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
04:18:44 STEP 384 H= 384:00 +CPU= 47.093
04:19:13 STEP 385 H= 385:00 +CPU= 29.334
04:19:45 STEP 386 H= 386:00 +CPU= 31.244
04:20:24 STEP 387 H= 387:00 +CPU= 39.320
04:20:58 STEP 388 H= 388:00 +CPU= 32.986
04:21:27 STEP 389 H= 389:00 +CPU= 28.858
04:22:17 STEP 390 H= 390:00 +CPU= 49.347
04:22:47 STEP 391 H= 391:00 +CPU= 30.441
04:23:19 STEP 392 H= 392:00 +CPU= 31.883
04:24:08 STEP 393 H= 393:00 +CPU= 48.514
04:24:41 STEP 394 H= 394:00 +CPU= 32.314
04:25:12 STEP 395 H= 395:00 +CPU= 30.951
04:26:07 STEP 396 H= 396:00 +CPU= 53.830
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000396
04:26:38 STEP 397 H= 397:00 +CPU= 30.880
04:27:10 STEP 398 H= 398:00 +CPU= 31.595
04:27:54 STEP 399 H= 399:00 +CPU= 43.041
04:28:20 STEP 400 H= 400:00 +CPU= 25.372
04:28:45 STEP 401 H= 401:00 +CPU= 24.658
04:29:36 STEP 402 H= 402:00 +CPU= 51.410
04:30:07 STEP 403 H= 403:00 +CPU= 30.460
04:30:40 STEP 404 H= 404:00 +CPU= 32.356
04:31:25 STEP 405 H= 405:00 +CPU= 44.753
04:31:57 STEP 406 H= 406:00 +CPU= 32.005
04:32:26 STEP 407 H= 407:00 +CPU= 28.574
04:33:20 STEP 408 H= 408:00 +CPU= 52.801
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611_16.zip
Uploading the intermediate file: upload_file_16.zip
04:33:51 STEP 409 H= 409:00 +CPU= 30.359
04:34:23 STEP 410 H= 410:00 +CPU= 32.001
04:35:12 STEP 411 H= 411:00 +CPU= 48.926
04:35:45 STEP 412 H= 412:00 +CPU= 32.196
04:36:16 STEP 413 H= 413:00 +CPU= 30.879
04:37:04 STEP 414 H= 414:00 +CPU= 48.158
04:37:33 STEP 415 H= 415:00 +CPU= 28.313
04:38:05 STEP 416 H= 416:00 +CPU= 31.562
04:38:54 STEP 417 H= 417:00 +CPU= 49.064
04:39:26 STEP 418 H= 418:00 +CPU= 31.721
04:39:58 STEP 419 H= 419:00 +CPU= 30.882
04:40:52 STEP 420 H= 420:00 +CPU= 53.943
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000420
04:41:23 STEP 421 H= 421:00 +CPU= 30.912
04:41:57 STEP 422 H= 422:00 +CPU= 33.099
04:42:45 STEP 423 H= 423:00 +CPU= 47.214
04:43:17 STEP 424 H= 424:00 +CPU= 31.731
04:43:48 STEP 425 H= 425:00 +CPU= 31.060
04:44:39 STEP 426 H= 426:00 +CPU= 50.163
04:45:10 STEP 427 H= 427:00 +CPU= 30.679
04:45:42 STEP 428 H= 428:00 +CPU= 31.723
04:46:25 STEP 429 H= 429:00 +CPU= 43.339
04:46:54 STEP 430 H= 430:00 +CPU= 28.850
04:47:29 STEP 431 H= 431:00 +CPU= 32.677
04:48:34 STEP 432 H= 432:00 +CPU= 56.811
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611_17.zip
Uploading the intermediate file: upload_file_17.zip
04:49:06 STEP 433 H= 433:00 +CPU= 31.027
04:49:38 STEP 434 H= 434:00 +CPU= 31.520
04:50:23 STEP 435 H= 435:00 +CPU= 45.115
04:50:57 STEP 436 H= 436:00 +CPU= 32.537
04:51:28 STEP 437 H= 437:00 +CPU= 30.747
04:52:18 STEP 438 H= 438:00 +CPU= 50.260
04:52:49 STEP 439 H= 439:00 +CPU= 31.105
04:53:22 STEP 440 H= 440:00 +CPU= 32.051
04:54:11 STEP 441 H= 441:00 +CPU= 48.846
04:54:43 STEP 442 H= 442:00 +CPU= 32.058
04:55:15 STEP 443 H= 443:00 +CPU= 30.921
04:56:10 STEP 444 H= 444:00 +CPU= 54.593
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000444
04:56:41 STEP 445 H= 445:00 +CPU= 30.576
04:57:14 STEP 446 H= 446:00 +CPU= 33.071
04:58:04 STEP 447 H= 447:00 +CPU= 49.645
04:58:36 STEP 448 H= 448:00 +CPU= 31.541
04:59:01 STEP 449 H= 449:00 +CPU= 25.363
04:59:53 STEP 450 H= 450:00 +CPU= 50.368
05:00:26 STEP 451 H= 451:00 +CPU= 32.623
05:00:57 STEP 452 H= 452:00 +CPU= 30.262
05:01:44 STEP 453 H= 453:00 +CPU= 46.828
05:02:18 STEP 454 H= 454:00 +CPU= 33.374
05:02:50 STEP 455 H= 455:00 +CPU= 31.717
05:03:44 STEP 456 H= 456:00 +CPU= 48.931
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611_18.zip
Uploading the intermediate file: upload_file_18.zip
05:04:15 STEP 457 H= 457:00 +CPU= 30.824
05:04:52 STEP 458 H= 458:00 +CPU= 36.472
05:05:45 STEP 459 H= 459:00 +CPU= 51.889
05:06:22 STEP 460 H= 460:00 +CPU= 35.689
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0088
(argv4) batchid: 978
(argv5) wuid: 12194732
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_960d830ed1079cdda12516796a75c269 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0088_2009050100_123_978_12194732.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0088_2009050100_123_978_12194732.zip
ic_ancil_file: ic_ancil_12194732
ifsdata_file: ifsdata_12194732
climate_data_file: clim_data_12194732
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9e4885bb6912a692627d53e72648dac to: /var/lib/boinc-client/slots/3/ic_ancil_12194732.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12194732.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_37671b8b8a934fc17a0052f4686ae980 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12194732.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12194732.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12194732.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12194732.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12194732.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12194732.xml
-- Model is restarting --
Adjusting last_iter, 461, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12194732.xml
last_cpu_time: 24106
upload_file_number: 19
last_iter: 455
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611
The child process has been launched with process id: 21894
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:21894:21894] [20230114:052321:1673670201:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
05:25:19 STEP 456 H= 456:00 +CPU= 56.778
05:25:52 STEP 457 H= 457:00 +CPU= 32.439
05:26:24 STEP 458 H= 458:00 +CPU= 32.000
05:27:14 STEP 459 H= 459:00 +CPU= 49.449
05:27:47 STEP 460 H= 460:00 +CPU= 32.575
05:28:18 STEP 461 H= 461:00 +CPU= 30.995
05:29:09 STEP 462 H= 462:00 +CPU= 50.024
05:29:40 STEP 463 H= 463:00 +CPU= 31.008
05:30:12 STEP 464 H= 464:00 +CPU= 32.151
05:30:57 STEP 465 H= 465:00 +CPU= 44.512
05:31:29 STEP 466 H= 466:00 +CPU= 31.923
05:32:01 STEP 467 H= 467:00 +CPU= 31.441
05:32:56 STEP 468 H= 468:00 +CPU= 54.270
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000468
05:33:28 STEP 469 H= 469:00 +CPU= 31.864
05:34:00 STEP 470 H= 470:00 +CPU= 31.463
05:34:49 STEP 471 H= 471:00 +CPU= 49.237
05:35:22 STEP 472 H= 472:00 +CPU= 32.263
05:35:49 STEP 473 H= 473:00 +CPU= 26.876
05:36:39 STEP 474 H= 474:00 +CPU= 50.106
05:37:11 STEP 475 H= 475:00 +CPU= 31.467
05:37:43 STEP 476 H= 476:00 +CPU= 31.890
05:38:32 STEP 477 H= 477:00 +CPU= 48.676
05:39:04 STEP 478 H= 478:00 +CPU= 31.793
05:39:34 STEP 479 H= 479:00 +CPU= 26.447
05:40:39 STEP 480 H= 480:00 +CPU= 55.301
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611_19.zip
Uploading the intermediate file: upload_file_19.zip
05:41:10 STEP 481 H= 481:00 +CPU= 30.359
05:41:40 STEP 482 H= 482:00 +CPU= 29.841
05:42:24 STEP 483 H= 483:00 +CPU= 44.555
05:42:56 STEP 484 H= 484:00 +CPU= 30.895
05:43:22 STEP 485 H= 485:00 +CPU= 26.618
05:44:12 STEP 486 H= 486:00 +CPU= 49.787
05:44:43 STEP 487 H= 487:00 +CPU= 30.325
05:45:14 STEP 488 H= 488:00 +CPU= 31.331
05:46:04 STEP 489 H= 489:00 +CPU= 48.997
05:46:35 STEP 490 H= 490:00 +CPU= 31.406
05:47:08 STEP 491 H= 491:00 +CPU= 31.647
05:47:58 STEP 492 H= 492:00 +CPU= 48.868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000492
05:48:29 STEP 493 H= 493:00 +CPU= 31.627
05:49:09 STEP 494 H= 494:00 +CPU= 34.905
05:49:58 STEP 495 H= 495:00 +CPU= 49.286
05:50:23 STEP 496 H= 496:00 +CPU= 24.948
05:50:54 STEP 497 H= 497:00 +CPU= 30.919
05:51:44 STEP 498 H= 498:00 +CPU= 49.765
05:52:15 STEP 499 H= 499:00 +CPU= 30.603
05:52:47 STEP 500 H= 500:00 +CPU= 31.437
05:53:31 STEP 501 H= 501:00 +CPU= 44.395
05:54:03 STEP 502 H= 502:00 +CPU= 30.792
05:54:35 STEP 503 H= 503:00 +CPU= 31.263
05:55:38 STEP 504 H= 504:00 +CPU= 57.543
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000492
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611_20.zip
Uploading the intermediate file: upload_file_20.zip
05:56:06 STEP 505 H= 505:00 +CPU= 27.361
05:56:38 STEP 506 H= 506:00 +CPU= 31.863
05:57:27 STEP 507 H= 507:00 +CPU= 49.557
05:58:00 STEP 508 H= 508:00 +CPU= 32.344
05:58:31 STEP 509 H= 509:00 +CPU= 30.633
05:59:19 STEP 510 H= 510:00 +CPU= 47.459
05:59:51 STEP 511 H= 511:00 +CPU= 31.606
06:00:27 STEP 512 H= 512:00 +CPU= 31.768
06:01:07 STEP 513 H= 513:00 +CPU= 39.690
06:01:33 STEP 514 H= 514:00 +CPU= 26.018
06:02:02 STEP 515 H= 515:00 +CPU= 28.768
06:02:57 STEP 516 H= 516:00 +CPU= 54.208
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000516
06:03:27 STEP 517 H= 517:00 +CPU= 30.208
06:04:01 STEP 518 H= 518:00 +CPU= 32.799
06:04:49 STEP 519 H= 519:00 +CPU= 48.485
06:05:17 STEP 520 H= 520:00 +CPU= 26.989
06:05:47 STEP 521 H= 521:00 +CPU= 29.588
06:06:26 STEP 522 H= 522:00 +CPU= 39.204
06:06:55 STEP 523 H= 523:00 +CPU= 28.760
06:07:26 STEP 524 H= 524:00 +CPU= 31.144
06:08:15 STEP 525 H= 525:00 +CPU= 48.633
06:08:47 STEP 526 H= 526:00 +CPU= 31.890
06:09:20 STEP 527 H= 527:00 +CPU= 31.633
06:10:22 STEP 528 H= 528:00 +CPU= 54.146
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMGGhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMSHhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732/ICMUAhq0f+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611_21.zip
Uploading the intermediate file: upload_file_21.zip
06:10:53 STEP 529 H= 529:00 +CPU= 31.097
06:11:26 STEP 530 H= 530:00 +CPU= 32.206
06:12:15 STEP 531 H= 531:00 +CPU= 49.145
06:12:46 STEP 532 H= 532:00 +CPU= 30.947
06:13:15 STEP 533 H= 533:00 +CPU= 28.125
06:14:06 STEP 534 H= 534:00 +CPU= 50.135
06:14:36 STEP 535 H= 535:00 +CPU= 30.665
06:15:03 STEP 536 H= 536:00 +CPU= 26.951
06:15:51 STEP 537 H= 537:00 +CPU= 47.378
06:16:23 STEP 538 H= 538:00 +CPU= 31.317
06:16:54 STEP 539 H= 539:00 +CPU= 30.924
06:17:41 STEP 540 H= 540:00 +CPU= 47.279
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000540
06:18:13 STEP 541 H= 541:00 +CPU= 26.025
06:18:45 STEP 542 H= 542:00 +CPU= 31.574
06:19:34 STEP 543 H= 543:00 +CPU= 49.215
06:20:05 STEP 544 H= 544:00 +CPU= 28.983
06:20:30 STEP 545 H= 545:00 +CPU= 25.289
06:21:19 STEP 546 H= 546:00 +CPU= 48.923
06:21:50 STEP 547 H= 547:00 +CPU= 30.310
06:22:22 STEP 548 H= 548:00 +CPU= 31.791
06:23:11 STEP 549 H= 549:00 +CPU= 49.091
06:23:44 STEP 550 H= 550:00 +CPU= 32.598
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0088
(argv4) batchid: 978
(argv5) wuid: 12194732
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194732
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_960d830ed1079cdda12516796a75c269 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0088_2009050100_123_978_12194732.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0088_2009050100_123_978_12194732.zip
ic_ancil_file: ic_ancil_12194732
ifsdata_file: ifsdata_12194732
climate_data_file: clim_data_12194732
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9e4885bb6912a692627d53e72648dac to: /var/lib/boinc-client/slots/3/ic_ancil_12194732.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12194732.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_37671b8b8a934fc17a0052f4686ae980 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12194732.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12194732.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12194732.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12194732.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12194732.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12194732.xml
-- Model is restarting --
Adjusting last_iter, 551, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12194732.xml
last_cpu_time: 27622
upload_file_number: 22
last_iter: 527
last_upload: 1900800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0088_2009050100_123_978_12194732_0_r1180374611
The child process has been launched with process id: 22930
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:22930:22930] [20230114:062540:1673673940:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
06:27:19 STEP 528 H= 528:00 +CPU= 56.105
06:27:50 STEP 529 H= 529:00 +CPU= 31.127
06:28:16 STEP 530 H= 530:00 +CPU= 26.153
06:29:05 STEP 531 H= 531:00 +CPU= 48.126
06:29:31 STEP 532 H= 532:00 +CPU= 26.029
06:29:59 STEP 533 H= 533:00 +CPU= 28.249
06:30:46 STEP 534 H= 534:00 +CPU= 46.221
06:31:15 STEP 535 H= 535:00 +CPU= 29.318
06:31:47 STEP 536 H= 536:00 +CPU= 31.397
06:32:32 STEP 537 H= 537:00 +CPU= 45.205
06:33:04 STEP 538 H= 538:00 +CPU= 31.796
06:33:35 STEP 539 H= 539:00 +CPU= 30.710
06:34:29 STEP 540 H= 540:00 +CPU= 48.219
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000540
06:34:59 STEP 541 H= 541:00 +CPU= 30.505
06:35:31 STEP 542 H= 542:00 +CPU= 31.318
06:36:16 STEP 543 H= 543:00 +CPU= 44.454
06:36:47 STEP 544 H= 544:00 +CPU= 31.269
06:37:18 STEP 545 H= 545:00 +CPU= 31.142
06:38:06 STEP 546 H= 546:00 +CPU= 47.248
06:38:36 STEP 547 H= 547:00 +CPU= 29.763
06:39:03 STEP 548 H= 548:00 +CPU= 26.353
06:39:53 STEP 549 H= 549:00 +CPU= 49.676
06:40:24 STEP 550 H= 550:00 +CPU= 31.248
06:40:57 STEP 551 H= 551:00 +CPU= 32.535
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
