climateprediction.net home page
Task 22271313

Task 22271313

Name oifs_43r3_ps_0318_2000050100_123_969_12185962_0
Workunit 12185962
Created 22 Dec 2022, 21:08:01 UTC
Sent 22 Dec 2022, 21:22:49 UTC
Report deadline 21 Jan 2023, 21:22:49 UTC
Received 28 Dec 2022, 17:42:45 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1488917
Run time 5 days 18 hours 11 min 16 sec
CPU time 23 hours 20 min 37 sec
Validate state Invalid
Credit 6,531.84
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,408.54 MB
Peak swap size 4,978.06 MB
Peak disk usage 2,652.53 MB
Stderr 
<core_client_version>7.16.17</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
CPU= 46.860
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0318
(argv4) batchid: 969
(argv5) wuid: 12185962
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/31
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/31/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/31/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ae305a224e06437cb560df237e326c7c to: /var/lib/boinc/slots/31/oifs_43r3_ps_0318_2000050100_123_969_12185962.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/31/oifs_43r3_ps_0318_2000050100_123_969_12185962.zip
ic_ancil_file: ic_ancil_12185962
ifsdata_file: ifsdata_12185962
climate_data_file: clim_data_12185962
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_beaf2ac5fa0ae1d5805c201a439f08b8 to: /var/lib/boinc/slots/31/ic_ancil_12185962.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/31/ic_ancil_12185962.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a09c86675ec165aca45d518230ea597b to: /var/lib/boinc/slots/31/ifsdata/ifsdata_12185962.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/31/ifsdata/ifsdata_12185962.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/31/159l_2/clim_data_12185962.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/31/159l_2/clim_data_12185962.zip
Checking for progress XML file: /var/lib/boinc/slots/31/progress_file_12185962.xml
Opened progress file ok : /var/lib/boinc/slots/31/progress_file_12185962.xml
-- Model is restarting --
Adjusting last_iter, 1077, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/31/progress_file_12185962.xml
last_cpu_time: 60949
upload_file_number: 44
last_iter: 1055
last_upload: 3801600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546
The child process has been launched with process id: 55
Executing the command: /var/lib/boinc/slots/31/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.003] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.005] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.005] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.005] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.005] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.005] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.005] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:55:55] [20221227:013520:1672104920:0.005] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0318
(argv4) batchid: 969
(argv5) wuid: 12185962
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/31
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/31/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/31/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ae305a224e06437cb560df237e326c7c to: /var/lib/boinc/slots/31/oifs_43r3_ps_0318_2000050100_123_969_12185962.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/31/oifs_43r3_ps_0318_2000050100_123_969_12185962.zip
ic_ancil_file: ic_ancil_12185962
ifsdata_file: ifsdata_12185962
climate_data_file: clim_data_12185962
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_beaf2ac5fa0ae1d5805c201a439f08b8 to: /var/lib/boinc/slots/31/ic_ancil_12185962.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/31/ic_ancil_12185962.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a09c86675ec165aca45d518230ea597b to: /var/lib/boinc/slots/31/ifsdata/ifsdata_12185962.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/31/ifsdata/ifsdata_12185962.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/31/159l_2/clim_data_12185962.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/31/159l_2/clim_data_12185962.zip
Checking for progress XML file: /var/lib/boinc/slots/31/progress_file_12185962.xml
Opened progress file ok : /var/lib/boinc/slots/31/progress_file_12185962.xml
-- Model is restarting --
Adjusting last_iter, 1056, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/31/progress_file_12185962.xml
last_cpu_time: 61013
upload_file_number: 44
last_iter: 1055
last_upload: 3801600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546
The child process has been launched with process id: 64
Executing the command: /var/lib/boinc/slots/31/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:64:64] [20221227:022634:1672107994:0.004] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0318
(argv4) batchid: 969
(argv5) wuid: 12185962
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/31
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/31/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/31/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ae305a224e06437cb560df237e326c7c to: /var/lib/boinc/slots/31/oifs_43r3_ps_0318_2000050100_123_969_12185962.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/31/oifs_43r3_ps_0318_2000050100_123_969_12185962.zip
ic_ancil_file: ic_ancil_12185962
ifsdata_file: ifsdata_12185962
climate_data_file: clim_data_12185962
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_beaf2ac5fa0ae1d5805c201a439f08b8 to: /var/lib/boinc/slots/31/ic_ancil_12185962.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/31/ic_ancil_12185962.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a09c86675ec165aca45d518230ea597b to: /var/lib/boinc/slots/31/ifsdata/ifsdata_12185962.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/31/ifsdata/ifsdata_12185962.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/31/159l_2/clim_data_12185962.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/31/159l_2/clim_data_12185962.zip
Checking for progress XML file: /var/lib/boinc/slots/31/progress_file_12185962.xml
Opened progress file ok : /var/lib/boinc/slots/31/progress_file_12185962.xml
-- Model is restarting --
Adjusting last_iter, 1056, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/31/progress_file_12185962.xml
last_cpu_time: 61024
upload_file_number: 44
last_iter: 1055
last_upload: 3801600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546
The child process has been launched with process id: 59
Executing the command: /var/lib/boinc/slots/31/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:6c1875ee1f44:1:1:59:59] [20221227:023541:1672108541:0.003] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:46:26 STEP 1056 H=1056:00 +CPU=107.423
  02:52:10 STEP 1057 H=1057:00 +CPU= 57.436
  02:58:40 STEP 1058 H=1058:00 +CPU= 65.414
  03:07:32 STEP 1059 H=1059:00 +CPU= 91.220
  03:12:46 STEP 1060 H=1060:00 +CPU= 54.910
  03:18:43 STEP 1061 H=1061:00 +CPU= 59.230
  03:27:47 STEP 1062 H=1062:00 +CPU= 93.232
  03:33:23 STEP 1063 H=1063:00 +CPU= 58.103
  03:39:05 STEP 1064 H=1064:00 +CPU= 58.466
  03:47:39 STEP 1065 H=1065:00 +CPU= 85.669
  03:52:54 STEP 1066 H=1066:00 +CPU= 53.467
  03:58:12 STEP 1067 H=1067:00 +CPU= 53.182
  04:08:04 STEP 1068 H=1068:00 +CPU= 99.936
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001068
  04:13:38 STEP 1069 H=1069:00 +CPU= 57.403
  04:19:47 STEP 1070 H=1070:00 +CPU= 61.024
  04:29:32 STEP 1071 H=1071:00 +CPU= 95.465
  04:35:05 STEP 1072 H=1072:00 +CPU= 55.941
  04:41:01 STEP 1073 H=1073:00 +CPU= 59.790
  04:50:01 STEP 1074 H=1074:00 +CPU= 90.411
  04:55:31 STEP 1075 H=1075:00 +CPU= 54.863
  05:00:41 STEP 1076 H=1076:00 +CPU= 53.759
  05:08:50 STEP 1077 H=1077:00 +CPU= 83.102
  05:13:49 STEP 1078 H=1078:00 +CPU= 51.886
  05:19:13 STEP 1079 H=1079:00 +CPU= 51.533
  05:27:50 STEP 1080 H=1080:00 +CPU= 87.881
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_44.zip
Uploading the intermediate file: upload_file_44.zip
  05:32:37 STEP 1081 H=1081:00 +CPU= 49.004
  05:37:40 STEP 1082 H=1082:00 +CPU= 51.113
  05:45:17 STEP 1083 H=1083:00 +CPU= 77.827
  05:50:09 STEP 1084 H=1084:00 +CPU= 50.825
  05:54:51 STEP 1085 H=1085:00 +CPU= 49.245
  06:02:22 STEP 1086 H=1086:00 +CPU= 77.660
  06:07:08 STEP 1087 H=1087:00 +CPU= 47.844
  06:11:30 STEP 1088 H=1088:00 +CPU= 45.891
  06:18:48 STEP 1089 H=1089:00 +CPU= 75.003
  06:23:46 STEP 1090 H=1090:00 +CPU= 51.755
  06:28:45 STEP 1091 H=1091:00 +CPU= 50.780
  06:36:53 STEP 1092 H=1092:00 +CPU= 84.320
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001092
  06:41:56 STEP 1093 H=1093:00 +CPU= 50.941
  06:46:57 STEP 1094 H=1094:00 +CPU= 51.031
  06:55:25 STEP 1095 H=1095:00 +CPU= 81.745
  07:00:18 STEP 1096 H=1096:00 +CPU= 50.466
  07:05:33 STEP 1097 H=1097:00 +CPU= 51.434
  07:12:40 STEP 1098 H=1098:00 +CPU= 74.304
  07:17:03 STEP 1099 H=1099:00 +CPU= 45.380
  07:22:05 STEP 1100 H=1100:00 +CPU= 52.102
  07:29:51 STEP 1101 H=1101:00 +CPU= 77.815
  07:35:05 STEP 1102 H=1102:00 +CPU= 51.710
  07:40:05 STEP 1103 H=1103:00 +CPU= 51.904
  07:49:25 STEP 1104 H=1104:00 +CPU= 93.332
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_45.zip
Uploading the intermediate file: upload_file_45.zip
  07:54:33 STEP 1105 H=1105:00 +CPU= 50.637
  07:59:29 STEP 1106 H=1106:00 +CPU= 51.473
  08:06:48 STEP 1107 H=1107:00 +CPU= 73.952
  08:11:54 STEP 1108 H=1108:00 +CPU= 51.524
  08:16:50 STEP 1109 H=1109:00 +CPU= 49.590
  08:23:56 STEP 1110 H=1110:00 +CPU= 73.235
  08:28:32 STEP 1111 H=1111:00 +CPU= 46.434
  08:33:16 STEP 1112 H=1112:00 +CPU= 49.095
  08:40:49 STEP 1113 H=1113:00 +CPU= 75.086
  08:45:43 STEP 1114 H=1114:00 +CPU= 50.208
  08:50:16 STEP 1115 H=1115:00 +CPU= 46.215
  08:58:50 STEP 1116 H=1116:00 +CPU= 85.751
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001116
  09:03:30 STEP 1117 H=1117:00 +CPU= 47.924
  09:08:26 STEP 1118 H=1118:00 +CPU= 49.796
  09:15:55 STEP 1119 H=1119:00 +CPU= 75.986
  09:20:41 STEP 1120 H=1120:00 +CPU= 49.353
  09:25:21 STEP 1121 H=1121:00 +CPU= 48.450
  09:33:19 STEP 1122 H=1122:00 +CPU= 79.489
  09:37:34 STEP 1123 H=1123:00 +CPU= 44.640
  09:42:33 STEP 1124 H=1124:00 +CPU= 51.212
  09:49:58 STEP 1125 H=1125:00 +CPU= 75.837
  09:54:56 STEP 1126 H=1126:00 +CPU= 48.907
  10:00:07 STEP 1127 H=1127:00 +CPU= 51.154
  10:09:16 STEP 1128 H=1128:00 +CPU= 92.101
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_46.zip
Uploading the intermediate file: upload_file_46.zip
  10:13:58 STEP 1129 H=1129:00 +CPU= 47.528
  10:18:58 STEP 1130 H=1130:00 +CPU= 49.361
  10:26:13 STEP 1131 H=1131:00 +CPU= 74.113
  10:30:57 STEP 1132 H=1132:00 +CPU= 49.443
  10:35:52 STEP 1133 H=1133:00 +CPU= 49.966
  10:43:56 STEP 1134 H=1134:00 +CPU= 80.654
  10:48:34 STEP 1135 H=1135:00 +CPU= 47.361
  10:53:28 STEP 1136 H=1136:00 +CPU= 50.069
  11:01:00 STEP 1137 H=1137:00 +CPU= 74.724
  11:05:37 STEP 1138 H=1138:00 +CPU= 47.401
  11:10:27 STEP 1139 H=1139:00 +CPU= 48.041
  11:19:03 STEP 1140 H=1140:00 +CPU= 85.106
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001140
  11:23:56 STEP 1141 H=1141:00 +CPU= 49.122
  11:28:48 STEP 1142 H=1142:00 +CPU= 48.971
  11:36:04 STEP 1143 H=1143:00 +CPU= 74.302
  11:40:44 STEP 1144 H=1144:00 +CPU= 48.212
  11:45:20 STEP 1145 H=1145:00 +CPU= 46.106
  11:53:28 STEP 1146 H=1146:00 +CPU= 79.136
  11:58:23 STEP 1147 H=1147:00 +CPU= 50.036
  12:03:32 STEP 1148 H=1148:00 +CPU= 50.199
  12:10:55 STEP 1149 H=1149:00 +CPU= 75.330
  12:15:54 STEP 1150 H=1150:00 +CPU= 49.975
  12:20:54 STEP 1151 H=1151:00 +CPU= 49.892
  12:29:36 STEP 1152 H=1152:00 +CPU= 87.847
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_47.zip
Uploading the intermediate file: upload_file_47.zip
  12:34:14 STEP 1153 H=1153:00 +CPU= 46.451
  12:39:10 STEP 1154 H=1154:00 +CPU= 48.487
  12:46:30 STEP 1155 H=1155:00 +CPU= 74.872
  12:50:53 STEP 1156 H=1156:00 +CPU= 45.897
  12:55:29 STEP 1157 H=1157:00 +CPU= 47.495
  13:03:49 STEP 1158 H=1158:00 +CPU= 80.611
  13:08:42 STEP 1159 H=1159:00 +CPU= 47.812
  13:13:33 STEP 1160 H=1160:00 +CPU= 49.699
  13:20:45 STEP 1161 H=1161:00 +CPU= 74.084
  13:26:08 STEP 1162 H=1162:00 +CPU= 50.812
  13:30:53 STEP 1163 H=1163:00 +CPU= 48.554
  13:38:50 STEP 1164 H=1164:00 +CPU= 81.207
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001164
  13:43:38 STEP 1165 H=1165:00 +CPU= 48.161
  13:48:30 STEP 1166 H=1166:00 +CPU= 48.736
  13:56:11 STEP 1167 H=1167:00 +CPU= 77.392
  14:00:52 STEP 1168 H=1168:00 +CPU= 48.101
  14:05:57 STEP 1169 H=1169:00 +CPU= 48.445
  14:14:11 STEP 1170 H=1170:00 +CPU= 80.844
  14:19:03 STEP 1171 H=1171:00 +CPU= 48.098
  14:23:25 STEP 1172 H=1172:00 +CPU= 45.825
  14:30:56 STEP 1173 H=1173:00 +CPU= 75.169
  14:35:57 STEP 1174 H=1174:00 +CPU= 50.197
  14:40:56 STEP 1175 H=1175:00 +CPU= 50.130
  14:50:22 STEP 1176 H=1176:00 +CPU= 91.643
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_48.zip
Uploading the intermediate file: upload_file_48.zip
  14:54:58 STEP 1177 H=1177:00 +CPU= 46.400
  14:59:32 STEP 1178 H=1178:00 +CPU= 47.113
  15:07:08 STEP 1179 H=1179:00 +CPU= 77.050
  15:11:47 STEP 1180 H=1180:00 +CPU= 47.153
  15:16:49 STEP 1181 H=1181:00 +CPU= 48.609
  15:25:38 STEP 1182 H=1182:00 +CPU= 82.091
  15:30:24 STEP 1183 H=1183:00 +CPU= 49.143
  15:35:39 STEP 1184 H=1184:00 +CPU= 51.245
  15:43:00 STEP 1185 H=1185:00 +CPU= 74.297
  15:47:46 STEP 1186 H=1186:00 +CPU= 48.500
  15:52:03 STEP 1187 H=1187:00 +CPU= 44.875
  16:00:10 STEP 1188 H=1188:00 +CPU= 82.847
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001188
  16:05:00 STEP 1189 H=1189:00 +CPU= 50.333
  16:10:08 STEP 1190 H=1190:00 +CPU= 52.641
  16:17:56 STEP 1191 H=1191:00 +CPU= 78.442
  16:23:05 STEP 1192 H=1192:00 +CPU= 51.542
  16:27:46 STEP 1193 H=1193:00 +CPU= 48.403
  16:35:24 STEP 1194 H=1194:00 +CPU= 77.409
  16:40:31 STEP 1195 H=1195:00 +CPU= 50.147
  16:45:07 STEP 1196 H=1196:00 +CPU= 47.461
  16:52:53 STEP 1197 H=1197:00 +CPU= 75.667
  16:57:28 STEP 1198 H=1198:00 +CPU= 47.197
  17:02:31 STEP 1199 H=1199:00 +CPU= 50.353
  17:10:57 STEP 1200 H=1200:00 +CPU= 87.589
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_49.zip
Uploading the intermediate file: upload_file_49.zip
  17:15:21 STEP 1201 H=1201:00 +CPU= 45.610
  17:20:16 STEP 1202 H=1202:00 +CPU= 49.128
  17:28:00 STEP 1203 H=1203:00 +CPU= 76.636
  17:33:15 STEP 1204 H=1204:00 +CPU= 50.051
  17:38:16 STEP 1205 H=1205:00 +CPU= 48.444
  17:45:59 STEP 1206 H=1206:00 +CPU= 78.003
  17:50:48 STEP 1207 H=1207:00 +CPU= 47.372
  17:55:39 STEP 1208 H=1208:00 +CPU= 47.658
  18:03:35 STEP 1209 H=1209:00 +CPU= 78.892
  18:07:58 STEP 1210 H=1210:00 +CPU= 45.969
  18:12:49 STEP 1211 H=1211:00 +CPU= 47.321
  18:21:07 STEP 1212 H=1212:00 +CPU= 85.718
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001212
  18:26:14 STEP 1213 H=1213:00 +CPU= 49.943
  18:31:30 STEP 1214 H=1214:00 +CPU= 52.395
  18:39:37 STEP 1215 H=1215:00 +CPU= 78.961
  18:44:41 STEP 1216 H=1216:00 +CPU= 51.032
  18:49:33 STEP 1217 H=1217:00 +CPU= 48.315
  18:57:12 STEP 1218 H=1218:00 +CPU= 77.309
  19:01:58 STEP 1219 H=1219:00 +CPU= 48.128
  19:06:51 STEP 1220 H=1220:00 +CPU= 49.576
  19:14:24 STEP 1221 H=1221:00 +CPU= 76.367
  19:19:23 STEP 1222 H=1222:00 +CPU= 50.427
  19:24:27 STEP 1223 H=1223:00 +CPU= 49.645
  19:33:25 STEP 1224 H=1224:00 +CPU= 89.430
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_50.zip
Uploading the intermediate file: upload_file_50.zip
  19:38:24 STEP 1225 H=1225:00 +CPU= 49.030
  19:43:49 STEP 1226 H=1226:00 +CPU= 51.331
  19:51:05 STEP 1227 H=1227:00 +CPU= 74.450
  19:56:08 STEP 1228 H=1228:00 +CPU= 49.242
  20:01:33 STEP 1229 H=1229:00 +CPU= 49.844
  20:09:17 STEP 1230 H=1230:00 +CPU= 78.261
  20:14:08 STEP 1231 H=1231:00 +CPU= 48.191
  20:18:46 STEP 1232 H=1232:00 +CPU= 46.866
  20:25:43 STEP 1233 H=1233:00 +CPU= 71.983
  20:31:23 STEP 1234 H=1234:00 +CPU= 52.367
  20:36:02 STEP 1235 H=1235:00 +CPU= 47.155
  20:44:33 STEP 1236 H=1236:00 +CPU= 85.212
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001236
  20:49:33 STEP 1237 H=1237:00 +CPU= 48.543
  20:54:28 STEP 1238 H=1238:00 +CPU= 50.709
  21:01:31 STEP 1239 H=1239:00 +CPU= 73.501
  21:06:22 STEP 1240 H=1240:00 +CPU= 49.329
  21:11:22 STEP 1241 H=1241:00 +CPU= 49.513
  21:19:23 STEP 1242 H=1242:00 +CPU= 80.745
  21:24:14 STEP 1243 H=1243:00 +CPU= 48.693
  21:29:00 STEP 1244 H=1244:00 +CPU= 48.241
  21:36:20 STEP 1245 H=1245:00 +CPU= 74.031
  21:41:15 STEP 1246 H=1246:00 +CPU= 49.122
  21:46:07 STEP 1247 H=1247:00 +CPU= 48.631
  21:54:53 STEP 1248 H=1248:00 +CPU= 87.596
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_51.zip
Uploading the intermediate file: upload_file_51.zip
  21:59:44 STEP 1249 H=1249:00 +CPU= 47.869
  22:04:46 STEP 1250 H=1250:00 +CPU= 49.909
  22:12:40 STEP 1251 H=1251:00 +CPU= 78.069
  22:17:48 STEP 1252 H=1252:00 +CPU= 51.191
  22:22:22 STEP 1253 H=1253:00 +CPU= 46.941
  22:30:18 STEP 1254 H=1254:00 +CPU= 78.417
  22:36:10 STEP 1255 H=1255:00 +CPU= 47.926
  22:40:52 STEP 1256 H=1256:00 +CPU= 47.381
  22:48:24 STEP 1257 H=1257:00 +CPU= 77.800
  22:53:43 STEP 1258 H=1258:00 +CPU= 51.344
  22:58:42 STEP 1259 H=1259:00 +CPU= 49.309
  23:07:31 STEP 1260 H=1260:00 +CPU= 87.434
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001260
  23:12:10 STEP 1261 H=1261:00 +CPU= 47.595
  23:16:50 STEP 1262 H=1262:00 +CPU= 49.093
  23:24:51 STEP 1263 H=1263:00 +CPU= 80.840
  23:30:00 STEP 1264 H=1264:00 +CPU= 50.781
  23:35:05 STEP 1265 H=1265:00 +CPU= 50.848
  23:42:52 STEP 1266 H=1266:00 +CPU= 78.163
  23:47:20 STEP 1267 H=1267:00 +CPU= 46.306
  23:52:30 STEP 1268 H=1268:00 +CPU= 50.315
  23:59:57 STEP 1269 H=1269:00 +CPU= 75.606
  00:05:09 STEP 1270 H=1270:00 +CPU= 51.556
  00:10:03 STEP 1271 H=1271:00 +CPU= 50.203
  00:18:59 STEP 1272 H=1272:00 +CPU= 88.791
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_52.zip
Uploading the intermediate file: upload_file_52.zip
  00:23:51 STEP 1273 H=1273:00 +CPU= 49.311
  00:28:48 STEP 1274 H=1274:00 +CPU= 51.353
  00:37:19 STEP 1275 H=1275:00 +CPU= 80.478
  00:42:06 STEP 1276 H=1276:00 +CPU= 48.389
  00:46:54 STEP 1277 H=1277:00 +CPU= 49.371
  00:54:11 STEP 1278 H=1278:00 +CPU= 76.002
  00:58:39 STEP 1279 H=1279:00 +CPU= 45.979
  01:03:24 STEP 1280 H=1280:00 +CPU= 49.496
  01:10:58 STEP 1281 H=1281:00 +CPU= 75.883
  01:16:03 STEP 1282 H=1282:00 +CPU= 51.742
  01:20:42 STEP 1283 H=1283:00 +CPU= 47.953
  01:29:31 STEP 1284 H=1284:00 +CPU= 87.744
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001284
  01:34:16 STEP 1285 H=1285:00 +CPU= 49.248
  01:38:57 STEP 1286 H=1286:00 +CPU= 48.756
  01:46:39 STEP 1287 H=1287:00 +CPU= 77.077
  01:51:42 STEP 1288 H=1288:00 +CPU= 51.817
  01:56:09 STEP 1289 H=1289:00 +CPU= 46.297
  02:03:42 STEP 1290 H=1290:00 +CPU= 76.469
  02:08:34 STEP 1291 H=1291:00 +CPU= 47.905
  02:13:14 STEP 1292 H=1292:00 +CPU= 48.545
  02:20:20 STEP 1293 H=1293:00 +CPU= 73.566
  02:25:29 STEP 1294 H=1294:00 +CPU= 52.124
  02:30:23 STEP 1295 H=1295:00 +CPU= 50.247
  02:38:33 STEP 1296 H=1296:00 +CPU= 83.740
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_53.zip
Uploading the intermediate file: upload_file_53.zip
Uploading trickle at timestep: 4665600
  02:43:28 STEP 1297 H=1297:00 +CPU= 50.404
  02:47:57 STEP 1298 H=1298:00 +CPU= 46.576
  02:55:36 STEP 1299 H=1299:00 +CPU= 78.375
  03:00:21 STEP 1300 H=1300:00 +CPU= 48.695
  03:04:52 STEP 1301 H=1301:00 +CPU= 46.251
  03:12:41 STEP 1302 H=1302:00 +CPU= 79.247
  03:17:33 STEP 1303 H=1303:00 +CPU= 49.729
  03:22:26 STEP 1304 H=1304:00 +CPU= 50.905
  03:29:31 STEP 1305 H=1305:00 +CPU= 73.629
  03:34:24 STEP 1306 H=1306:00 +CPU= 50.262
  03:38:53 STEP 1307 H=1307:00 +CPU= 46.680
  03:47:26 STEP 1308 H=1308:00 +CPU= 88.232
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001308
  03:52:32 STEP 1309 H=1309:00 +CPU= 51.224
  03:57:38 STEP 1310 H=1310:00 +CPU= 52.181
  04:05:46 STEP 1311 H=1311:00 +CPU= 78.805
  04:10:26 STEP 1312 H=1312:00 +CPU= 48.852
  04:15:15 STEP 1313 H=1313:00 +CPU= 47.236
  04:22:39 STEP 1314 H=1314:00 +CPU= 75.158
  04:27:33 STEP 1315 H=1315:00 +CPU= 48.253
  04:32:22 STEP 1316 H=1316:00 +CPU= 49.277
  04:39:58 STEP 1317 H=1317:00 +CPU= 78.739
  04:45:05 STEP 1318 H=1318:00 +CPU= 50.405
  04:49:39 STEP 1319 H=1319:00 +CPU= 47.569
  04:58:15 STEP 1320 H=1320:00 +CPU= 88.300
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_54.zip
Uploading the intermediate file: upload_file_54.zip
  05:03:02 STEP 1321 H=1321:00 +CPU= 47.772
  05:07:59 STEP 1322 H=1322:00 +CPU= 50.149
  05:15:32 STEP 1323 H=1323:00 +CPU= 78.300
  05:20:19 STEP 1324 H=1324:00 +CPU= 49.896
  05:25:00 STEP 1325 H=1325:00 +CPU= 49.070
  05:32:53 STEP 1326 H=1326:00 +CPU= 77.984
  05:37:23 STEP 1327 H=1327:00 +CPU= 46.169
  05:42:35 STEP 1328 H=1328:00 +CPU= 49.346
  05:49:49 STEP 1329 H=1329:00 +CPU= 74.469
  05:54:53 STEP 1330 H=1330:00 +CPU= 50.435
  05:59:35 STEP 1331 H=1331:00 +CPU= 48.797
  06:07:51 STEP 1332 H=1332:00 +CPU= 84.921
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001332
  06:12:50 STEP 1333 H=1333:00 +CPU= 49.293
  06:17:39 STEP 1334 H=1334:00 +CPU= 48.959
  06:24:53 STEP 1335 H=1335:00 +CPU= 74.399
  06:30:04 STEP 1336 H=1336:00 +CPU= 51.603
  06:35:01 STEP 1337 H=1337:00 +CPU= 50.163
  06:42:49 STEP 1338 H=1338:00 +CPU= 79.296
  06:47:59 STEP 1339 H=1339:00 +CPU= 50.294
  06:52:53 STEP 1340 H=1340:00 +CPU= 50.737
  07:00:44 STEP 1341 H=1341:00 +CPU= 77.604
  07:05:20 STEP 1342 H=1342:00 +CPU= 47.461
  07:10:12 STEP 1343 H=1343:00 +CPU= 48.568
  07:19:04 STEP 1344 H=1344:00 +CPU= 88.186
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_55.zip
Uploading the intermediate file: upload_file_55.zip
  07:23:24 STEP 1345 H=1345:00 +CPU= 44.963
  07:28:09 STEP 1346 H=1346:00 +CPU= 49.027
  07:35:51 STEP 1347 H=1347:00 +CPU= 77.132
  07:41:17 STEP 1348 H=1348:00 +CPU= 53.160
  07:46:02 STEP 1349 H=1349:00 +CPU= 48.453
  07:54:26 STEP 1350 H=1350:00 +CPU= 81.700
  07:59:29 STEP 1351 H=1351:00 +CPU= 49.895
  08:04:33 STEP 1352 H=1352:00 +CPU= 50.681
  08:13:07 STEP 1353 H=1353:00 +CPU= 79.784
  08:17:57 STEP 1354 H=1354:00 +CPU= 48.990
  08:22:09 STEP 1355 H=1355:00 +CPU= 44.140
  08:30:25 STEP 1356 H=1356:00 +CPU= 83.499
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001356
  08:35:18 STEP 1357 H=1357:00 +CPU= 47.568
  08:40:00 STEP 1358 H=1358:00 +CPU= 48.781
  08:46:35 STEP 1359 H=1359:00 +CPU= 68.799
  08:51:22 STEP 1360 H=1360:00 +CPU= 49.310
  08:56:33 STEP 1361 H=1361:00 +CPU= 50.497
  09:04:44 STEP 1362 H=1362:00 +CPU= 81.389
  09:09:35 STEP 1363 H=1363:00 +CPU= 47.927
  09:14:34 STEP 1364 H=1364:00 +CPU= 50.774
  09:22:18 STEP 1365 H=1365:00 +CPU= 78.038
  09:27:56 STEP 1366 H=1366:00 +CPU= 53.117
  09:32:57 STEP 1367 H=1367:00 +CPU= 50.303
  09:41:45 STEP 1368 H=1368:00 +CPU= 88.730
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_56.zip
Uploading the intermediate file: upload_file_56.zip
  09:46:29 STEP 1369 H=1369:00 +CPU= 47.364
  09:51:16 STEP 1370 H=1370:00 +CPU= 48.232
  09:58:17 STEP 1371 H=1371:00 +CPU= 72.514
  10:03:12 STEP 1372 H=1372:00 +CPU= 50.155
  10:08:21 STEP 1373 H=1373:00 +CPU= 49.970
  10:15:46 STEP 1374 H=1374:00 +CPU= 75.453
  10:20:08 STEP 1375 H=1375:00 +CPU= 45.544
  10:24:52 STEP 1376 H=1376:00 +CPU= 48.600
  10:32:08 STEP 1377 H=1377:00 +CPU= 74.645
  10:37:04 STEP 1378 H=1378:00 +CPU= 49.980
  10:41:53 STEP 1379 H=1379:00 +CPU= 47.861
  10:50:07 STEP 1380 H=1380:00 +CPU= 82.998
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001380
  10:54:18 STEP 1381 H=1381:00 +CPU= 43.817
  10:59:08 STEP 1382 H=1382:00 +CPU= 50.464
  11:07:20 STEP 1383 H=1383:00 +CPU= 77.736
  11:12:43 STEP 1384 H=1384:00 +CPU= 52.672
  11:17:22 STEP 1385 H=1385:00 +CPU= 48.345
  11:24:53 STEP 1386 H=1386:00 +CPU= 77.687
  11:30:00 STEP 1387 H=1387:00 +CPU= 49.548
  11:34:41 STEP 1388 H=1388:00 +CPU= 48.505
  11:42:38 STEP 1389 H=1389:00 +CPU= 77.624
  11:48:06 STEP 1390 H=1390:00 +CPU= 53.308
  11:53:08 STEP 1391 H=1391:00 +CPU= 50.170
  12:01:26 STEP 1392 H=1392:00 +CPU= 85.307
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_57.zip
Uploading the intermediate file: upload_file_57.zip
  12:05:41 STEP 1393 H=1393:00 +CPU= 44.725
  12:10:25 STEP 1394 H=1394:00 +CPU= 49.573
  12:17:35 STEP 1395 H=1395:00 +CPU= 73.419
  12:22:58 STEP 1396 H=1396:00 +CPU= 52.365
  12:27:45 STEP 1397 H=1397:00 +CPU= 47.386
  12:35:58 STEP 1398 H=1398:00 +CPU= 81.310
  12:41:00 STEP 1399 H=1399:00 +CPU= 51.579
  12:46:33 STEP 1400 H=1400:00 +CPU= 52.807
  12:53:47 STEP 1401 H=1401:00 +CPU= 75.703
  12:58:46 STEP 1402 H=1402:00 +CPU= 51.230
  13:03:12 STEP 1403 H=1403:00 +CPU= 45.704
  13:11:40 STEP 1404 H=1404:00 +CPU= 84.111
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001404
  13:16:28 STEP 1405 H=1405:00 +CPU= 46.807
  13:20:55 STEP 1406 H=1406:00 +CPU= 45.844
  13:28:16 STEP 1407 H=1407:00 +CPU= 75.974
  13:32:47 STEP 1408 H=1408:00 +CPU= 46.671
  13:37:42 STEP 1409 H=1409:00 +CPU= 49.126
  13:45:00 STEP 1410 H=1410:00 +CPU= 75.873
  13:50:00 STEP 1411 H=1411:00 +CPU= 49.293
  13:55:07 STEP 1412 H=1412:00 +CPU= 49.809
  14:02:35 STEP 1413 H=1413:00 +CPU= 77.102
  14:07:31 STEP 1414 H=1414:00 +CPU= 49.901
  14:12:11 STEP 1415 H=1415:00 +CPU= 48.287
  14:21:28 STEP 1416 H=1416:00 +CPU= 89.900
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12185962/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0318_2000050100_123_969_12185962_0_r1431885546_58.zip
Uploading the intermediate file: upload_file_58.zip
  14:26:33 STEP 1417 H=1417:00 +CPU= 49.125
  14:31:26 STEP 1418 H=1418:00 +CPU= 50.166
  14:38:23 STEP 1419 H=1419:00 +CPU= 71.267
  14:43:19 STEP 1420 H=1420:00 +CPU= 48.674
  14:48:04 STEP 1421 H=1421:00 +CPU= 48.503
  14:55:39 STEP 1422 H=1422:00 +CPU= 77.331
  15:01:05 STEP 1423 H=1423:00 +CPU= 51.142
  15:06:33 STEP 1424 H=1424:00 +CPU= 52.927
  15:14:08 STEP 1425 H=1425:00 +CPU= 75.695
  15:19:00 STEP 1426 H=1426:00 +CPU= 50.116
  15:23:56 STEP 1427 H=1427:00 +CPU= 50.085
  15:32:31 STEP 1428 H=1428:00 +CPU= 84.989
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+001428
  15:37:20 STEP 1429 H=1429:00 +CPU= 49.598
  15:42:17 STEP 1430 H=1430:00 +CPU= 49.865
  15:50:02 STEP 1431 H=1431:00 +CPU= 75.981
  15:54:25 STEP 1432 H=1432:00 +CPU= 46.039
  15:59:02 STEP 1433 H=1433:00 +CPU= 46.297
  16:06:32 STEP 1434 H=1434:00 +CPU= 75.849
  16:11:12 STEP 1435 H=1435:00 +CPU= 48.110
  16:16:10 STEP 1436 H=1436:00 +CPU= 50.269
  16:23:58 STEP 1437 H=1437:00 +CPU= 77.774
  16:28:50 STEP 1438 H=1438:00 +CPU= 48.007
  16:33:35 STEP 1439 H=1439:00 +CPU= 49.123
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:59:34 1488917 22271313 oifs_43r3_ps_0318_2000050100_123_969_12185962_0 4,665,600 75,557 0.0162
09 May 2023 15:57:07 1488917 22271313 oifs_43r3_ps_0318_2000050100_123_969_12185962_0 3,456,000 54,119 0.0157
09 May 2023 15:55:13 1488917 22271313 oifs_43r3_ps_0318_2000050100_123_969_12185962_0 2,592,000 40,057 0.0155
09 May 2023 15:52:49 1488917 22271313 oifs_43r3_ps_0318_2000050100_123_969_12185962_0 1,728,000 25,981 0.0150
09 May 2023 15:50:52 1488917 22271313 oifs_43r3_ps_0318_2000050100_123_969_12185962_0 864,000 12,826 0.0148


©2024 climateprediction.net