climateprediction.net home page
Task 22279252

Task 22279252

Name oifs_43r3_ps_0817_2007050100_123_976_12193461_0
Workunit 12193461
Created 23 Dec 2022, 16:47:46 UTC
Sent 4 Jan 2023, 13:54:59 UTC
Report deadline 3 Feb 2023, 13:54:59 UTC
Received 11 Jan 2023, 13:46:13 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1536947
Run time 14 hours 1 min 23 sec
CPU time 3 hours 49 min 4 sec
Validate state Invalid
Credit 5,806.08
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,784.64 MB
Peak swap size 4,978.15 MB
Peak disk usage 7,401.33 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
est received from the BOINC client, suspending the child process
Resuming the child process
  16:28:54 STEP 1220 H=1220:00 +CPU= 22.131
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:29:31 STEP 1221 H=1221:00 +CPU= 32.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:29:54 STEP 1222 H=1222:00 +CPU= 21.179
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:30:15 STEP 1223 H=1223:00 +CPU= 19.506
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:31:02 STEP 1224 H=1224:00 +CPU= 41.034
Moving to projects directory: /root/slots/9/ICMGGhq0f+001224
Moving to projects directory: /root/slots/9/ICMSHhq0f+001224
Moving to projects directory: /root/slots/9/ICMUAhq0f+001224
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001200
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001200
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001200
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001212
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001212
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001212
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612_13.zip
Uploading the intermediate file: upload_file_13.zip
  16:31:24 STEP 1225 H=1225:00 +CPU= 21.743
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:31:49 STEP 1226 H=1226:00 +CPU= 22.516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:32:26 STEP 1227 H=1227:00 +CPU= 32.919
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:32:50 STEP 1228 H=1228:00 +CPU= 22.344
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:13 STEP 1229 H=1229:00 +CPU= 21.198
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:52 STEP 1230 H=1230:00 +CPU= 34.239
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:45 STEP 1231 H=1231:00 +CPU= 22.722
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:10 STEP 1232 H=1232:00 +CPU= 22.554
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:48 STEP 1233 H=1233:00 +CPU= 34.492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:11 STEP 1234 H=1234:00 +CPU= 20.844
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:34 STEP 1235 H=1235:00 +CPU= 19.792
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:13 STEP 1236 H=1236:00 +CPU= 34.210
Moving to projects directory: /root/slots/9/ICMGGhq0f+001236
Moving to projects directory: /root/slots/9/ICMSHhq0f+001236
Moving to projects directory: /root/slots/9/ICMUAhq0f+001236
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:33 STEP 1237 H=1237:00 +CPU= 18.360
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:55 STEP 1238 H=1238:00 +CPU= 20.228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2007050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0817
(argv4) batchid: 976
(argv5) wuid: 12193461
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/9
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12193461
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b37ee2a4c9bc26bf42c65b341a2f9d54 to: /root/slots/9/oifs_43r3_ps_0817_2007050100_123_976_12193461.zip
Unzipping the namelist zip file: /root/slots/9/oifs_43r3_ps_0817_2007050100_123_976_12193461.zip
ic_ancil_file: ic_ancil_12193461
ifsdata_file: ifsdata_12193461
climate_data_file: clim_data_12193461
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_859c21f9ca2d97c8156e43912754ce87 to: /root/slots/9/ic_ancil_12193461.zip
Unzipping the IC ancils zip file: /root/slots/9/ic_ancil_12193461.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5c583d9103d29f124203e00280f80983 to: /root/slots/9/ifsdata/ifsdata_12193461.zip
Unzipping the ifsdata_zip file: /root/slots/9/ifsdata/ifsdata_12193461.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/9/159l_2/clim_data_12193461.zip
Unzipping the climate data zip file: /root/slots/9/159l_2/clim_data_12193461.zip
Checking for progress XML file: /root/slots/9/progress_file_12193461.xml
Opened progress file ok : /root/slots/9/progress_file_12193461.xml
-- Model is restarting --
Adjusting last_iter, 1239, to previous model restart step.
Creating progress file: /root/slots/9/progress_file_12193461.xml
last_cpu_time: 9969
upload_file_number: 14
last_iter: 1223
last_upload: 4406400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612
The child process has been launched with process id: 7670
Executing the command: /root/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7670:7670] [20230109:161343:1673277223:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:30 STEP 1224 H=1224:00 +CPU= 37.354
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:14:56 STEP 1225 H=1225:00 +CPU= 22.831
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:25 STEP 1226 H=1226:00 +CPU= 23.312
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:05 STEP 1227 H=1227:00 +CPU= 34.569
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:16:31 STEP 1228 H=1228:00 +CPU= 23.588
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:17:24 STEP 1229 H=1229:00 +CPU= 23.666
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:08 STEP 1230 H=1230:00 +CPU= 38.119
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:51 STEP 1231 H=1231:00 +CPU= 23.650
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:19:53 STEP 1232 H=1232:00 +CPU= 25.252
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:20:51 STEP 1233 H=1233:00 +CPU= 35.986
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:16 STEP 1234 H=1234:00 +CPU= 22.830
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:21:42 STEP 1235 H=1235:00 +CPU= 22.350
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:22:27 STEP 1236 H=1236:00 +CPU= 38.021
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/9/ICMGGhq0f+001236
Moving to projects directory: /root/slots/9/ICMSHhq0f+001236
Moving to projects directory: /root/slots/9/ICMUAhq0f+001236
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:05 STEP 1237 H=1237:00 +CPU= 22.778
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:23:48 STEP 1238 H=1238:00 +CPU= 24.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:24:32 STEP 1239 H=1239:00 +CPU= 35.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:25:15 STEP 1240 H=1240:00 +CPU= 24.710
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:25:56 STEP 1241 H=1241:00 +CPU= 25.105
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:26:57 STEP 1242 H=1242:00 +CPU= 40.393
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:27:41 STEP 1243 H=1243:00 +CPU= 25.419
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:28:35 STEP 1244 H=1244:00 +CPU= 26.340
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:29:56 STEP 1245 H=1245:00 +CPU= 40.337
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:30:39 STEP 1246 H=1246:00 +CPU= 26.163
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:31:11 STEP 1247 H=1247:00 +CPU= 26.015
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:32:41 STEP 1248 H=1248:00 +CPU= 46.193
Moving to projects directory: /root/slots/9/ICMGGhq0f+001248
Moving to projects directory: /root/slots/9/ICMSHhq0f+001248
Moving to projects directory: /root/slots/9/ICMUAhq0f+001248
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001224
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001224
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001224
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001236
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001236
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001236
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612_14.zip
Uploading the intermediate file: upload_file_14.zip
  16:33:03 STEP 1249 H=1249:00 +CPU= 21.059
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:33:34 STEP 1250 H=1250:00 +CPU= 24.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:11 STEP 1251 H=1251:00 +CPU= 31.713
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:34 STEP 1252 H=1252:00 +CPU= 19.572
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:34:56 STEP 1253 H=1253:00 +CPU= 19.343
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:33 STEP 1254 H=1254:00 +CPU= 31.201
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:35:55 STEP 1255 H=1255:00 +CPU= 18.963
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:18 STEP 1256 H=1256:00 +CPU= 19.771
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:36:55 STEP 1257 H=1257:00 +CPU= 31.258
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:18 STEP 1258 H=1258:00 +CPU= 20.100
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:37:37 STEP 1259 H=1259:00 +CPU= 16.613
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:38:08 STEP 1260 H=1260:00 +CPU= 28.802
Moving to projects directory: /root/slots/9/ICMGGhq0f+001260
Moving to projects directory: /root/slots/9/ICMSHhq0f+001260
Moving to projects directory: /root/slots/9/ICMUAhq0f+001260
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:38:26 STEP 1261 H=1261:00 +CPU= 16.053
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:38:45 STEP 1262 H=1262:00 +CPU= 16.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:16 STEP 1263 H=1263:00 +CPU= 26.816
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:35 STEP 1264 H=1264:00 +CPU= 16.951
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:39:53 STEP 1265 H=1265:00 +CPU= 16.003
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:40:22 STEP 1266 H=1266:00 +CPU= 26.621
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:40:40 STEP 1267 H=1267:00 +CPU= 16.141
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:40:59 STEP 1268 H=1268:00 +CPU= 16.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:41:27 STEP 1269 H=1269:00 +CPU= 25.761
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:41:45 STEP 1270 H=1270:00 +CPU= 16.484
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:42:04 STEP 1271 H=1271:00 +CPU= 16.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:42:38 STEP 1272 H=1272:00 +CPU= 29.752
Moving to projects directory: /root/slots/9/ICMGGhq0f+001272
Moving to projects directory: /root/slots/9/ICMSHhq0f+001272
Moving to projects directory: /root/slots/9/ICMUAhq0f+001272
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001248
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001248
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001248
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001260
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001260
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001260
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612_15.zip
Uploading the intermediate file: upload_file_15.zip
  16:42:54 STEP 1273 H=1273:00 +CPU= 15.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:43:12 STEP 1274 H=1274:00 +CPU= 16.573
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:43:41 STEP 1275 H=1275:00 +CPU= 26.285
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:00 STEP 1276 H=1276:00 +CPU= 16.566
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:18 STEP 1277 H=1277:00 +CPU= 15.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:44:46 STEP 1278 H=1278:00 +CPU= 26.409
  16:45:02 STEP 1279 H=1279:00 +CPU= 15.941
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:45:21 STEP 1280 H=1280:00 +CPU= 16.457
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:45:51 STEP 1281 H=1281:00 +CPU= 25.952
  16:46:08 STEP 1282 H=1282:00 +CPU= 16.530
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:46:26 STEP 1283 H=1283:00 +CPU= 16.000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:46:58 STEP 1284 H=1284:00 +CPU= 28.436
Moving to projects directory: /root/slots/9/ICMGGhq0f+001284
Moving to projects directory: /root/slots/9/ICMSHhq0f+001284
Moving to projects directory: /root/slots/9/ICMUAhq0f+001284
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:16 STEP 1285 H=1285:00 +CPU= 15.934
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:35 STEP 1286 H=1286:00 +CPU= 16.717
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:48:03 STEP 1287 H=1287:00 +CPU= 25.758
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:48:22 STEP 1288 H=1288:00 +CPU= 16.642
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:48:40 STEP 1289 H=1289:00 +CPU= 16.019
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:49:11 STEP 1290 H=1290:00 +CPU= 26.575
  16:49:27 STEP 1291 H=1291:00 +CPU= 16.072
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:49:45 STEP 1292 H=1292:00 +CPU= 16.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:16 STEP 1293 H=1293:00 +CPU= 26.066
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:34 STEP 1294 H=1294:00 +CPU= 16.433
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:50:53 STEP 1295 H=1295:00 +CPU= 16.537
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:51:26 STEP 1296 H=1296:00 +CPU= 30.494
Moving to projects directory: /root/slots/9/ICMGGhq0f+001296
Moving to projects directory: /root/slots/9/ICMSHhq0f+001296
Moving to projects directory: /root/slots/9/ICMUAhq0f+001296
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001272
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001272
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001272
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001284
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001284
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001284
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612_16.zip
Uploading the intermediate file: upload_file_16.zip
  16:51:42 STEP 1297 H=1297:00 +CPU= 15.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:52:00 STEP 1298 H=1298:00 +CPU= 16.460
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:52:28 STEP 1299 H=1299:00 +CPU= 25.639
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:52:46 STEP 1300 H=1300:00 +CPU= 16.399
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:53:05 STEP 1301 H=1301:00 +CPU= 16.131
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:53:36 STEP 1302 H=1302:00 +CPU= 26.828
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:53:54 STEP 1303 H=1303:00 +CPU= 15.820
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:54:13 STEP 1304 H=1304:00 +CPU= 16.736
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:54:41 STEP 1305 H=1305:00 +CPU= 26.361
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:00 STEP 1306 H=1306:00 +CPU= 16.557
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:18 STEP 1307 H=1307:00 +CPU= 15.985
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:55:50 STEP 1308 H=1308:00 +CPU= 28.494
Moving to projects directory: /root/slots/9/ICMGGhq0f+001308
Moving to projects directory: /root/slots/9/ICMSHhq0f+001308
Moving to projects directory: /root/slots/9/ICMUAhq0f+001308
  16:56:07 STEP 1309 H=1309:00 +CPU= 16.023
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:56:25 STEP 1310 H=1310:00 +CPU= 16.576
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:56:56 STEP 1311 H=1311:00 +CPU= 26.279
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:57:15 STEP 1312 H=1312:00 +CPU= 16.557
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:57:32 STEP 1313 H=1313:00 +CPU= 15.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:58:01 STEP 1314 H=1314:00 +CPU= 26.526
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:58:19 STEP 1315 H=1315:00 +CPU= 15.960
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:58:38 STEP 1316 H=1316:00 +CPU= 16.652
  16:59:04 STEP 1317 H=1317:00 +CPU= 26.068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:59:23 STEP 1318 H=1318:00 +CPU= 16.706
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:59:42 STEP 1319 H=1319:00 +CPU= 16.687
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:00:14 STEP 1320 H=1320:00 +CPU= 30.187
Moving to projects directory: /root/slots/9/ICMGGhq0f+001320
Moving to projects directory: /root/slots/9/ICMSHhq0f+001320
Moving to projects directory: /root/slots/9/ICMUAhq0f+001320
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001296
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001296
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001296
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001308
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001308
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001308
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612_17.zip
Uploading the intermediate file: upload_file_17.zip
  17:00:30 STEP 1321 H=1321:00 +CPU= 15.842
  17:00:47 STEP 1322 H=1322:00 +CPU= 16.636
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:01:18 STEP 1323 H=1323:00 +CPU= 28.384
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:01:44 STEP 1324 H=1324:00 +CPU= 20.012
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:03:43 STEP 1325 H=1325:00 +CPU= 18.848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:06:03 STEP 1326 H=1326:00 +CPU= 30.990
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:06:46 STEP 1327 H=1327:00 +CPU= 18.441
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:09:15 STEP 1328 H=1328:00 +CPU= 18.827
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:09:47 STEP 1329 H=1329:00 +CPU= 28.965
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:10:12 STEP 1330 H=1330:00 +CPU= 18.930
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:11:10 STEP 1331 H=1331:00 +CPU= 19.275
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:12:06 STEP 1332 H=1332:00 +CPU= 32.177
Moving to projects directory: /root/slots/9/ICMGGhq0f+001332
Moving to projects directory: /root/slots/9/ICMSHhq0f+001332
Moving to projects directory: /root/slots/9/ICMUAhq0f+001332
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:12:37 STEP 1333 H=1333:00 +CPU= 18.298
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:13:30 STEP 1334 H=1334:00 +CPU= 18.887
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:14:15 STEP 1335 H=1335:00 +CPU= 29.581
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:15:00 STEP 1336 H=1336:00 +CPU= 19.247
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:15:30 STEP 1337 H=1337:00 +CPU= 18.395
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:16:15 STEP 1338 H=1338:00 +CPU= 29.949
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:16:35 STEP 1339 H=1339:00 +CPU= 17.773
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:16:57 STEP 1340 H=1340:00 +CPU= 18.672
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:17:31 STEP 1341 H=1341:00 +CPU= 29.278
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:17:52 STEP 1342 H=1342:00 +CPU= 18.399
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:18:14 STEP 1343 H=1343:00 +CPU= 18.610
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:18:53 STEP 1344 H=1344:00 +CPU= 33.678
Moving to projects directory: /root/slots/9/ICMGGhq0f+001344
Moving to projects directory: /root/slots/9/ICMSHhq0f+001344
Moving to projects directory: /root/slots/9/ICMUAhq0f+001344
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001320
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001320
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001320
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001332
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001332
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001332
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612_18.zip
Uploading the intermediate file: upload_file_18.zip
  17:19:12 STEP 1345 H=1345:00 +CPU= 17.924
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:19:33 STEP 1346 H=1346:00 +CPU= 18.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:20:06 STEP 1347 H=1347:00 +CPU= 29.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:20:27 STEP 1348 H=1348:00 +CPU= 18.517
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:20:48 STEP 1349 H=1349:00 +CPU= 18.106
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:21:24 STEP 1350 H=1350:00 +CPU= 30.053
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:21:45 STEP 1351 H=1351:00 +CPU= 18.089
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:22:06 STEP 1352 H=1352:00 +CPU= 18.795
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:22:41 STEP 1353 H=1353:00 +CPU= 29.393
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:23:02 STEP 1354 H=1354:00 +CPU= 18.730
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:23:33 STEP 1355 H=1355:00 +CPU= 18.158
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:24:11 STEP 1356 H=1356:00 +CPU= 32.213
Moving to projects directory: /root/slots/9/ICMGGhq0f+001356
Moving to projects directory: /root/slots/9/ICMSHhq0f+001356
Moving to projects directory: /root/slots/9/ICMUAhq0f+001356
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:24:32 STEP 1357 H=1357:00 +CPU= 17.830
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:24:53 STEP 1358 H=1358:00 +CPU= 18.475
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:25:24 STEP 1359 H=1359:00 +CPU= 26.629
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:25:43 STEP 1360 H=1360:00 +CPU= 16.348
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:26:01 STEP 1361 H=1361:00 +CPU= 16.092
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:26:30 STEP 1362 H=1362:00 +CPU= 26.734
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:26:48 STEP 1363 H=1363:00 +CPU= 15.856
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:27:07 STEP 1364 H=1364:00 +CPU= 16.636
  17:27:33 STEP 1365 H=1365:00 +CPU= 25.993
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:27:51 STEP 1366 H=1366:00 +CPU= 16.240
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:28:10 STEP 1367 H=1367:00 +CPU= 16.527
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:28:45 STEP 1368 H=1368:00 +CPU= 31.912
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/9/ICMGGhq0f+001368
Moving to projects directory: /root/slots/9/ICMSHhq0f+001368
Moving to projects directory: /root/slots/9/ICMUAhq0f+001368
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001344
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001344
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001344
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMGGhq0f+001356
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMSHhq0f+001356
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12193461/ICMUAhq0f+001356
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612_19.zip
Uploading the intermediate file: upload_file_19.zip
  17:29:18 STEP 1369 H=1369:00 +CPU= 18.685
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2007050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0817
(argv4) batchid: 976
(argv5) wuid: 12193461
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/9
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12193461
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b37ee2a4c9bc26bf42c65b341a2f9d54 to: /root/slots/9/oifs_43r3_ps_0817_2007050100_123_976_12193461.zip
Unzipping the namelist zip file: /root/slots/9/oifs_43r3_ps_0817_2007050100_123_976_12193461.zip
ic_ancil_file: ic_ancil_12193461
ifsdata_file: ifsdata_12193461
climate_data_file: clim_data_12193461
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_859c21f9ca2d97c8156e43912754ce87 to: /root/slots/9/ic_ancil_12193461.zip
Unzipping the IC ancils zip file: /root/slots/9/ic_ancil_12193461.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5c583d9103d29f124203e00280f80983 to: /root/slots/9/ifsdata/ifsdata_12193461.zip
Unzipping the ifsdata_zip file: /root/slots/9/ifsdata/ifsdata_12193461.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/9/159l_2/clim_data_12193461.zip
Unzipping the climate data zip file: /root/slots/9/159l_2/clim_data_12193461.zip
Checking for progress XML file: /root/slots/9/progress_file_12193461.xml
Opened progress file ok : /root/slots/9/progress_file_12193461.xml
-- Model is restarting --
Adjusting last_iter, 1369, to previous model restart step.
Creating progress file: /root/slots/9/progress_file_12193461.xml
last_cpu_time: 13239
upload_file_number: 20
last_iter: 1367
last_upload: 4924800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0817_2007050100_123_976_12193461_0_r388888612
The child process has been launched with process id: 8108
Executing the command: /root/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:8108:8108] [20230111:143336:1673444016:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:34:11 STEP 1368 H=1368:00 +CPU= 28.712
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:34:29 STEP 1369 H=1369:00 +CPU= 15.802
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:34:47 STEP 1370 H=1370:00 +CPU= 16.144
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:35:15 STEP 1371 H=1371:00 +CPU= 25.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:35:33 STEP 1372 H=1372:00 +CPU= 16.266
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:35:51 STEP 1373 H=1373:00 +CPU= 15.667
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:36:24 STEP 1374 H=1374:00 +CPU= 28.299
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:36:42 STEP 1375 H=1375:00 +CPU= 15.981
  14:36:58 STEP 1376 H=1376:00 +CPU= 16.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:37:28 STEP 1377 H=1377:00 +CPU= 25.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:37:47 STEP 1378 H=1378:00 +CPU= 16.373
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:38:05 STEP 1379 H=1379:00 +CPU= 15.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:38:35 STEP 1380 H=1380:00 +CPU= 28.688
Moving to projects directory: /root/slots/9/ICMGGhq0f+001380
Moving to projects directory: /root/slots/9/ICMSHhq0f+001380
Moving to projects directory: /root/slots/9/ICMUAhq0f+001380
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:38:54 STEP 1381 H=1381:00 +CPU= 16.289
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:39:12 STEP 1382 H=1382:00 +CPU= 16.371
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:39:42 STEP 1383 H=1383:00 +CPU= 25.764
  14:39:59 STEP 1384 H=1384:00 +CPU= 16.638
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:40:17 STEP 1385 H=1385:00 +CPU= 15.845
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:40:47 STEP 1386 H=1386:00 +CPU= 26.169
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:41:05 STEP 1387 H=1387:00 +CPU= 15.961
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:41:23 STEP 1388 H=1388:00 +CPU= 16.387
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:41:51 STEP 1389 H=1389:00 +CPU= 25.857
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:10 STEP 1390 H=1390:00 +CPU= 16.268
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:28 STEP 1391 H=1391:00 +CPU= 16.321
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:59 STEP 1392 H=1392:00 +CPU= 29.368
Suspend request received from the BOINC client, suspending the child process
Abort request received from the BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:12:50 1536947 22279252 oifs_43r3_ps_0817_2007050100_123_976_12193461_0 4,147,200 7,068 0.0017
09 May 2023 16:12:50 1536947 22279252 oifs_43r3_ps_0817_2007050100_123_976_12193461_0 4,147,200 7,068 0.0017
09 May 2023 16:12:44 1536947 22279252 oifs_43r3_ps_0817_2007050100_123_976_12193461_0 3,024,000 30,918 0.0102
09 May 2023 16:09:17 1536947 22279252 oifs_43r3_ps_0817_2007050100_123_976_12193461_0 1,123,200 12,154 0.0108


©2024 cpdn.org