Name | oifs_43r3_ps_0460_1981050100_123_950_12167104_0 |
Workunit | 12167104 |
Created | 20 Dec 2022, 19:34:46 UTC |
Sent | 20 Dec 2022, 19:43:04 UTC |
Report deadline | 19 Jan 2023, 19:43:04 UTC |
Received | 21 Dec 2022, 6:13:32 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1519861 |
Run time | 9 hours 14 min |
CPU time | 7 hours 3 min 56 sec |
Validate state | Invalid |
Credit | 3,991.68 |
Device peak FLOPS | 4.57 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,381.69 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 7,682.05 MB |
Stderr | <core_client_version>7.14.2</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> 3r3_ps_0460_1981050100_123_950_12167104.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0460_1981050100_123_950_12167104.zip ic_ancil_file: ic_ancil_12167104 ifsdata_file: ifsdata_12167104 climate_data_file: clim_data_12167104 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_68d381a9e1c550f72b731870547dfcad to: /var/lib/boinc-client/slots/12/ic_ancil_12167104.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12167104.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ce85c9d34336b5a82f194b1396b48873 to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12167104.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12167104.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12167104.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12167104.zip Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12167104.xml Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12167104.xml -- Model is restarting -- Adjusting last_iter, 571, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12167104.xml last_cpu_time: 18580 upload_file_number: 23 last_iter: 551 last_upload: 1987200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505 The child process has been launched with process id: 32420 Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32420:32420] [20221221:032457:1671593097:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 03:25:58 STEP 552 H= 552:00 +CPU= 36.256 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1981050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0460 (argv4) batchid: 950 (argv5) wuid: 12167104 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/12 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_e705484d46f33894f27d1becd343edb9 to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0460_1981050100_123_950_12167104.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0460_1981050100_123_950_12167104.zip ic_ancil_file: ic_ancil_12167104 ifsdata_file: ifsdata_12167104 climate_data_file: clim_data_12167104 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_68d381a9e1c550f72b731870547dfcad to: /var/lib/boinc-client/slots/12/ic_ancil_12167104.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12167104.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ce85c9d34336b5a82f194b1396b48873 to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12167104.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12167104.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12167104.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12167104.zip Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12167104.xml Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12167104.xml -- Model is restarting -- Adjusting last_iter, 553, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12167104.xml last_cpu_time: 18628 upload_file_number: 23 last_iter: 551 last_upload: 1987200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505 The child process has been launched with process id: 32444 Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32444:32444] [20221221:032740:1671593260:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 03:28:37 STEP 552 H= 552:00 +CPU= 38.767 03:29:00 STEP 553 H= 553:00 +CPU= 20.116 03:29:20 STEP 554 H= 554:00 +CPU= 20.198 03:30:03 STEP 555 H= 555:00 +CPU= 31.258 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1981050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0460 (argv4) batchid: 950 (argv5) wuid: 12167104 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/12 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_e705484d46f33894f27d1becd343edb9 to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0460_1981050100_123_950_12167104.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0460_1981050100_123_950_12167104.zip ic_ancil_file: ic_ancil_12167104 ifsdata_file: ifsdata_12167104 climate_data_file: clim_data_12167104 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_68d381a9e1c550f72b731870547dfcad to: /var/lib/boinc-client/slots/12/ic_ancil_12167104.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12167104.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ce85c9d34336b5a82f194b1396b48873 to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12167104.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12167104.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12167104.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12167104.zip Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12167104.xml Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12167104.xml -- Model is restarting -- Adjusting last_iter, 556, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12167104.xml last_cpu_time: 18747 upload_file_number: 23 last_iter: 551 last_upload: 1987200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505 The child process has been launched with process id: 32475 Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:32475:32475] [20221221:033139:1671593499:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 03:33:14 STEP 552 H= 552:00 +CPU= 40.118 03:33:33 STEP 553 H= 553:00 +CPU= 19.174 03:34:08 STEP 554 H= 554:00 +CPU= 21.774 03:34:40 STEP 555 H= 555:00 +CPU= 31.386 03:35:01 STEP 556 H= 556:00 +CPU= 20.314 03:35:22 STEP 557 H= 557:00 +CPU= 19.695 03:36:10 STEP 558 H= 558:00 +CPU= 32.012 03:36:29 STEP 559 H= 559:00 +CPU= 19.244 03:36:49 STEP 560 H= 560:00 +CPU= 19.866 03:37:56 STEP 561 H= 561:00 +CPU= 31.554 03:38:18 STEP 562 H= 562:00 +CPU= 20.787 03:38:38 STEP 563 H= 563:00 +CPU= 19.508 03:39:32 STEP 564 H= 564:00 +CPU= 35.932 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000564 03:39:51 STEP 565 H= 565:00 +CPU= 19.341 03:40:12 STEP 566 H= 566:00 +CPU= 19.952 03:40:52 STEP 567 H= 567:00 +CPU= 31.701 03:41:11 STEP 568 H= 568:00 +CPU= 19.408 03:41:30 STEP 569 H= 569:00 +CPU= 18.905 03:42:25 STEP 570 H= 570:00 +CPU= 32.046 03:42:45 STEP 571 H= 571:00 +CPU= 19.550 03:43:05 STEP 572 H= 572:00 +CPU= 19.946 03:43:56 STEP 573 H= 573:00 +CPU= 31.224 03:44:16 STEP 574 H= 574:00 +CPU= 19.632 03:44:36 STEP 575 H= 575:00 +CPU= 19.623 03:45:51 STEP 576 H= 576:00 +CPU= 39.250 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000576 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000576 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000564 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000564 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000564 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_23.zip Uploading the intermediate file: upload_file_23.zip 03:46:11 STEP 577 H= 577:00 +CPU= 19.435 03:46:31 STEP 578 H= 578:00 +CPU= 19.788 03:47:09 STEP 579 H= 579:00 +CPU= 31.679 03:47:29 STEP 580 H= 580:00 +CPU= 20.803 03:47:49 STEP 581 H= 581:00 +CPU= 19.090 03:48:20 STEP 582 H= 582:00 +CPU= 30.832 03:48:39 STEP 583 H= 583:00 +CPU= 19.433 03:48:59 STEP 584 H= 584:00 +CPU= 19.890 03:49:33 STEP 585 H= 585:00 +CPU= 31.293 03:50:12 STEP 586 H= 586:00 +CPU= 20.489 03:50:35 STEP 587 H= 587:00 +CPU= 19.900 03:51:41 STEP 588 H= 588:00 +CPU= 38.108 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000588 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000588 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000588 03:52:01 STEP 589 H= 589:00 +CPU= 19.664 03:52:21 STEP 590 H= 590:00 +CPU= 19.791 03:53:12 STEP 591 H= 591:00 +CPU= 32.520 03:53:32 STEP 592 H= 592:00 +CPU= 19.570 03:54:19 STEP 593 H= 593:00 +CPU= 20.275 03:54:55 STEP 594 H= 594:00 +CPU= 31.169 03:55:14 STEP 595 H= 595:00 +CPU= 18.970 03:55:33 STEP 596 H= 596:00 +CPU= 19.130 03:56:11 STEP 597 H= 597:00 +CPU= 30.629 03:57:54 STEP 598 H= 598:00 +CPU= 20.752 03:58:46 STEP 599 H= 599:00 +CPU= 21.543 03:59:30 STEP 600 H= 600:00 +CPU= 36.576 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000600 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000600 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000588 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000588 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000588 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_24.zip Uploading the intermediate file: upload_file_24.zip 04:00:19 STEP 601 H= 601:00 +CPU= 19.777 04:00:38 STEP 602 H= 602:00 +CPU= 19.481 04:01:18 STEP 603 H= 603:00 +CPU= 30.012 04:01:38 STEP 604 H= 604:00 +CPU= 18.829 04:01:56 STEP 605 H= 605:00 +CPU= 18.311 04:03:30 STEP 606 H= 606:00 +CPU= 33.294 04:04:34 STEP 607 H= 607:00 +CPU= 20.572 04:04:59 STEP 608 H= 608:00 +CPU= 19.745 04:05:36 STEP 609 H= 609:00 +CPU= 31.346 04:06:11 STEP 610 H= 610:00 +CPU= 20.456 04:06:30 STEP 611 H= 611:00 +CPU= 18.944 04:07:14 STEP 612 H= 612:00 +CPU= 34.228 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000612 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000612 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000612 04:08:17 STEP 613 H= 613:00 +CPU= 20.794 04:08:39 STEP 614 H= 614:00 +CPU= 19.519 04:09:23 STEP 615 H= 615:00 +CPU= 31.849 04:09:43 STEP 616 H= 616:00 +CPU= 19.539 04:10:02 STEP 617 H= 617:00 +CPU= 18.961 04:10:53 STEP 618 H= 618:00 +CPU= 30.902 04:11:16 STEP 619 H= 619:00 +CPU= 19.309 04:11:36 STEP 620 H= 620:00 +CPU= 19.343 04:13:31 STEP 621 H= 621:00 +CPU= 32.300 04:14:07 STEP 622 H= 622:00 +CPU= 20.385 04:14:27 STEP 623 H= 623:00 +CPU= 19.485 04:15:14 STEP 624 H= 624:00 +CPU= 35.629 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000624 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000624 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000612 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000612 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000612 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_25.zip Uploading the intermediate file: upload_file_25.zip 04:15:32 STEP 625 H= 625:00 +CPU= 17.873 04:15:51 STEP 626 H= 626:00 +CPU= 18.422 04:16:19 STEP 627 H= 627:00 +CPU= 28.482 04:16:39 STEP 628 H= 628:00 +CPU= 19.380 04:16:58 STEP 629 H= 629:00 +CPU= 18.790 04:17:51 STEP 630 H= 630:00 +CPU= 32.086 04:18:23 STEP 631 H= 631:00 +CPU= 19.911 04:18:54 STEP 632 H= 632:00 +CPU= 20.713 04:19:25 STEP 633 H= 633:00 +CPU= 29.995 04:19:45 STEP 634 H= 634:00 +CPU= 19.450 04:20:04 STEP 635 H= 635:00 +CPU= 18.905 04:20:39 STEP 636 H= 636:00 +CPU= 33.466 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000636 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000636 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000636 04:20:58 STEP 637 H= 637:00 +CPU= 18.925 04:21:18 STEP 638 H= 638:00 +CPU= 19.556 04:21:47 STEP 639 H= 639:00 +CPU= 28.635 04:22:06 STEP 640 H= 640:00 +CPU= 18.757 04:22:25 STEP 641 H= 641:00 +CPU= 18.969 04:22:55 STEP 642 H= 642:00 +CPU= 30.335 04:23:14 STEP 643 H= 643:00 +CPU= 18.868 04:23:34 STEP 644 H= 644:00 +CPU= 19.485 04:24:04 STEP 645 H= 645:00 +CPU= 29.880 04:24:24 STEP 646 H= 646:00 +CPU= 19.537 04:24:44 STEP 647 H= 647:00 +CPU= 19.833 04:25:24 STEP 648 H= 648:00 +CPU= 36.526 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000648 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000648 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000636 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000636 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000636 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_26.zip Uploading the intermediate file: upload_file_26.zip 04:25:44 STEP 649 H= 649:00 +CPU= 19.336 04:26:04 STEP 650 H= 650:00 +CPU= 19.758 04:26:34 STEP 651 H= 651:00 +CPU= 29.897 04:26:53 STEP 652 H= 652:00 +CPU= 19.573 04:27:13 STEP 653 H= 653:00 +CPU= 18.965 04:27:43 STEP 654 H= 654:00 +CPU= 30.517 04:28:02 STEP 655 H= 655:00 +CPU= 19.030 04:28:22 STEP 656 H= 656:00 +CPU= 19.529 04:28:53 STEP 657 H= 657:00 +CPU= 30.742 04:29:13 STEP 658 H= 658:00 +CPU= 19.740 04:29:32 STEP 659 H= 659:00 +CPU= 19.011 04:30:07 STEP 660 H= 660:00 +CPU= 34.508 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000660 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000660 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000660 04:30:26 STEP 661 H= 661:00 +CPU= 18.939 04:30:46 STEP 662 H= 662:00 +CPU= 19.489 04:31:16 STEP 663 H= 663:00 +CPU= 29.816 04:31:35 STEP 664 H= 664:00 +CPU= 19.466 04:31:54 STEP 665 H= 665:00 +CPU= 18.887 04:32:25 STEP 666 H= 666:00 +CPU= 30.600 04:32:44 STEP 667 H= 667:00 +CPU= 19.230 04:33:04 STEP 668 H= 668:00 +CPU= 19.500 04:33:33 STEP 669 H= 669:00 +CPU= 29.737 04:33:53 STEP 670 H= 670:00 +CPU= 19.415 04:34:12 STEP 671 H= 671:00 +CPU= 19.497 04:34:48 STEP 672 H= 672:00 +CPU= 35.688 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000672 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000672 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000660 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000660 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000660 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_27.zip Uploading the intermediate file: upload_file_27.zip 04:35:08 STEP 673 H= 673:00 +CPU= 18.980 04:35:28 STEP 674 H= 674:00 +CPU= 19.436 04:35:57 STEP 675 H= 675:00 +CPU= 29.707 04:36:17 STEP 676 H= 676:00 +CPU= 19.413 04:36:36 STEP 677 H= 677:00 +CPU= 18.905 04:37:05 STEP 678 H= 678:00 +CPU= 29.030 04:37:24 STEP 679 H= 679:00 +CPU= 18.854 04:37:43 STEP 680 H= 680:00 +CPU= 19.432 04:38:13 STEP 681 H= 681:00 +CPU= 29.894 04:38:33 STEP 682 H= 682:00 +CPU= 19.545 04:38:52 STEP 683 H= 683:00 +CPU= 18.948 04:39:25 STEP 684 H= 684:00 +CPU= 33.168 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000684 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000684 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000684 04:39:44 STEP 685 H= 685:00 +CPU= 18.894 04:40:03 STEP 686 H= 686:00 +CPU= 19.103 04:40:32 STEP 687 H= 687:00 +CPU= 28.776 04:40:51 STEP 688 H= 688:00 +CPU= 18.810 04:41:09 STEP 689 H= 689:00 +CPU= 18.336 04:41:39 STEP 690 H= 690:00 +CPU= 29.360 04:41:57 STEP 691 H= 691:00 +CPU= 18.470 04:42:17 STEP 692 H= 692:00 +CPU= 19.407 04:42:46 STEP 693 H= 693:00 +CPU= 29.733 04:43:06 STEP 694 H= 694:00 +CPU= 19.436 04:43:25 STEP 695 H= 695:00 +CPU= 19.404 04:44:00 STEP 696 H= 696:00 +CPU= 35.050 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000696 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000696 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000684 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_28.zip Uploading the intermediate file: upload_file_28.zip 04:44:20 STEP 697 H= 697:00 +CPU= 18.838 04:44:39 STEP 698 H= 698:00 +CPU= 19.350 04:45:09 STEP 699 H= 699:00 +CPU= 29.699 04:45:28 STEP 700 H= 700:00 +CPU= 19.358 04:45:47 STEP 701 H= 701:00 +CPU= 18.892 04:46:18 STEP 702 H= 702:00 +CPU= 30.444 04:46:37 STEP 703 H= 703:00 +CPU= 18.942 04:46:56 STEP 704 H= 704:00 +CPU= 19.464 04:47:26 STEP 705 H= 705:00 +CPU= 29.886 04:47:46 STEP 706 H= 706:00 +CPU= 19.516 04:48:05 STEP 707 H= 707:00 +CPU= 18.962 04:48:38 STEP 708 H= 708:00 +CPU= 33.194 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000708 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000708 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000708 04:48:57 STEP 709 H= 709:00 +CPU= 18.974 04:49:17 STEP 710 H= 710:00 +CPU= 19.535 04:49:47 STEP 711 H= 711:00 +CPU= 29.870 04:50:06 STEP 712 H= 712:00 +CPU= 19.561 04:50:25 STEP 713 H= 713:00 +CPU= 18.952 04:50:56 STEP 714 H= 714:00 +CPU= 30.507 04:51:15 STEP 715 H= 715:00 +CPU= 18.826 04:51:33 STEP 716 H= 716:00 +CPU= 18.770 04:52:04 STEP 717 H= 717:00 +CPU= 29.740 04:52:24 STEP 718 H= 718:00 +CPU= 19.543 04:52:44 STEP 719 H= 719:00 +CPU= 19.533 04:53:23 STEP 720 H= 720:00 +CPU= 35.797 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000720 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000720 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000708 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_29.zip Uploading the intermediate file: upload_file_29.zip 04:53:42 STEP 721 H= 721:00 +CPU= 18.847 04:54:01 STEP 722 H= 722:00 +CPU= 19.333 04:54:32 STEP 723 H= 723:00 +CPU= 30.177 04:54:52 STEP 724 H= 724:00 +CPU= 19.731 04:55:11 STEP 725 H= 725:00 +CPU= 19.273 04:55:42 STEP 726 H= 726:00 +CPU= 30.687 04:56:01 STEP 727 H= 727:00 +CPU= 18.989 04:56:20 STEP 728 H= 728:00 +CPU= 19.564 04:56:50 STEP 729 H= 729:00 +CPU= 29.930 04:57:10 STEP 730 H= 730:00 +CPU= 19.588 04:57:29 STEP 731 H= 731:00 +CPU= 19.020 04:58:02 STEP 732 H= 732:00 +CPU= 32.686 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000732 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000732 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000732 04:58:21 STEP 733 H= 733:00 +CPU= 19.018 04:58:41 STEP 734 H= 734:00 +CPU= 19.565 04:59:10 STEP 735 H= 735:00 +CPU= 29.626 04:59:29 STEP 736 H= 736:00 +CPU= 18.848 04:59:48 STEP 737 H= 737:00 +CPU= 18.303 05:00:19 STEP 738 H= 738:00 +CPU= 30.548 05:00:38 STEP 739 H= 739:00 +CPU= 19.021 05:00:58 STEP 740 H= 740:00 +CPU= 19.801 05:01:28 STEP 741 H= 741:00 +CPU= 30.384 05:01:48 STEP 742 H= 742:00 +CPU= 19.558 05:02:07 STEP 743 H= 743:00 +CPU= 19.473 05:02:43 STEP 744 H= 744:00 +CPU= 35.655 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000744 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000744 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000732 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_30.zip Uploading the intermediate file: upload_file_30.zip 05:03:02 STEP 745 H= 745:00 +CPU= 18.853 05:03:22 STEP 746 H= 746:00 +CPU= 19.352 05:03:51 STEP 747 H= 747:00 +CPU= 29.094 05:04:11 STEP 748 H= 748:00 +CPU= 19.401 05:04:30 STEP 749 H= 749:00 +CPU= 18.599 05:04:59 STEP 750 H= 750:00 +CPU= 29.339 05:05:18 STEP 751 H= 751:00 +CPU= 18.367 05:05:36 STEP 752 H= 752:00 +CPU= 18.826 05:06:06 STEP 753 H= 753:00 +CPU= 29.751 05:06:26 STEP 754 H= 754:00 +CPU= 19.381 05:06:45 STEP 755 H= 755:00 +CPU= 18.875 05:07:18 STEP 756 H= 756:00 +CPU= 32.991 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000756 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000756 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000756 05:07:36 STEP 757 H= 757:00 +CPU= 18.491 05:07:55 STEP 758 H= 758:00 +CPU= 18.970 05:08:25 STEP 759 H= 759:00 +CPU= 29.738 05:08:44 STEP 760 H= 760:00 +CPU= 18.954 05:09:03 STEP 761 H= 761:00 +CPU= 18.984 05:09:33 STEP 762 H= 762:00 +CPU= 29.468 05:09:51 STEP 763 H= 763:00 +CPU= 18.634 05:10:11 STEP 764 H= 764:00 +CPU= 19.232 05:10:40 STEP 765 H= 765:00 +CPU= 29.096 05:10:59 STEP 766 H= 766:00 +CPU= 19.470 05:11:19 STEP 767 H= 767:00 +CPU= 18.963 05:11:55 STEP 768 H= 768:00 +CPU= 35.940 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000768 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000768 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000756 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_31.zip Uploading the intermediate file: upload_file_31.zip 05:12:15 STEP 769 H= 769:00 +CPU= 18.735 05:12:34 STEP 770 H= 770:00 +CPU= 19.136 05:13:04 STEP 771 H= 771:00 +CPU= 29.622 05:13:24 STEP 772 H= 772:00 +CPU= 19.155 05:13:43 STEP 773 H= 773:00 +CPU= 19.102 05:14:13 STEP 774 H= 774:00 +CPU= 30.148 05:14:32 STEP 775 H= 775:00 +CPU= 18.634 05:14:52 STEP 776 H= 776:00 +CPU= 19.467 05:15:22 STEP 777 H= 777:00 +CPU= 30.114 05:15:42 STEP 778 H= 778:00 +CPU= 19.571 05:16:01 STEP 779 H= 779:00 +CPU= 19.055 05:16:33 STEP 780 H= 780:00 +CPU= 32.419 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000780 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000780 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000780 05:16:52 STEP 781 H= 781:00 +CPU= 18.568 05:17:11 STEP 782 H= 782:00 +CPU= 18.862 05:17:40 STEP 783 H= 783:00 +CPU= 28.917 05:17:59 STEP 784 H= 784:00 +CPU= 18.936 05:18:17 STEP 785 H= 785:00 +CPU= 17.985 05:18:46 STEP 786 H= 786:00 +CPU= 28.830 05:19:04 STEP 787 H= 787:00 +CPU= 18.556 05:19:24 STEP 788 H= 788:00 +CPU= 19.265 05:19:53 STEP 789 H= 789:00 +CPU= 29.615 05:20:13 STEP 790 H= 790:00 +CPU= 19.365 05:20:32 STEP 791 H= 791:00 +CPU= 18.973 05:21:07 STEP 792 H= 792:00 +CPU= 35.417 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000792 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000792 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000792 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMGGhq0f+000780 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMSHhq0f+000780 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104/ICMUAhq0f+000780 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505_32.zip Uploading the intermediate file: upload_file_32.zip 05:21:27 STEP 793 H= 793:00 +CPU= 18.373 Uploading trickle at timestep: 2851200 05:21:46 STEP 794 H= 794:00 +CPU= 19.502 05:22:16 STEP 795 H= 795:00 +CPU= 29.974 05:22:36 STEP 796 H= 796:00 +CPU= 19.516 05:22:55 STEP 797 H= 797:00 +CPU= 18.943 05:23:25 STEP 798 H= 798:00 +CPU= 30.514 05:23:44 STEP 799 H= 799:00 +CPU= 18.869 05:24:04 STEP 800 H= 800:00 +CPU= 19.472 05:24:34 STEP 801 H= 801:00 +CPU= 29.753 05:24:53 STEP 802 H= 802:00 +CPU= 19.441 05:25:12 STEP 803 H= 803:00 +CPU= 18.988 05:25:45 STEP 804 H= 804:00 +CPU= 33.124 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000804 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000804 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000804 05:26:04 STEP 805 H= 805:00 +CPU= 18.918 05:26:24 STEP 806 H= 806:00 +CPU= 19.477 05:26:54 STEP 807 H= 807:00 +CPU= 29.792 05:27:13 STEP 808 H= 808:00 +CPU= 19.458 05:27:32 STEP 809 H= 809:00 +CPU= 18.890 05:28:02 STEP 810 H= 810:00 +CPU= 30.442 05:28:21 STEP 811 H= 811:00 +CPU= 18.881 05:28:41 STEP 812 H= 812:00 +CPU= 19.404 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1981050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0460 (argv4) batchid: 950 (argv5) wuid: 12167104 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/12 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12167104 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_e705484d46f33894f27d1becd343edb9 to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0460_1981050100_123_950_12167104.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0460_1981050100_123_950_12167104.zip ic_ancil_file: ic_ancil_12167104 ifsdata_file: ifsdata_12167104 climate_data_file: clim_data_12167104 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_68d381a9e1c550f72b731870547dfcad to: /var/lib/boinc-client/slots/12/ic_ancil_12167104.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12167104.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ce85c9d34336b5a82f194b1396b48873 to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12167104.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12167104.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12167104.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12167104.zip Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12167104.xml Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12167104.xml -- Model is restarting -- Adjusting last_iter, 813, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12167104.xml last_cpu_time: 24880 upload_file_number: 33 last_iter: 791 last_upload: 2851200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0460_1981050100_123_950_12167104_0_r1144774505 The child process has been launched with process id: 373 Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:prj-boinc:1:1:373:373] [20221221:053019:1671600619:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 05:31:24 STEP 792 H= 792:00 +CPU= 36.741 05:31:45 STEP 793 H= 793:00 +CPU= 19.808 05:32:04 STEP 794 H= 794:00 +CPU= 18.797 05:32:33 STEP 795 H= 795:00 +CPU= 28.662 05:32:52 STEP 796 H= 796:00 +CPU= 19.275 05:33:21 STEP 797 H= 797:00 +CPU= 19.981 05:33:50 STEP 798 H= 798:00 +CPU= 28.595 05:34:08 STEP 799 H= 799:00 +CPU= 17.724 05:34:26 STEP 800 H= 800:00 +CPU= 18.403 05:34:55 STEP 801 H= 801:00 +CPU= 28.232 05:35:13 STEP 802 H= 802:00 +CPU= 18.459 05:35:31 STEP 803 H= 803:00 +CPU= 17.949 05:36:12 STEP 804 H= 804:00 +CPU= 32.416 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+000804 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+000804 Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+000804 05:36:31 STEP 805 H= 805:00 +CPU= 18.026 05:37:01 STEP 806 H= 806:00 +CPU= 19.677 05:37:29 STEP 807 H= 807:00 +CPU= 28.397 05:37:48 STEP 808 H= 808:00 +CPU= 18.430 05:38:06 STEP 809 H= 809:00 +CPU= 18.007 05:39:59 STEP 810 H= 810:00 +CPU= 32.469 05:40:35 STEP 811 H= 811:00 +CPU= 21.210 05:40:56 STEP 812 H= 812:00 +CPU= 18.758 05:41:25 STEP 813 H= 813:00 +CPU= 28.190 05:41:45 STEP 814 H= 814:00 +CPU= 18.564 05:42:04 STEP 815 H= 815:00 +CPU= 18.338 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 15:45:27 | 1519861 | 22251317 | oifs_43r3_ps_0460_1981050100_123_950_12167104_0 | 2,851,200 | 24,409 | 0.0086 |
©2024 cpdn.org