Name | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 |
Workunit | 12174667 |
Created | 21 Dec 2022, 15:27:31 UTC |
Sent | 21 Dec 2022, 17:10:47 UTC |
Report deadline | 20 Jan 2023, 17:10:47 UTC |
Received | 24 Dec 2022, 1:28:53 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1531398 |
Run time | 23 hours 46 min 57 sec |
CPU time | 18 hours 34 min 27 sec |
Validate state | Invalid |
Credit | 13,789.44 |
Device peak FLOPS | 7.99 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,787.32 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 7,178.29 MB |
Stderr | <core_client_version>7.20.2</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> 17 STEP 2485 H=2485:00 +CPU= 23.528 22:15:43 STEP 2486 H=2486:00 +CPU= 23.725 22:16:19 STEP 2487 H=2487:00 +CPU= 34.984 22:16:44 STEP 2488 H=2488:00 +CPU= 23.693 22:17:07 STEP 2489 H=2489:00 +CPU= 22.859 22:17:43 STEP 2490 H=2490:00 +CPU= 34.455 22:18:07 STEP 2491 H=2491:00 +CPU= 23.097 22:18:32 STEP 2492 H=2492:00 +CPU= 24.086 22:19:09 STEP 2493 H=2493:00 +CPU= 34.728 22:19:34 STEP 2494 H=2494:00 +CPU= 24.330 22:20:00 STEP 2495 H=2495:00 +CPU= 24.895 22:20:46 STEP 2496 H=2496:00 +CPU= 41.350 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002496 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002496 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002496 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002472 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002472 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002472 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002484 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002484 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002484 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_82.zip Uploading the intermediate file: upload_file_82.zip 22:21:12 STEP 2497 H=2497:00 +CPU= 22.094 22:21:39 STEP 2498 H=2498:00 +CPU= 23.413 22:22:17 STEP 2499 H=2499:00 +CPU= 32.246 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0023 (argv4) batchid: 958 (argv5) wuid: 12174667 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/21 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_7db76cc0ef87c52e0949750383e2c0fb to: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip Unzipping the namelist zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip ic_ancil_file: ic_ancil_12174667 ifsdata_file: ifsdata_12174667 climate_data_file: clim_data_12174667 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_7d56b2d96fd43574459082e0fe824891 to: /var/lib/boinc/slots/21/ic_ancil_12174667.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/21/ic_ancil_12174667.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3f05a2cabd27cda893615872183ddf12 to: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip Unzipping the climate data zip file: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip Checking for progress XML file: /var/lib/boinc/slots/21/progress_file_12174667.xml Opened progress file ok : /var/lib/boinc/slots/21/progress_file_12174667.xml -- Model is restarting -- Adjusting last_iter, 2500, to previous model restart step. Creating progress file: /var/lib/boinc/slots/21/progress_file_12174667.xml last_cpu_time: 57265 upload_file_number: 83 last_iter: 2495 last_upload: 8985600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911 The child process has been launched with process id: 6332 Executing the command: /var/lib/boinc/slots/21/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:6332:6332] [20221222:232659:1671751619:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0023 (argv4) batchid: 958 (argv5) wuid: 12174667 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/21 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_7db76cc0ef87c52e0949750383e2c0fb to: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip Unzipping the namelist zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip ic_ancil_file: ic_ancil_12174667 ifsdata_file: ifsdata_12174667 climate_data_file: clim_data_12174667 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_7d56b2d96fd43574459082e0fe824891 to: /var/lib/boinc/slots/21/ic_ancil_12174667.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/21/ic_ancil_12174667.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3f05a2cabd27cda893615872183ddf12 to: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip Unzipping the climate data zip file: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip Checking for progress XML file: /var/lib/boinc/slots/21/progress_file_12174667.xml Opened progress file ok : /var/lib/boinc/slots/21/progress_file_12174667.xml -- Model is restarting -- Adjusting last_iter, 2496, to previous model restart step. Creating progress file: /var/lib/boinc/slots/21/progress_file_12174667.xml last_cpu_time: 57273 upload_file_number: 83 last_iter: 2495 last_upload: 8985600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911 The child process has been launched with process id: 7022 Executing the command: /var/lib/boinc/slots/21/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:7022:7022] [20221223:122633:1671798393:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 12:27:20 STEP 2496 H=2496:00 +CPU= 41.051 12:27:43 STEP 2497 H=2497:00 +CPU= 23.202 12:28:02 STEP 2498 H=2498:00 +CPU= 18.403 12:28:30 STEP 2499 H=2499:00 +CPU= 27.736 12:28:52 STEP 2500 H=2500:00 +CPU= 20.312 12:29:14 STEP 2501 H=2501:00 +CPU= 20.336 12:29:47 STEP 2502 H=2502:00 +CPU= 31.494 12:30:06 STEP 2503 H=2503:00 +CPU= 19.552 12:30:28 STEP 2504 H=2504:00 +CPU= 21.196 12:31:00 STEP 2505 H=2505:00 +CPU= 30.987 12:31:23 STEP 2506 H=2506:00 +CPU= 22.508 12:31:46 STEP 2507 H=2507:00 +CPU= 21.610 12:32:33 STEP 2508 H=2508:00 +CPU= 39.900 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002508 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002508 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002508 12:33:00 STEP 2509 H=2509:00 +CPU= 23.901 12:33:32 STEP 2510 H=2510:00 +CPU= 25.415 12:34:13 STEP 2511 H=2511:00 +CPU= 35.850 12:34:44 STEP 2512 H=2512:00 +CPU= 25.809 12:35:11 STEP 2513 H=2513:00 +CPU= 23.852 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:36:09 STEP 2514 H=2514:00 +CPU= 35.505 12:36:42 STEP 2515 H=2515:00 +CPU= 24.262 12:37:15 STEP 2516 H=2516:00 +CPU= 25.048 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:38:35 STEP 2517 H=2517:00 +CPU= 34.355 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:39:31 STEP 2518 H=2518:00 +CPU= 24.560 12:40:04 STEP 2519 H=2519:00 +CPU= 25.991 12:40:51 STEP 2520 H=2520:00 +CPU= 40.589 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002520 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002520 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002520 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002496 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002496 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002496 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002508 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002508 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002508 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_83.zip Uploading the intermediate file: upload_file_83.zip 12:41:17 STEP 2521 H=2521:00 +CPU= 22.433 12:41:47 STEP 2522 H=2522:00 +CPU= 23.619 12:42:26 STEP 2523 H=2523:00 +CPU= 33.358 12:42:56 STEP 2524 H=2524:00 +CPU= 23.651 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:43:35 STEP 2525 H=2525:00 +CPU= 22.086 12:44:18 STEP 2526 H=2526:00 +CPU= 34.846 12:44:47 STEP 2527 H=2527:00 +CPU= 23.839 12:45:16 STEP 2528 H=2528:00 +CPU= 23.839 12:45:57 STEP 2529 H=2529:00 +CPU= 34.135 12:46:27 STEP 2530 H=2530:00 +CPU= 24.178 12:46:53 STEP 2531 H=2531:00 +CPU= 22.459 12:47:30 STEP 2532 H=2532:00 +CPU= 32.480 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002532 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002532 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002532 12:47:54 STEP 2533 H=2533:00 +CPU= 19.704 12:48:22 STEP 2534 H=2534:00 +CPU= 21.399 12:48:57 STEP 2535 H=2535:00 +CPU= 29.170 12:49:21 STEP 2536 H=2536:00 +CPU= 19.893 12:49:44 STEP 2537 H=2537:00 +CPU= 19.335 12:50:24 STEP 2538 H=2538:00 +CPU= 30.973 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:51:17 STEP 2539 H=2539:00 +CPU= 22.693 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:51:56 STEP 2540 H=2540:00 +CPU= 23.463 12:52:38 STEP 2541 H=2541:00 +CPU= 34.556 12:53:07 STEP 2542 H=2542:00 +CPU= 24.133 12:53:38 STEP 2543 H=2543:00 +CPU= 24.807 12:54:28 STEP 2544 H=2544:00 +CPU= 40.471 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002544 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002544 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002544 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002520 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002520 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002520 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002532 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002532 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002532 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_84.zip Uploading the intermediate file: upload_file_84.zip 12:54:58 STEP 2545 H=2545:00 +CPU= 23.902 12:55:27 STEP 2546 H=2546:00 +CPU= 23.799 12:56:09 STEP 2547 H=2547:00 +CPU= 34.164 12:56:38 STEP 2548 H=2548:00 +CPU= 23.936 12:57:07 STEP 2549 H=2549:00 +CPU= 23.478 12:57:49 STEP 2550 H=2550:00 +CPU= 35.137 12:58:18 STEP 2551 H=2551:00 +CPU= 23.509 12:58:46 STEP 2552 H=2552:00 +CPU= 23.644 12:59:24 STEP 2553 H=2553:00 +CPU= 32.999 12:59:50 STEP 2554 H=2554:00 +CPU= 21.693 13:00:15 STEP 2555 H=2555:00 +CPU= 21.035 13:00:59 STEP 2556 H=2556:00 +CPU= 35.991 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002556 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002556 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002556 13:01:26 STEP 2557 H=2557:00 +CPU= 21.774 13:01:48 STEP 2558 H=2558:00 +CPU= 20.370 13:02:24 STEP 2559 H=2559:00 +CPU= 31.326 13:02:51 STEP 2560 H=2560:00 +CPU= 22.068 13:03:12 STEP 2561 H=2561:00 +CPU= 19.562 13:03:51 STEP 2562 H=2562:00 +CPU= 32.501 13:04:18 STEP 2563 H=2563:00 +CPU= 22.484 13:04:46 STEP 2564 H=2564:00 +CPU= 22.678 13:05:25 STEP 2565 H=2565:00 +CPU= 32.869 13:05:53 STEP 2566 H=2566:00 +CPU= 22.926 13:06:21 STEP 2567 H=2567:00 +CPU= 22.872 13:07:08 STEP 2568 H=2568:00 +CPU= 39.038 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002568 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002568 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002568 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002544 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002544 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002544 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002556 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002556 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002556 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_85.zip Uploading the intermediate file: upload_file_85.zip 13:07:35 STEP 2569 H=2569:00 +CPU= 22.266 13:08:02 STEP 2570 H=2570:00 +CPU= 22.303 13:08:42 STEP 2571 H=2571:00 +CPU= 33.092 13:09:09 STEP 2572 H=2572:00 +CPU= 22.518 13:09:35 STEP 2573 H=2573:00 +CPU= 21.892 13:10:12 STEP 2574 H=2574:00 +CPU= 32.170 13:10:37 STEP 2575 H=2575:00 +CPU= 21.423 13:11:02 STEP 2576 H=2576:00 +CPU= 21.224 13:11:38 STEP 2577 H=2577:00 +CPU= 31.262 13:12:05 STEP 2578 H=2578:00 +CPU= 22.561 13:12:32 STEP 2579 H=2579:00 +CPU= 22.036 13:13:15 STEP 2580 H=2580:00 +CPU= 36.650 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002580 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002580 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002580 13:13:40 STEP 2581 H=2581:00 +CPU= 22.006 13:14:05 STEP 2582 H=2582:00 +CPU= 22.210 13:14:44 STEP 2583 H=2583:00 +CPU= 33.124 13:15:11 STEP 2584 H=2584:00 +CPU= 23.393 13:15:38 STEP 2585 H=2585:00 +CPU= 23.350 13:16:19 STEP 2586 H=2586:00 +CPU= 35.109 13:16:46 STEP 2587 H=2587:00 +CPU= 23.302 13:17:13 STEP 2588 H=2588:00 +CPU= 23.187 13:17:52 STEP 2589 H=2589:00 +CPU= 34.016 13:18:21 STEP 2590 H=2590:00 +CPU= 23.734 13:18:50 STEP 2591 H=2591:00 +CPU= 24.450 13:19:37 STEP 2592 H=2592:00 +CPU= 40.114 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002592 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002592 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002592 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002568 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002568 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002568 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002580 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002580 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002580 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_86.zip Uploading the intermediate file: upload_file_86.zip 13:20:05 STEP 2593 H=2593:00 +CPU= 23.981 13:20:35 STEP 2594 H=2594:00 +CPU= 24.658 13:21:19 STEP 2595 H=2595:00 +CPU= 36.595 13:21:48 STEP 2596 H=2596:00 +CPU= 24.123 13:22:15 STEP 2597 H=2597:00 +CPU= 22.666 13:22:57 STEP 2598 H=2598:00 +CPU= 36.503 13:23:27 STEP 2599 H=2599:00 +CPU= 24.605 13:23:57 STEP 2600 H=2600:00 +CPU= 25.490 13:24:38 STEP 2601 H=2601:00 +CPU= 35.688 13:25:09 STEP 2602 H=2602:00 +CPU= 25.711 13:25:38 STEP 2603 H=2603:00 +CPU= 25.052 13:26:27 STEP 2604 H=2604:00 +CPU= 42.470 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002604 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002604 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002604 13:26:59 STEP 2605 H=2605:00 +CPU= 26.140 13:27:33 STEP 2606 H=2606:00 +CPU= 27.057 13:28:23 STEP 2607 H=2607:00 +CPU= 40.540 13:28:57 STEP 2608 H=2608:00 +CPU= 27.784 13:29:32 STEP 2609 H=2609:00 +CPU= 27.812 13:30:24 STEP 2610 H=2610:00 +CPU= 42.238 13:30:56 STEP 2611 H=2611:00 +CPU= 26.468 13:31:32 STEP 2612 H=2612:00 +CPU= 28.366 13:32:22 STEP 2613 H=2613:00 +CPU= 41.596 13:32:55 STEP 2614 H=2614:00 +CPU= 27.654 13:33:30 STEP 2615 H=2615:00 +CPU= 28.905 13:34:29 STEP 2616 H=2616:00 +CPU= 48.878 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002616 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002616 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002616 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002592 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002592 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002592 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002604 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002604 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002604 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_87.zip Uploading the intermediate file: upload_file_87.zip 13:35:02 STEP 2617 H=2617:00 +CPU= 27.376 13:35:36 STEP 2618 H=2618:00 +CPU= 28.560 13:36:28 STEP 2619 H=2619:00 +CPU= 42.049 13:37:02 STEP 2620 H=2620:00 +CPU= 28.265 13:37:36 STEP 2621 H=2621:00 +CPU= 27.904 13:38:23 STEP 2622 H=2622:00 +CPU= 40.405 13:38:56 STEP 2623 H=2623:00 +CPU= 26.869 13:39:30 STEP 2624 H=2624:00 +CPU= 27.548 13:40:21 STEP 2625 H=2625:00 +CPU= 41.759 13:40:55 STEP 2626 H=2626:00 +CPU= 27.639 13:41:29 STEP 2627 H=2627:00 +CPU= 28.203 13:42:26 STEP 2628 H=2628:00 +CPU= 46.036 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002628 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002628 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002628 13:42:59 STEP 2629 H=2629:00 +CPU= 27.942 13:43:35 STEP 2630 H=2630:00 +CPU= 29.219 13:44:25 STEP 2631 H=2631:00 +CPU= 42.094 13:44:59 STEP 2632 H=2632:00 +CPU= 28.546 13:45:34 STEP 2633 H=2633:00 +CPU= 28.433 13:46:25 STEP 2634 H=2634:00 +CPU= 41.865 13:46:58 STEP 2635 H=2635:00 +CPU= 27.570 13:47:33 STEP 2636 H=2636:00 +CPU= 28.789 13:48:23 STEP 2637 H=2637:00 +CPU= 41.959 13:48:57 STEP 2638 H=2638:00 +CPU= 28.099 13:49:33 STEP 2639 H=2639:00 +CPU= 29.806 13:50:31 STEP 2640 H=2640:00 +CPU= 47.899 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002640 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002640 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002640 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002616 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002616 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002616 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002628 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002628 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002628 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_88.zip Uploading the intermediate file: upload_file_88.zip 13:51:05 STEP 2641 H=2641:00 +CPU= 28.215 13:51:38 STEP 2642 H=2642:00 +CPU= 28.560 13:52:27 STEP 2643 H=2643:00 +CPU= 40.906 13:53:01 STEP 2644 H=2644:00 +CPU= 28.233 13:53:34 STEP 2645 H=2645:00 +CPU= 27.930 13:54:22 STEP 2646 H=2646:00 +CPU= 40.209 13:54:55 STEP 2647 H=2647:00 +CPU= 27.990 13:55:29 STEP 2648 H=2648:00 +CPU= 28.281 13:56:18 STEP 2649 H=2649:00 +CPU= 40.781 13:56:52 STEP 2650 H=2650:00 +CPU= 28.557 13:57:25 STEP 2651 H=2651:00 +CPU= 27.917 13:58:19 STEP 2652 H=2652:00 +CPU= 44.302 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002652 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002652 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002652 13:58:48 STEP 2653 H=2653:00 +CPU= 25.403 13:59:18 STEP 2654 H=2654:00 +CPU= 25.654 14:00:01 STEP 2655 H=2655:00 +CPU= 36.948 14:00:31 STEP 2656 H=2656:00 +CPU= 26.100 14:01:00 STEP 2657 H=2657:00 +CPU= 24.700 14:01:42 STEP 2658 H=2658:00 +CPU= 36.845 14:02:10 STEP 2659 H=2659:00 +CPU= 24.269 14:02:41 STEP 2660 H=2660:00 +CPU= 25.160 14:03:24 STEP 2661 H=2661:00 +CPU= 36.788 14:03:54 STEP 2662 H=2662:00 +CPU= 25.140 14:04:26 STEP 2663 H=2663:00 +CPU= 26.201 14:05:17 STEP 2664 H=2664:00 +CPU= 42.526 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002664 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002664 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002664 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002640 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002640 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002640 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002652 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002652 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002652 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_89.zip Uploading the intermediate file: upload_file_89.zip 14:05:47 STEP 2665 H=2665:00 +CPU= 24.690 14:06:16 STEP 2666 H=2666:00 +CPU= 24.886 14:07:00 STEP 2667 H=2667:00 +CPU= 36.952 14:07:29 STEP 2668 H=2668:00 +CPU= 25.491 14:07:58 STEP 2669 H=2669:00 +CPU= 24.604 14:08:43 STEP 2670 H=2670:00 +CPU= 37.521 14:09:12 STEP 2671 H=2671:00 +CPU= 24.683 14:09:41 STEP 2672 H=2672:00 +CPU= 25.000 14:10:26 STEP 2673 H=2673:00 +CPU= 36.840 14:10:56 STEP 2674 H=2674:00 +CPU= 25.128 14:11:25 STEP 2675 H=2675:00 +CPU= 24.349 14:12:14 STEP 2676 H=2676:00 +CPU= 41.158 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002676 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002676 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002676 14:12:43 STEP 2677 H=2677:00 +CPU= 24.392 14:13:13 STEP 2678 H=2678:00 +CPU= 25.091 14:13:56 STEP 2679 H=2679:00 +CPU= 36.834 14:14:26 STEP 2680 H=2680:00 +CPU= 25.325 14:14:54 STEP 2681 H=2681:00 +CPU= 24.545 14:15:35 STEP 2682 H=2682:00 +CPU= 37.146 14:16:03 STEP 2683 H=2683:00 +CPU= 24.641 14:16:31 STEP 2684 H=2684:00 +CPU= 25.129 14:17:12 STEP 2685 H=2685:00 +CPU= 36.785 14:17:39 STEP 2686 H=2686:00 +CPU= 24.698 14:18:08 STEP 2687 H=2687:00 +CPU= 26.081 14:18:56 STEP 2688 H=2688:00 +CPU= 42.582 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002688 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002688 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002688 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002664 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002664 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002664 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002676 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002676 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002676 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_90.zip Uploading the intermediate file: upload_file_90.zip 14:19:25 STEP 2689 H=2689:00 +CPU= 24.482 14:19:52 STEP 2690 H=2690:00 +CPU= 24.862 14:20:29 STEP 2691 H=2691:00 +CPU= 35.086 14:20:55 STEP 2692 H=2692:00 +CPU= 24.607 14:21:20 STEP 2693 H=2693:00 +CPU= 23.965 14:21:59 STEP 2694 H=2694:00 +CPU= 36.462 14:22:23 STEP 2695 H=2695:00 +CPU= 23.513 14:22:50 STEP 2696 H=2696:00 +CPU= 24.927 14:23:32 STEP 2697 H=2697:00 +CPU= 36.814 14:24:02 STEP 2698 H=2698:00 +CPU= 24.979 14:24:30 STEP 2699 H=2699:00 +CPU= 24.709 14:25:15 STEP 2700 H=2700:00 +CPU= 40.254 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002700 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002700 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002700 14:25:44 STEP 2701 H=2701:00 +CPU= 24.724 14:26:12 STEP 2702 H=2702:00 +CPU= 24.668 14:26:54 STEP 2703 H=2703:00 +CPU= 36.705 14:27:23 STEP 2704 H=2704:00 +CPU= 25.044 14:27:52 STEP 2705 H=2705:00 +CPU= 24.499 14:28:36 STEP 2706 H=2706:00 +CPU= 37.586 14:29:06 STEP 2707 H=2707:00 +CPU= 24.778 14:29:37 STEP 2708 H=2708:00 +CPU= 25.082 14:30:21 STEP 2709 H=2709:00 +CPU= 36.807 14:30:51 STEP 2710 H=2710:00 +CPU= 25.230 14:31:23 STEP 2711 H=2711:00 +CPU= 26.030 14:32:15 STEP 2712 H=2712:00 +CPU= 42.003 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002712 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002712 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002712 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002688 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002688 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002688 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002700 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002700 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002700 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_91.zip Uploading the intermediate file: upload_file_91.zip 14:32:44 STEP 2713 H=2713:00 +CPU= 24.205 14:33:14 STEP 2714 H=2714:00 +CPU= 24.592 14:33:59 STEP 2715 H=2715:00 +CPU= 37.015 14:34:29 STEP 2716 H=2716:00 +CPU= 25.256 14:34:58 STEP 2717 H=2717:00 +CPU= 24.073 14:35:44 STEP 2718 H=2718:00 +CPU= 37.629 14:36:13 STEP 2719 H=2719:00 +CPU= 24.217 14:36:43 STEP 2720 H=2720:00 +CPU= 24.868 14:37:28 STEP 2721 H=2721:00 +CPU= 37.069 14:37:58 STEP 2722 H=2722:00 +CPU= 25.037 14:38:26 STEP 2723 H=2723:00 +CPU= 23.890 14:39:07 STEP 2724 H=2724:00 +CPU= 39.010 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002724 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002724 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002724 14:39:33 STEP 2725 H=2725:00 +CPU= 23.664 14:40:00 STEP 2726 H=2726:00 +CPU= 23.830 14:40:41 STEP 2727 H=2727:00 +CPU= 36.414 14:41:10 STEP 2728 H=2728:00 +CPU= 24.630 14:41:38 STEP 2729 H=2729:00 +CPU= 23.345 14:42:26 STEP 2730 H=2730:00 +CPU= 39.067 14:42:58 STEP 2731 H=2731:00 +CPU= 25.624 14:43:32 STEP 2732 H=2732:00 +CPU= 26.394 14:44:22 STEP 2733 H=2733:00 +CPU= 39.934 14:44:56 STEP 2734 H=2734:00 +CPU= 26.615 14:45:29 STEP 2735 H=2735:00 +CPU= 26.416 14:46:28 STEP 2736 H=2736:00 +CPU= 46.476 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002736 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002736 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002736 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002712 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002712 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002712 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002724 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002724 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002724 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_92.zip Uploading the intermediate file: upload_file_92.zip Uploading trickle at timestep: 9849600 14:47:02 STEP 2737 H=2737:00 +CPU= 26.135 14:47:35 STEP 2738 H=2738:00 +CPU= 26.225 14:48:21 STEP 2739 H=2739:00 +CPU= 37.698 14:48:51 STEP 2740 H=2740:00 +CPU= 24.732 14:49:19 STEP 2741 H=2741:00 +CPU= 23.277 14:50:05 STEP 2742 H=2742:00 +CPU= 38.031 14:50:34 STEP 2743 H=2743:00 +CPU= 23.538 14:51:03 STEP 2744 H=2744:00 +CPU= 24.087 14:51:49 STEP 2745 H=2745:00 +CPU= 37.602 14:52:19 STEP 2746 H=2746:00 +CPU= 24.644 14:52:47 STEP 2747 H=2747:00 +CPU= 23.766 14:53:41 STEP 2748 H=2748:00 +CPU= 41.656 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002748 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002748 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002748 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0023 (argv4) batchid: 958 (argv5) wuid: 12174667 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/21 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_7db76cc0ef87c52e0949750383e2c0fb to: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip Unzipping the namelist zip file: /var/lib/boinc/slots/21/oifs_43r3_ps_0023_1989050100_123_958_12174667.zip ic_ancil_file: ic_ancil_12174667 ifsdata_file: ifsdata_12174667 climate_data_file: clim_data_12174667 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_7d56b2d96fd43574459082e0fe824891 to: /var/lib/boinc/slots/21/ic_ancil_12174667.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/21/ic_ancil_12174667.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3f05a2cabd27cda893615872183ddf12 to: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/21/ifsdata/ifsdata_12174667.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip Unzipping the climate data zip file: /var/lib/boinc/slots/21/159l_2/clim_data_12174667.zip Checking for progress XML file: /var/lib/boinc/slots/21/progress_file_12174667.xml Opened progress file ok : /var/lib/boinc/slots/21/progress_file_12174667.xml -- Model is restarting -- Adjusting last_iter, 2749, to previous model restart step. Creating progress file: /var/lib/boinc/slots/21/progress_file_12174667.xml last_cpu_time: 64619 upload_file_number: 93 last_iter: 2735 last_upload: 9849600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911 The child process has been launched with process id: 3784 Executing the command: /var/lib/boinc/slots/21/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:tweenis:1:1:3784:3784] [20221223:183811:1671820691:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 18:38:57 STEP 2736 H=2736:00 +CPU= 38.864 18:39:22 STEP 2737 H=2737:00 +CPU= 23.793 18:39:49 STEP 2738 H=2738:00 +CPU= 24.987 18:40:25 STEP 2739 H=2739:00 +CPU= 34.536 18:40:52 STEP 2740 H=2740:00 +CPU= 24.925 18:41:17 STEP 2741 H=2741:00 +CPU= 23.780 18:41:54 STEP 2742 H=2742:00 +CPU= 34.368 18:42:19 STEP 2743 H=2743:00 +CPU= 24.160 18:42:45 STEP 2744 H=2744:00 +CPU= 24.569 18:43:22 STEP 2745 H=2745:00 +CPU= 34.559 18:43:48 STEP 2746 H=2746:00 +CPU= 24.824 18:44:13 STEP 2747 H=2747:00 +CPU= 23.654 18:44:54 STEP 2748 H=2748:00 +CPU= 38.573 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002748 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002748 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002748 18:45:20 STEP 2749 H=2749:00 +CPU= 24.599 18:45:46 STEP 2750 H=2750:00 +CPU= 25.032 18:46:21 STEP 2751 H=2751:00 +CPU= 33.317 18:46:47 STEP 2752 H=2752:00 +CPU= 24.757 18:47:12 STEP 2753 H=2753:00 +CPU= 23.933 18:47:49 STEP 2754 H=2754:00 +CPU= 34.599 18:48:14 STEP 2755 H=2755:00 +CPU= 24.005 18:48:40 STEP 2756 H=2756:00 +CPU= 24.065 18:49:15 STEP 2757 H=2757:00 +CPU= 33.859 18:49:41 STEP 2758 H=2758:00 +CPU= 24.362 18:50:07 STEP 2759 H=2759:00 +CPU= 24.864 18:50:50 STEP 2760 H=2760:00 +CPU= 40.557 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002760 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002760 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002760 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002736 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002736 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002736 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002748 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002748 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002748 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_93.zip Uploading the intermediate file: upload_file_93.zip 18:51:16 STEP 2761 H=2761:00 +CPU= 24.196 18:51:42 STEP 2762 H=2762:00 +CPU= 23.963 18:52:12 STEP 2763 H=2763:00 +CPU= 29.198 18:52:28 STEP 2764 H=2764:00 +CPU= 15.549 18:52:44 STEP 2765 H=2765:00 +CPU= 15.235 18:53:11 STEP 2766 H=2766:00 +CPU= 25.961 18:53:29 STEP 2767 H=2767:00 +CPU= 17.364 18:53:47 STEP 2768 H=2768:00 +CPU= 17.682 18:54:14 STEP 2769 H=2769:00 +CPU= 25.740 18:54:33 STEP 2770 H=2770:00 +CPU= 18.133 18:54:51 STEP 2771 H=2771:00 +CPU= 17.403 18:55:20 STEP 2772 H=2772:00 +CPU= 28.260 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002772 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002772 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002772 18:55:38 STEP 2773 H=2773:00 +CPU= 17.340 18:55:56 STEP 2774 H=2774:00 +CPU= 17.594 18:56:23 STEP 2775 H=2775:00 +CPU= 25.686 18:56:41 STEP 2776 H=2776:00 +CPU= 17.757 18:56:59 STEP 2777 H=2777:00 +CPU= 17.230 18:57:26 STEP 2778 H=2778:00 +CPU= 26.181 18:57:44 STEP 2779 H=2779:00 +CPU= 17.171 18:58:02 STEP 2780 H=2780:00 +CPU= 17.690 18:58:29 STEP 2781 H=2781:00 +CPU= 25.767 18:58:48 STEP 2782 H=2782:00 +CPU= 17.997 18:59:07 STEP 2783 H=2783:00 +CPU= 18.159 18:59:39 STEP 2784 H=2784:00 +CPU= 30.389 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002784 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002784 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002784 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002760 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002760 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002760 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002772 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002772 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002772 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_94.zip Uploading the intermediate file: upload_file_94.zip 18:59:56 STEP 2785 H=2785:00 +CPU= 16.941 19:00:15 STEP 2786 H=2786:00 +CPU= 17.899 19:00:42 STEP 2787 H=2787:00 +CPU= 26.040 19:01:02 STEP 2788 H=2788:00 +CPU= 18.374 19:01:20 STEP 2789 H=2789:00 +CPU= 17.504 19:01:47 STEP 2790 H=2790:00 +CPU= 26.541 19:02:06 STEP 2791 H=2791:00 +CPU= 17.712 19:02:24 STEP 2792 H=2792:00 +CPU= 17.402 19:02:51 STEP 2793 H=2793:00 +CPU= 25.621 19:03:09 STEP 2794 H=2794:00 +CPU= 17.593 19:03:27 STEP 2795 H=2795:00 +CPU= 17.290 19:03:57 STEP 2796 H=2796:00 +CPU= 28.872 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002796 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002796 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002796 19:04:15 STEP 2797 H=2797:00 +CPU= 17.594 19:04:34 STEP 2798 H=2798:00 +CPU= 17.729 19:05:01 STEP 2799 H=2799:00 +CPU= 25.866 19:05:19 STEP 2800 H=2800:00 +CPU= 17.699 19:05:37 STEP 2801 H=2801:00 +CPU= 16.941 19:06:05 STEP 2802 H=2802:00 +CPU= 26.693 19:06:23 STEP 2803 H=2803:00 +CPU= 17.268 19:06:41 STEP 2804 H=2804:00 +CPU= 17.571 19:07:08 STEP 2805 H=2805:00 +CPU= 26.049 19:07:27 STEP 2806 H=2806:00 +CPU= 17.924 19:07:45 STEP 2807 H=2807:00 +CPU= 18.020 19:08:17 STEP 2808 H=2808:00 +CPU= 30.505 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002808 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002808 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002808 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002784 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002784 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002784 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002796 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002796 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002796 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_95.zip Uploading the intermediate file: upload_file_95.zip 19:08:36 STEP 2809 H=2809:00 +CPU= 17.297 19:08:54 STEP 2810 H=2810:00 +CPU= 18.046 19:09:21 STEP 2811 H=2811:00 +CPU= 25.971 19:09:40 STEP 2812 H=2812:00 +CPU= 18.266 19:09:59 STEP 2813 H=2813:00 +CPU= 17.484 19:10:26 STEP 2814 H=2814:00 +CPU= 26.765 19:10:45 STEP 2815 H=2815:00 +CPU= 17.518 19:11:03 STEP 2816 H=2816:00 +CPU= 17.428 19:11:30 STEP 2817 H=2817:00 +CPU= 26.300 19:11:49 STEP 2818 H=2818:00 +CPU= 18.106 19:12:07 STEP 2819 H=2819:00 +CPU= 17.345 19:12:37 STEP 2820 H=2820:00 +CPU= 29.089 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002820 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002820 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002820 19:12:55 STEP 2821 H=2821:00 +CPU= 17.263 19:13:14 STEP 2822 H=2822:00 +CPU= 17.487 19:13:41 STEP 2823 H=2823:00 +CPU= 26.040 19:13:59 STEP 2824 H=2824:00 +CPU= 18.252 19:14:17 STEP 2825 H=2825:00 +CPU= 17.224 19:14:45 STEP 2826 H=2826:00 +CPU= 26.693 19:15:03 STEP 2827 H=2827:00 +CPU= 17.532 19:15:22 STEP 2828 H=2828:00 +CPU= 17.815 19:15:49 STEP 2829 H=2829:00 +CPU= 25.845 19:16:07 STEP 2830 H=2830:00 +CPU= 17.907 19:16:26 STEP 2831 H=2831:00 +CPU= 18.088 19:16:58 STEP 2832 H=2832:00 +CPU= 30.580 Moving to projects directory: /var/lib/boinc/slots/21/ICMGGhq0f+002832 Moving to projects directory: /var/lib/boinc/slots/21/ICMSHhq0f+002832 Moving to projects directory: /var/lib/boinc/slots/21/ICMUAhq0f+002832 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002808 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002808 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002808 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMGGhq0f+002820 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMSHhq0f+002820 Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12174667/ICMUAhq0f+002820 Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0023_1989050100_123_958_12174667_0_r1844748911_96.zip Uploading the intermediate file: upload_file_96.zip 19:17:16 STEP 2833 H=2833:00 +CPU= 17.132 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 15:50:44 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 9,849,600 | 64,260 | 0.0065 |
09 May 2023 15:48:29 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 8,294,400 | 50,659 | 0.0061 |
09 May 2023 15:47:46 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 6,220,800 | 32,584 | 0.0052 |
09 May 2023 15:47:35 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 5,356,800 | 26,241 | 0.0049 |
09 May 2023 15:47:24 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 4,492,800 | 19,902 | 0.0044 |
09 May 2023 15:47:13 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 3,628,800 | 13,576 | 0.0037 |
09 May 2023 15:47:07 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 2,764,800 | 7,198 | 0.0026 |
09 May 2023 15:46:35 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 1,728,000 | 12,020 | 0.0070 |
09 May 2023 15:46:29 | 1531398 | 22259336 | oifs_43r3_ps_0023_1989050100_123_958_12174667_0 | 864,000 | 6,605 | 0.0076 |
©2024 cpdn.org