Name | oifs_43r3_ps_0514_2020050100_123_989_12206158_0 |
Workunit | 12206158 |
Created | 25 Dec 2022, 2:10:48 UTC |
Sent | 13 Jan 2023, 17:08:02 UTC |
Report deadline | 12 Feb 2023, 17:08:02 UTC |
Received | 20 Jan 2023, 18:46:01 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1535622 |
Run time | 15 hours 43 min 11 sec |
CPU time | 15 hours 28 min 19 sec |
Validate state | Invalid |
Credit | 12,700.80 |
Device peak FLOPS | 6.43 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,418.53 MB |
Peak swap size | 4,978.16 MB |
Peak disk usage | 7,674.34 MB |
Stderr | <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> hub/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV [EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS CLDPP T/S (1002) [EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS [EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : POSTPHY_LAYER [EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : POSTPHY [EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CPPHDDHE [EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130741:1674151661:6378.923] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130741:1674151661:6378.923] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown oifs_43r3_model.e 0000000000ECB221 cpphddhe_ 310 cpphddhe.F90 oifs_43r3_model.e 0000000000EC7BB0 postphy_ 1033 postphy.F90 oifs_43r3_model.e 0000000000E9A3FB postphy_layer_ 144 postphy_layer.F90 oifs_43r3_model.e 0000000000E5022E ec_phys_ 945 ec_phys.F90 oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90 oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90 oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90 oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90 oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90 oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90 oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2020050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0514 (argv4) batchid: 989 (argv5) wuid: 12206158 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/6 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_950ee5a026717edb7679c775bdd36e2f to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip ic_ancil_file: ic_ancil_12206158 ifsdata_file: ifsdata_12206158 climate_data_file: clim_data_12206158 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_4ead1188f398400684c6ffb3e68f2703 to: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_63a9cb1954bf64a591cb5b44413d64f7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12206158.xml -- Model is restarting -- Adjusting last_iter, 2607, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml last_cpu_time: 50131 upload_file_number: 108 last_iter: 2591 last_upload: 9331200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037 The child process has been launched with process id: 3143 Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process double free or corruption (top) (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2020050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0514 (argv4) batchid: 989 (argv5) wuid: 12206158 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/6 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_950ee5a026717edb7679c775bdd36e2f to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip ic_ancil_file: ic_ancil_12206158 ifsdata_file: ifsdata_12206158 climate_data_file: clim_data_12206158 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_4ead1188f398400684c6ffb3e68f2703 to: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_63a9cb1954bf64a591cb5b44413d64f7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12206158.xml -- Model is restarting -- Adjusting last_iter, 2592, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml last_cpu_time: 50150 upload_file_number: 108 last_iter: 2591 last_upload: 9331200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037 The child process has been launched with process id: 3068 Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:43:54 STEP 2592 H=2592:00 +CPU= 34.969 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:44:22 STEP 2593 H=2593:00 +CPU= 16.821 08:44:40 STEP 2594 H=2594:00 +CPU= 17.004 08:45:06 STEP 2595 H=2595:00 +CPU= 26.459 08:45:24 STEP 2596 H=2596:00 +CPU= 17.526 08:45:42 STEP 2597 H=2597:00 +CPU= 17.431 08:46:09 STEP 2598 H=2598:00 +CPU= 26.643 08:46:26 STEP 2599 H=2599:00 +CPU= 16.812 08:46:44 STEP 2600 H=2600:00 +CPU= 17.598 08:47:10 STEP 2601 H=2601:00 +CPU= 25.866 08:47:27 STEP 2602 H=2602:00 +CPU= 16.948 08:47:45 STEP 2603 H=2603:00 +CPU= 16.712 08:48:13 STEP 2604 H=2604:00 +CPU= 28.055 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002604 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002604 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002604 08:48:30 STEP 2605 H=2605:00 +CPU= 16.742 08:48:47 STEP 2606 H=2606:00 +CPU= 16.944 08:49:13 STEP 2607 H=2607:00 +CPU= 25.544 08:49:30 STEP 2608 H=2608:00 +CPU= 17.259 08:49:49 STEP 2609 H=2609:00 +CPU= 18.606 08:50:19 STEP 2610 H=2610:00 +CPU= 29.405 08:50:37 STEP 2611 H=2611:00 +CPU= 17.110 08:50:56 STEP 2612 H=2612:00 +CPU= 19.415 08:51:25 STEP 2613 H=2613:00 +CPU= 27.575 08:51:41 STEP 2614 H=2614:00 +CPU= 16.803 08:51:59 STEP 2615 H=2615:00 +CPU= 16.930 08:52:30 STEP 2616 H=2616:00 +CPU= 31.041 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002616 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002616 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002616 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002592 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002592 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002592 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002604 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002604 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002604 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_108.zip Uploading the intermediate file: upload_file_108.zip 08:52:47 STEP 2617 H=2617:00 +CPU= 17.221 08:53:05 STEP 2618 H=2618:00 +CPU= 17.189 08:53:31 STEP 2619 H=2619:00 +CPU= 25.861 08:53:33 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 08:53:47 STEP 2620 H=2620:00 +CPU= 16.225 08:54:05 STEP 2621 H=2621:00 +CPU= 16.969 08:54:31 STEP 2622 H=2622:00 +CPU= 25.688 08:54:47 STEP 2623 H=2623:00 +CPU= 15.690 08:55:04 STEP 2624 H=2624:00 +CPU= 16.764 08:55:29 STEP 2625 H=2625:00 +CPU= 25.542 08:55:47 STEP 2626 H=2626:00 +CPU= 17.417 08:56:03 STEP 2627 H=2627:00 +CPU= 16.103 08:56:32 STEP 2628 H=2628:00 +CPU= 28.467 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002628 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002628 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002628 08:56:48 STEP 2629 H=2629:00 +CPU= 15.995 08:57:04 STEP 2630 H=2630:00 +CPU= 15.941 08:57:32 STEP 2631 H=2631:00 +CPU= 26.707 08:57:51 STEP 2632 H=2632:00 +CPU= 18.890 08:58:10 STEP 2633 H=2633:00 +CPU= 18.168 08:58:39 STEP 2634 H=2634:00 +CPU= 28.278 08:58:58 STEP 2635 H=2635:00 +CPU= 18.139 08:59:17 STEP 2636 H=2636:00 +CPU= 18.523 08:59:44 STEP 2637 H=2637:00 +CPU= 27.363 09:00:03 STEP 2638 H=2638:00 +CPU= 18.339 09:00:22 STEP 2639 H=2639:00 +CPU= 18.459 09:00:58 STEP 2640 H=2640:00 +CPU= 32.372 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002640 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002640 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002640 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002616 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002616 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002616 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002628 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002628 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002628 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_109.zip Uploading the intermediate file: upload_file_109.zip 09:01:17 STEP 2641 H=2641:00 +CPU= 18.251 09:01:35 STEP 2642 H=2642:00 +CPU= 17.976 09:02:00 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 09:02:03 STEP 2643 H=2643:00 +CPU= 27.404 09:02:22 STEP 2644 H=2644:00 +CPU= 18.474 09:02:40 STEP 2645 H=2645:00 +CPU= 17.619 09:03:08 STEP 2646 H=2646:00 +CPU= 27.989 09:03:27 STEP 2647 H=2647:00 +CPU= 17.772 09:03:45 STEP 2648 H=2648:00 +CPU= 18.164 09:04:14 STEP 2649 H=2649:00 +CPU= 28.068 09:04:33 STEP 2650 H=2650:00 +CPU= 18.354 09:04:52 STEP 2651 H=2651:00 +CPU= 18.622 09:05:23 STEP 2652 H=2652:00 +CPU= 30.751 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002652 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002652 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002652 09:05:42 STEP 2653 H=2653:00 +CPU= 18.391 09:06:02 STEP 2654 H=2654:00 +CPU= 19.220 09:06:31 STEP 2655 H=2655:00 +CPU= 28.035 09:06:50 STEP 2656 H=2656:00 +CPU= 18.989 09:07:09 STEP 2657 H=2657:00 +CPU= 18.522 09:07:38 STEP 2658 H=2658:00 +CPU= 28.171 09:07:57 STEP 2659 H=2659:00 +CPU= 18.434 09:08:17 STEP 2660 H=2660:00 +CPU= 19.789 09:08:46 STEP 2661 H=2661:00 +CPU= 27.749 09:09:06 STEP 2662 H=2662:00 +CPU= 19.250 09:09:25 STEP 2663 H=2663:00 +CPU= 18.808 09:09:59 STEP 2664 H=2664:00 +CPU= 33.000 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002664 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002664 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002664 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002640 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002640 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002640 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002652 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002652 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002652 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_110.zip Uploading the intermediate file: upload_file_110.zip 09:10:18 STEP 2665 H=2665:00 +CPU= 18.708 09:10:38 STEP 2666 H=2666:00 +CPU= 19.545 09:11:04 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 09:11:08 STEP 2667 H=2667:00 +CPU= 29.472 09:11:28 STEP 2668 H=2668:00 +CPU= 19.127 09:11:47 STEP 2669 H=2669:00 +CPU= 18.840 09:12:16 STEP 2670 H=2670:00 +CPU= 28.376 09:12:36 STEP 2671 H=2671:00 +CPU= 18.726 09:12:56 STEP 2672 H=2672:00 +CPU= 19.481 09:13:25 STEP 2673 H=2673:00 +CPU= 29.026 09:13:45 STEP 2674 H=2674:00 +CPU= 19.354 09:14:04 STEP 2675 H=2675:00 +CPU= 18.580 09:14:37 STEP 2676 H=2676:00 +CPU= 31.220 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002676 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002676 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002676 09:14:56 STEP 2677 H=2677:00 +CPU= 19.117 09:15:16 STEP 2678 H=2678:00 +CPU= 19.126 09:15:45 STEP 2679 H=2679:00 +CPU= 28.126 09:16:05 STEP 2680 H=2680:00 +CPU= 19.326 09:16:24 STEP 2681 H=2681:00 +CPU= 18.446 09:16:53 STEP 2682 H=2682:00 +CPU= 28.678 09:17:12 STEP 2683 H=2683:00 +CPU= 18.556 09:17:31 STEP 2684 H=2684:00 +CPU= 18.827 09:18:00 STEP 2685 H=2685:00 +CPU= 27.974 09:18:19 STEP 2686 H=2686:00 +CPU= 18.710 09:18:40 STEP 2687 H=2687:00 +CPU= 19.790 09:19:14 STEP 2688 H=2688:00 +CPU= 33.775 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002688 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002688 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002688 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002664 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002664 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002664 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002676 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002676 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002676 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_111.zip Uploading the intermediate file: upload_file_111.zip 09:19:33 STEP 2689 H=2689:00 +CPU= 18.451 09:19:53 STEP 2690 H=2690:00 +CPU= 19.656 09:20:22 STEP 2691 H=2691:00 +CPU= 28.453 09:20:23 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 09:20:42 STEP 2692 H=2692:00 +CPU= 18.965 09:21:01 STEP 2693 H=2693:00 +CPU= 18.700 09:21:31 STEP 2694 H=2694:00 +CPU= 28.710 09:21:50 STEP 2695 H=2695:00 +CPU= 18.605 09:22:10 STEP 2696 H=2696:00 +CPU= 19.547 09:22:39 STEP 2697 H=2697:00 +CPU= 28.497 09:22:59 STEP 2698 H=2698:00 +CPU= 19.110 09:23:18 STEP 2699 H=2699:00 +CPU= 18.849 09:23:51 STEP 2700 H=2700:00 +CPU= 31.460 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002700 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002700 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002700 09:24:10 STEP 2701 H=2701:00 +CPU= 18.516 09:24:29 STEP 2702 H=2702:00 +CPU= 19.059 09:24:59 STEP 2703 H=2703:00 +CPU= 28.492 09:25:19 STEP 2704 H=2704:00 +CPU= 19.201 09:25:38 STEP 2705 H=2705:00 +CPU= 18.813 09:26:07 STEP 2706 H=2706:00 +CPU= 29.004 09:26:26 STEP 2707 H=2707:00 +CPU= 18.423 09:26:45 STEP 2708 H=2708:00 +CPU= 18.602 09:27:14 STEP 2709 H=2709:00 +CPU= 28.299 09:27:33 STEP 2710 H=2710:00 +CPU= 18.714 09:27:53 STEP 2711 H=2711:00 +CPU= 19.138 09:28:28 STEP 2712 H=2712:00 +CPU= 34.138 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002712 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002712 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002712 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002688 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002688 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002688 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002700 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002700 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002700 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_112.zip Uploading the intermediate file: upload_file_112.zip 09:28:47 STEP 2713 H=2713:00 +CPU= 18.112 09:29:07 STEP 2714 H=2714:00 +CPU= 19.331 09:29:34 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 09:29:36 STEP 2715 H=2715:00 +CPU= 28.300 09:29:55 STEP 2716 H=2716:00 +CPU= 18.732 09:30:14 STEP 2717 H=2717:00 +CPU= 18.539 09:30:44 STEP 2718 H=2718:00 +CPU= 29.211 09:31:04 STEP 2719 H=2719:00 +CPU= 18.762 09:31:23 STEP 2720 H=2720:00 +CPU= 19.100 09:31:52 STEP 2721 H=2721:00 +CPU= 27.867 09:32:11 STEP 2722 H=2722:00 +CPU= 19.087 09:32:31 STEP 2723 H=2723:00 +CPU= 19.251 09:33:02 STEP 2724 H=2724:00 +CPU= 30.338 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002724 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002724 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002724 09:33:21 STEP 2725 H=2725:00 +CPU= 18.451 09:33:41 STEP 2726 H=2726:00 +CPU= 19.103 09:34:10 STEP 2727 H=2727:00 +CPU= 28.035 09:34:30 STEP 2728 H=2728:00 +CPU= 19.183 09:34:49 STEP 2729 H=2729:00 +CPU= 18.469 09:35:17 STEP 2730 H=2730:00 +CPU= 27.841 09:35:36 STEP 2731 H=2731:00 +CPU= 18.792 09:35:56 STEP 2732 H=2732:00 +CPU= 19.309 09:36:25 STEP 2733 H=2733:00 +CPU= 28.209 09:36:44 STEP 2734 H=2734:00 +CPU= 18.719 09:37:03 STEP 2735 H=2735:00 +CPU= 18.575 09:37:38 STEP 2736 H=2736:00 +CPU= 33.431 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002736 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002736 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002736 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002712 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002712 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002712 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002724 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002724 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002724 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_113.zip Uploading the intermediate file: upload_file_113.zip 09:37:58 STEP 2737 H=2737:00 +CPU= 18.598 09:38:17 STEP 2738 H=2738:00 +CPU= 19.041 09:38:37 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 09:38:46 STEP 2739 H=2739:00 +CPU= 28.213 09:39:05 STEP 2740 H=2740:00 +CPU= 18.261 09:39:24 STEP 2741 H=2741:00 +CPU= 18.371 09:39:53 STEP 2742 H=2742:00 +CPU= 28.822 09:40:12 STEP 2743 H=2743:00 +CPU= 18.121 09:40:31 STEP 2744 H=2744:00 +CPU= 18.634 09:41:01 STEP 2745 H=2745:00 +CPU= 28.727 09:41:20 STEP 2746 H=2746:00 +CPU= 18.792 09:41:39 STEP 2747 H=2747:00 +CPU= 18.633 09:42:11 STEP 2748 H=2748:00 +CPU= 30.784 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002748 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002748 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002748 09:42:29 STEP 2749 H=2749:00 +CPU= 18.046 09:42:49 STEP 2750 H=2750:00 +CPU= 18.749 09:43:18 STEP 2751 H=2751:00 +CPU= 28.654 09:43:38 STEP 2752 H=2752:00 +CPU= 18.988 09:43:57 STEP 2753 H=2753:00 +CPU= 18.570 09:44:26 STEP 2754 H=2754:00 +CPU= 28.412 09:44:45 STEP 2755 H=2755:00 +CPU= 18.544 09:45:04 STEP 2756 H=2756:00 +CPU= 18.870 09:45:34 STEP 2757 H=2757:00 +CPU= 28.314 09:45:53 STEP 2758 H=2758:00 +CPU= 18.739 09:46:13 STEP 2759 H=2759:00 +CPU= 19.051 09:46:47 STEP 2760 H=2760:00 +CPU= 32.967 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002760 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002760 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002760 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002736 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002736 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002736 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002748 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002748 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002748 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_114.zip Uploading the intermediate file: upload_file_114.zip 09:47:06 STEP 2761 H=2761:00 +CPU= 18.462 09:47:25 STEP 2762 H=2762:00 +CPU= 18.802 09:47:43 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 09:47:54 STEP 2763 H=2763:00 +CPU= 28.099 09:48:14 STEP 2764 H=2764:00 +CPU= 19.127 09:48:32 STEP 2765 H=2765:00 +CPU= 18.088 09:49:01 STEP 2766 H=2766:00 +CPU= 28.712 09:49:20 STEP 2767 H=2767:00 +CPU= 17.992 09:49:39 STEP 2768 H=2768:00 +CPU= 18.513 09:50:08 STEP 2769 H=2769:00 +CPU= 27.977 09:50:27 STEP 2770 H=2770:00 +CPU= 18.684 09:50:45 STEP 2771 H=2771:00 +CPU= 17.961 09:51:17 STEP 2772 H=2772:00 +CPU= 30.874 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002772 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002772 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002772 09:51:35 STEP 2773 H=2773:00 +CPU= 18.022 09:51:55 STEP 2774 H=2774:00 +CPU= 18.628 09:52:24 STEP 2775 H=2775:00 +CPU= 28.273 09:52:43 STEP 2776 H=2776:00 +CPU= 18.979 09:53:02 STEP 2777 H=2777:00 +CPU= 18.305 09:53:31 STEP 2778 H=2778:00 +CPU= 28.758 09:53:50 STEP 2779 H=2779:00 +CPU= 18.218 09:54:10 STEP 2780 H=2780:00 +CPU= 19.176 09:54:39 STEP 2781 H=2781:00 +CPU= 28.751 09:54:59 STEP 2782 H=2782:00 +CPU= 18.678 09:55:18 STEP 2783 H=2783:00 +CPU= 18.923 09:55:53 STEP 2784 H=2784:00 +CPU= 33.717 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002784 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002784 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002784 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002760 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002760 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002760 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002772 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002772 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002772 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_115.zip Uploading the intermediate file: upload_file_115.zip 09:56:11 STEP 2785 H=2785:00 +CPU= 18.216 09:56:31 STEP 2786 H=2786:00 +CPU= 19.052 09:56:58 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 09:56:59 STEP 2787 H=2787:00 +CPU= 27.732 09:57:19 STEP 2788 H=2788:00 +CPU= 18.703 09:57:37 STEP 2789 H=2789:00 +CPU= 18.324 09:58:07 STEP 2790 H=2790:00 +CPU= 28.660 09:58:25 STEP 2791 H=2791:00 +CPU= 17.952 09:58:44 STEP 2792 H=2792:00 +CPU= 18.653 09:59:13 STEP 2793 H=2793:00 +CPU= 28.170 09:59:32 STEP 2794 H=2794:00 +CPU= 18.762 09:59:51 STEP 2795 H=2795:00 +CPU= 18.425 10:00:23 STEP 2796 H=2796:00 +CPU= 31.083 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002796 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002796 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002796 10:00:42 STEP 2797 H=2797:00 +CPU= 18.073 10:01:01 STEP 2798 H=2798:00 +CPU= 18.816 10:01:26 STEP 2799 H=2799:00 +CPU= 24.566 10:01:42 STEP 2800 H=2800:00 +CPU= 15.406 10:01:57 STEP 2801 H=2801:00 +CPU= 15.537 10:02:22 STEP 2802 H=2802:00 +CPU= 24.778 10:02:37 STEP 2803 H=2803:00 +CPU= 15.173 10:02:53 STEP 2804 H=2804:00 +CPU= 15.268 10:03:17 STEP 2805 H=2805:00 +CPU= 24.025 10:03:32 STEP 2806 H=2806:00 +CPU= 15.377 10:03:48 STEP 2807 H=2807:00 +CPU= 16.108 10:04:16 STEP 2808 H=2808:00 +CPU= 27.753 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002808 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002808 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002808 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002784 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002784 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002784 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002796 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002796 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002796 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_116.zip Uploading the intermediate file: upload_file_116.zip 10:04:31 STEP 2809 H=2809:00 +CPU= 14.917 10:04:47 STEP 2810 H=2810:00 +CPU= 16.102 10:05:12 STEP 2811 H=2811:00 +CPU= 24.127 10:05:15 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip 10:05:27 STEP 2812 H=2812:00 +CPU= 15.249 10:05:42 STEP 2813 H=2813:00 +CPU= 15.438 10:06:07 STEP 2814 H=2814:00 +CPU= 24.163 10:06:21 STEP 2815 H=2815:00 +CPU= 14.831 10:06:37 STEP 2816 H=2816:00 +CPU= 15.721 10:07:01 STEP 2817 H=2817:00 +CPU= 23.193 10:07:16 STEP 2818 H=2818:00 +CPU= 15.363 10:07:31 STEP 2819 H=2819:00 +CPU= 15.015 10:07:57 STEP 2820 H=2820:00 +CPU= 25.967 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002820 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002820 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002820 10:08:13 STEP 2821 H=2821:00 +CPU= 15.440 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.660] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2550MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2598 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAMODEL [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-WAMODEL 2 (1431) [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : IMPLSCH [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SNONLIN [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100829:1674227309:5138.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100829:1674227309:5138.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown oifs_43r3_model.e 0000000001A6C650 snonlin_ 271 snonlin.F oifs_43r3_model.e 0000000001A31273 implsch_ 285 implsch.F oifs_43r3_model.e 000000000168CBD7 wamodel_ 1008 wamodel.F oifs_43r3_model.e 0000000000872295 wavemdl_ 584 wavemdl.F oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90 oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90 oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90 oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2020050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0514 (argv4) batchid: 989 (argv5) wuid: 12206158 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/6 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_950ee5a026717edb7679c775bdd36e2f to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip ic_ancil_file: ic_ancil_12206158 ifsdata_file: ifsdata_12206158 climate_data_file: clim_data_12206158 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_4ead1188f398400684c6ffb3e68f2703 to: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_63a9cb1954bf64a591cb5b44413d64f7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12206158.xml -- Model is restarting -- Adjusting last_iter, 2822, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml last_cpu_time: 55130 upload_file_number: 117 last_iter: 2807 last_upload: 10108800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037 The child process has been launched with process id: 2839 Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:36:52 STEP 2808 H=2808:00 +CPU= 32.904 12:37:09 STEP 2809 H=2809:00 +CPU= 16.540 12:37:25 STEP 2810 H=2810:00 +CPU= 16.582 12:37:50 STEP 2811 H=2811:00 +CPU= 24.274 12:38:06 STEP 2812 H=2812:00 +CPU= 15.990 12:38:22 STEP 2813 H=2813:00 +CPU= 16.071 12:38:46 STEP 2814 H=2814:00 +CPU= 24.490 12:39:02 STEP 2815 H=2815:00 +CPU= 15.442 12:39:18 STEP 2816 H=2816:00 +CPU= 15.609 12:39:42 STEP 2817 H=2817:00 +CPU= 24.190 12:39:57 STEP 2818 H=2818:00 +CPU= 15.675 12:40:13 STEP 2819 H=2819:00 +CPU= 15.986 12:40:40 STEP 2820 H=2820:00 +CPU= 26.748 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002820 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002820 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002820 12:40:56 STEP 2821 H=2821:00 +CPU= 15.634 12:41:12 STEP 2822 H=2822:00 +CPU= 15.739 12:41:37 STEP 2823 H=2823:00 +CPU= 24.846 12:41:52 STEP 2824 H=2824:00 +CPU= 15.493 12:42:08 STEP 2825 H=2825:00 +CPU= 15.680 12:42:33 STEP 2826 H=2826:00 +CPU= 24.995 12:42:49 STEP 2827 H=2827:00 +CPU= 15.644 12:43:05 STEP 2828 H=2828:00 +CPU= 15.938 12:43:34 STEP 2829 H=2829:00 +CPU= 28.154 12:43:53 STEP 2830 H=2830:00 +CPU= 19.033 12:44:11 STEP 2831 H=2831:00 +CPU= 17.680 12:44:44 STEP 2832 H=2832:00 +CPU= 31.893 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002832 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002832 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002832 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002808 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002808 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002808 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002820 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002820 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002820 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_117.zip Uploading the intermediate file: upload_file_117.zip 12:45:03 STEP 2833 H=2833:00 +CPU= 17.226 12:45:20 STEP 2834 H=2834:00 +CPU= 16.909 12:45:45 STEP 2835 H=2835:00 +CPU= 25.456 12:45:50 (2323): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip Abort request received from BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:29:00 | 1535622 | 22292744 | oifs_43r3_ps_0514_2020050100_123_989_12206158_0 | 9,072,000 | 48,456 | 0.0053 |
09 May 2023 16:24:36 | 1535622 | 22292744 | oifs_43r3_ps_0514_2020050100_123_989_12206158_0 | 7,516,800 | 40,396 | 0.0054 |
09 May 2023 16:24:30 | 1535622 | 22292744 | oifs_43r3_ps_0514_2020050100_123_989_12206158_0 | 6,652,800 | 35,738 | 0.0054 |
09 May 2023 16:22:57 | 1535622 | 22292744 | oifs_43r3_ps_0514_2020050100_123_989_12206158_0 | 5,356,800 | 28,126 | 0.0053 |
09 May 2023 16:22:52 | 1535622 | 22292744 | oifs_43r3_ps_0514_2020050100_123_989_12206158_0 | 4,492,800 | 23,361 | 0.0052 |
09 May 2023 16:21:30 | 1535622 | 22292744 | oifs_43r3_ps_0514_2020050100_123_989_12206158_0 | 3,369,600 | 17,565 | 0.0052 |
09 May 2023 16:19:11 | 1535622 | 22292744 | oifs_43r3_ps_0514_2020050100_123_989_12206158_0 | 864,000 | 4,346 | 0.0050 |
©2024 cpdn.org