Name | oifs_43r3_ps_0931_1997050100_123_966_12183575_0 |
Workunit | 12183575 |
Created | 22 Dec 2022, 14:52:21 UTC |
Sent | 17 Jan 2023, 15:41:42 UTC |
Report deadline | 18 Mar 2023, 15:41:42 UTC |
Received | 9 Feb 2023, 10:10:28 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1507457 |
Run time | 8 hours 54 min 41 sec |
CPU time | 7 hours 49 min 32 sec |
Validate state | Invalid |
Credit | 3,386.88 |
Device peak FLOPS | 5.95 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,755.91 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,688.64 MB |
Stderr | <core_client_version>7.16.16</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> ns@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:133454:1675600494:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:48:11 STEP 1032 H=1032:00 +CPU= 28.883 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:49:43 STEP 1033 H=1033:00 +CPU= 16.700 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:16:39 STEP 1034 H=1034:00 +CPU= 16.382 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:21:10 STEP 1035 H=1035:00 +CPU= 25.481 Suspend request received from the BOINC client, suspending the child process Resuming the child process double free or corruption (top) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.651] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3993MB (heap), 4574MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.673] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.673] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.673] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.673] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.673] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3993 MB (maxheap), 4574 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.680] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS (1001) [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CLOUD_LAYER [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142230:1675603350:2855.690] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CLOUDSC [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142231:1675603351:2856.690] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:10552:10552] [20230205:142231:1675603351:2856.690] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown oifs_43r3_model.e 000000000101CBD8 cloudsc_ 850 cloudsc.F90 oifs_43r3_model.e 0000000001014D57 cloud_layer_ 131 cloud_layer.F90 oifs_43r3_model.e 0000000000F10C1B callpar_ 1011 callpar.F90 oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90 oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90 oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90 oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90 oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90 oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90 oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90 oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1997050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0931 (argv4) batchid: 966 (argv5) wuid: 12183575 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /media/nathan/512gb/boinc/slots/0 Project directory is: /media/nathan/512gb/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575 ..mkdir for temp folder for results failed Copying: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_300fd9cb50135bf5cb5b996106334d6a to: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_0931_1997050100_123_966_12183575.zip Unzipping the namelist zip file: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_0931_1997050100_123_966_12183575.zip ic_ancil_file: ic_ancil_12183575 ifsdata_file: ifsdata_12183575 climate_data_file: clim_data_12183575 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3c55903957ebcdaf3951b015145fa45 to: /media/nathan/512gb/boinc/slots/0/ic_ancil_12183575.zip Unzipping the IC ancils zip file: /media/nathan/512gb/boinc/slots/0/ic_ancil_12183575.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cda9331367b0b08951a627372fac3d6a to: /media/nathan/512gb/boinc/slots/0/ifsdata/ifsdata_12183575.zip Unzipping the ifsdata_zip file: /media/nathan/512gb/boinc/slots/0/ifsdata/ifsdata_12183575.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /media/nathan/512gb/boinc/slots/0/159l_2/clim_data_12183575.zip Unzipping the climate data zip file: /media/nathan/512gb/boinc/slots/0/159l_2/clim_data_12183575.zip Checking for progress XML file: /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml Opened progress file ok : /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml -- Model is restarting -- Adjusting last_iter, 1035, to previous model restart step. Creating progress file: /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml last_cpu_time: 25772 upload_file_number: 43 last_iter: 1031 last_upload: 3715200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0931_1997050100_123_966_12183575_0_r544234623 The child process has been launched with process id: 1952 Executing the command: /media/nathan/512gb/boinc/slots/0/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:225209:1675633929:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 22:52:39 STEP 1032 H=1032:00 +CPU= 23.335 22:52:52 STEP 1033 H=1033:00 +CPU= 13.097 22:53:06 STEP 1034 H=1034:00 +CPU= 13.541 22:53:26 STEP 1035 H=1035:00 +CPU= 20.919 22:53:40 STEP 1036 H=1036:00 +CPU= 13.542 22:53:53 STEP 1037 H=1037:00 +CPU= 13.118 22:54:14 STEP 1038 H=1038:00 +CPU= 21.394 22:54:27 STEP 1039 H=1039:00 +CPU= 13.123 22:54:41 STEP 1040 H=1040:00 +CPU= 13.551 22:55:02 STEP 1041 H=1041:00 +CPU= 20.927 22:55:15 STEP 1042 H=1042:00 +CPU= 13.567 22:55:28 STEP 1043 H=1043:00 +CPU= 13.138 22:55:52 STEP 1044 H=1044:00 +CPU= 23.188 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001044 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001044 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001044 22:56:05 STEP 1045 H=1045:00 +CPU= 13.149 22:56:18 STEP 1046 H=1046:00 +CPU= 13.571 22:56:39 STEP 1047 H=1047:00 +CPU= 20.939 22:56:53 STEP 1048 H=1048:00 +CPU= 13.563 22:57:06 STEP 1049 H=1049:00 +CPU= 13.230 22:57:28 STEP 1050 H=1050:00 +CPU= 21.430 22:57:41 STEP 1051 H=1051:00 +CPU= 13.119 22:57:54 STEP 1052 H=1052:00 +CPU= 13.545 22:58:15 STEP 1053 H=1053:00 +CPU= 20.944 22:58:29 STEP 1054 H=1054:00 +CPU= 13.542 22:58:42 STEP 1055 H=1055:00 +CPU= 13.499 22:59:07 STEP 1056 H=1056:00 +CPU= 24.295 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001056 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001056 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001056 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001032 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001032 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001032 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001044 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001044 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001044 Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0931_1997050100_123_966_12183575_0_r544234623_43.zip Uploading the intermediate file: upload_file_43.zip 22:59:20 STEP 1057 H=1057:00 +CPU= 13.117 22:59:33 STEP 1058 H=1058:00 +CPU= 13.528 22:59:54 STEP 1059 H=1059:00 +CPU= 20.945 23:00:08 STEP 1060 H=1060:00 +CPU= 13.544 23:00:21 STEP 1061 H=1061:00 +CPU= 13.120 23:00:42 STEP 1062 H=1062:00 +CPU= 21.438 23:00:55 STEP 1063 H=1063:00 +CPU= 13.139 23:01:09 STEP 1064 H=1064:00 +CPU= 13.568 23:01:30 STEP 1065 H=1065:00 +CPU= 20.966 23:01:44 STEP 1066 H=1066:00 +CPU= 13.568 23:01:57 STEP 1067 H=1067:00 +CPU= 13.135 23:02:20 STEP 1068 H=1068:00 +CPU= 23.233 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001068 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001068 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001068 23:02:33 STEP 1069 H=1069:00 +CPU= 13.155 23:02:47 STEP 1070 H=1070:00 +CPU= 13.580 23:03:08 STEP 1071 H=1071:00 +CPU= 21.055 23:03:25 STEP 1072 H=1072:00 +CPU= 14.705 23:03:39 STEP 1073 H=1073:00 +CPU= 13.819 23:04:02 STEP 1074 H=1074:00 +CPU= 22.442 23:04:16 STEP 1075 H=1075:00 +CPU= 13.541 23:04:30 STEP 1076 H=1076:00 +CPU= 14.064 23:04:54 STEP 1077 H=1077:00 +CPU= 22.210 23:05:09 STEP 1078 H=1078:00 +CPU= 14.279 23:05:24 STEP 1079 H=1079:00 +CPU= 14.363 23:06:02 STEP 1080 H=1080:00 +CPU= 27.024 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001080 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001080 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001080 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001056 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001056 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001056 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001068 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001068 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001068 Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0931_1997050100_123_966_12183575_0_r544234623_44.zip Uploading the intermediate file: upload_file_44.zip 23:06:23 STEP 1081 H=1081:00 +CPU= 15.017 Suspend request received from the BOINC client, suspending the child process Resuming the child process 23:07:00 STEP 1082 H=1082:00 +CPU= 15.393 23:07:34 STEP 1083 H=1083:00 +CPU= 23.538 23:07:56 STEP 1084 H=1084:00 +CPU= 15.444 Suspend request received from the BOINC client, suspending the child process Resuming the child process 23:09:30 STEP 1085 H=1085:00 +CPU= 15.148 Suspend request received from the BOINC client, suspending the child process Resuming the child process 23:10:26 STEP 1086 H=1086:00 +CPU= 24.737 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 23:21:32 STEP 1087 H=1087:00 +CPU= 15.631 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 23:28:14 STEP 1088 H=1088:00 +CPU= 15.935 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4035MB (heap), 4797MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4035 MB (maxheap), 4797 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADDRV [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADINTG [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-RADINTG-RADLSW (1210) [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADIATION_SCHEME [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_interface:radiation [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233318:1675636398:2468.671] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_mcica_lw:solver_mcica_lw [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233319:1675636399:2469.671] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:debian-desktop:1:1:1952:1952] [20230205:233319:1675636399:2469.671] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Stack trace terminated abnormally. (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1997050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0931 (argv4) batchid: 966 (argv5) wuid: 12183575 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /media/nathan/512gb/boinc/slots/0 Project directory is: /media/nathan/512gb/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575 ..mkdir for temp folder for results failed Copying: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_300fd9cb50135bf5cb5b996106334d6a to: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_0931_1997050100_123_966_12183575.zip Unzipping the namelist zip file: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_0931_1997050100_123_966_12183575.zip ic_ancil_file: ic_ancil_12183575 ifsdata_file: ifsdata_12183575 climate_data_file: clim_data_12183575 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3c55903957ebcdaf3951b015145fa45 to: /media/nathan/512gb/boinc/slots/0/ic_ancil_12183575.zip Unzipping the IC ancils zip file: /media/nathan/512gb/boinc/slots/0/ic_ancil_12183575.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cda9331367b0b08951a627372fac3d6a to: /media/nathan/512gb/boinc/slots/0/ifsdata/ifsdata_12183575.zip Unzipping the ifsdata_zip file: /media/nathan/512gb/boinc/slots/0/ifsdata/ifsdata_12183575.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /media/nathan/512gb/boinc/slots/0/159l_2/clim_data_12183575.zip Unzipping the climate data zip file: /media/nathan/512gb/boinc/slots/0/159l_2/clim_data_12183575.zip Checking for progress XML file: /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml Opened progress file ok : /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml -- Model is restarting -- Adjusting last_iter, 1089, to previous model restart step. Creating progress file: /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml last_cpu_time: 26731 upload_file_number: 45 last_iter: 1079 last_upload: 3888000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0931_1997050100_123_966_12183575_0_r544234623 The child process has been launched with process id: 1950 Executing the command: /media/nathan/512gb/boinc/slots/0/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:1950:1950] [20220807:152554:1659878754:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:26:23 STEP 1080 H=1080:00 +CPU= 23.397 17:48:23 STEP 1081 H=1081:00 +CPU= 14.270 17:48:37 STEP 1082 H=1082:00 +CPU= 13.831 17:49:00 STEP 1083 H=1083:00 +CPU= 21.856 17:49:14 STEP 1084 H=1084:00 +CPU= 14.180 17:49:28 STEP 1085 H=1085:00 +CPU= 13.650 17:49:51 STEP 1086 H=1086:00 +CPU= 22.325 17:50:06 STEP 1087 H=1087:00 +CPU= 13.973 17:50:21 STEP 1088 H=1088:00 +CPU= 14.449 17:50:43 STEP 1089 H=1089:00 +CPU= 21.915 17:50:58 STEP 1090 H=1090:00 +CPU= 14.067 17:51:20 STEP 1091 H=1091:00 +CPU= 14.956 17:51:52 STEP 1092 H=1092:00 +CPU= 25.802 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001092 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001092 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001092 17:52:15 STEP 1093 H=1093:00 +CPU= 14.939 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:52:48 STEP 1094 H=1094:00 +CPU= 15.306 17:53:20 STEP 1095 H=1095:00 +CPU= 23.666 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:00:11 STEP 1096 H=1096:00 +CPU= 16.119 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:07:42 STEP 1097 H=1097:00 +CPU= 15.546 18:08:18 STEP 1098 H=1098:00 +CPU= 24.968 18:08:40 STEP 1099 H=1099:00 +CPU= 15.290 18:09:03 STEP 1100 H=1100:00 +CPU= 15.751 18:09:38 STEP 1101 H=1101:00 +CPU= 24.494 18:10:02 STEP 1102 H=1102:00 +CPU= 15.773 18:10:25 STEP 1103 H=1103:00 +CPU= 15.777 18:11:07 STEP 1104 H=1104:00 +CPU= 28.490 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001104 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001104 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001104 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001080 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001080 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001080 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001092 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001092 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001092 Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0931_1997050100_123_966_12183575_0_r544234623_45.zip Uploading the intermediate file: upload_file_45.zip 18:11:30 STEP 1105 H=1105:00 +CPU= 15.324 18:11:52 STEP 1106 H=1106:00 +CPU= 15.849 18:12:28 STEP 1107 H=1107:00 +CPU= 24.594 18:12:51 STEP 1108 H=1108:00 +CPU= 15.826 18:13:13 STEP 1109 H=1109:00 +CPU= 15.297 18:13:49 STEP 1110 H=1110:00 +CPU= 25.102 18:14:11 STEP 1111 H=1111:00 +CPU= 15.320 18:14:34 STEP 1112 H=1112:00 +CPU= 15.817 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:15:21 STEP 1113 H=1113:00 +CPU= 24.653 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:23:05 STEP 1114 H=1114:00 +CPU= 15.525 18:23:22 STEP 1115 H=1115:00 +CPU= 14.795 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:25:13 STEP 1116 H=1116:00 +CPU= 26.954 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001116 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001116 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001116 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:27:49 STEP 1117 H=1117:00 +CPU= 15.372 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:29:46 STEP 1118 H=1118:00 +CPU= 15.870 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:31:07 STEP 1119 H=1119:00 +CPU= 24.608 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:45:39 STEP 1120 H=1120:00 +CPU= 15.910 18:45:55 STEP 1121 H=1121:00 +CPU= 14.912 18:46:21 STEP 1122 H=1122:00 +CPU= 24.223 18:46:37 STEP 1123 H=1123:00 +CPU= 14.854 18:46:54 STEP 1124 H=1124:00 +CPU= 15.282 18:47:19 STEP 1125 H=1125:00 +CPU= 23.697 18:47:36 STEP 1126 H=1126:00 +CPU= 15.266 18:47:52 STEP 1127 H=1127:00 +CPU= 15.249 18:48:21 STEP 1128 H=1128:00 +CPU= 27.192 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001128 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001128 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001128 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001104 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001104 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001104 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001116 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001116 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001116 Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0931_1997050100_123_966_12183575_0_r544234623_46.zip Uploading the intermediate file: upload_file_46.zip 18:48:37 STEP 1129 H=1129:00 +CPU= 14.618 18:48:53 STEP 1130 H=1130:00 +CPU= 15.077 18:49:19 STEP 1131 H=1131:00 +CPU= 23.476 (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1997050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0931 (argv4) batchid: 966 (argv5) wuid: 12183575 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /media/nathan/512gb/boinc/slots/0 Project directory is: /media/nathan/512gb/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575 ..mkdir for temp folder for results failed Copying: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_300fd9cb50135bf5cb5b996106334d6a to: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_0931_1997050100_123_966_12183575.zip Unzipping the namelist zip file: /media/nathan/512gb/boinc/slots/0/oifs_43r3_ps_0931_1997050100_123_966_12183575.zip ic_ancil_file: ic_ancil_12183575 ifsdata_file: ifsdata_12183575 climate_data_file: clim_data_12183575 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3c55903957ebcdaf3951b015145fa45 to: /media/nathan/512gb/boinc/slots/0/ic_ancil_12183575.zip Unzipping the IC ancils zip file: /media/nathan/512gb/boinc/slots/0/ic_ancil_12183575.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cda9331367b0b08951a627372fac3d6a to: /media/nathan/512gb/boinc/slots/0/ifsdata/ifsdata_12183575.zip Unzipping the ifsdata_zip file: /media/nathan/512gb/boinc/slots/0/ifsdata/ifsdata_12183575.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /media/nathan/512gb/boinc/slots/0/159l_2/clim_data_12183575.zip Unzipping the climate data zip file: /media/nathan/512gb/boinc/slots/0/159l_2/clim_data_12183575.zip Checking for progress XML file: /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml Opened progress file ok : /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml -- Model is restarting -- Adjusting last_iter, 1132, to previous model restart step. Creating progress file: /media/nathan/512gb/boinc/slots/0/progress_file_12183575.xml last_cpu_time: 27689 upload_file_number: 47 last_iter: 1127 last_upload: 4060800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0931_1997050100_123_966_12183575_0_r544234623 The child process has been launched with process id: 2593 Executing the command: /media/nathan/512gb/boinc/slots/0/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debian-desktop:1:1:2593:2593] [20230208:173651:1675874211:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 17:37:32 STEP 1128 H=1128:00 +CPU= 24.337 17:37:45 STEP 1129 H=1129:00 +CPU= 13.330 17:38:00 STEP 1130 H=1130:00 +CPU= 14.029 17:38:22 STEP 1131 H=1131:00 +CPU= 21.495 17:38:36 STEP 1132 H=1132:00 +CPU= 13.850 17:38:49 STEP 1133 H=1133:00 +CPU= 13.573 17:39:13 STEP 1134 H=1134:00 +CPU= 22.564 17:39:28 STEP 1135 H=1135:00 +CPU= 14.036 17:39:43 STEP 1136 H=1136:00 +CPU= 14.130 17:40:05 STEP 1137 H=1137:00 +CPU= 22.039 17:40:19 STEP 1138 H=1138:00 +CPU= 14.122 17:40:33 STEP 1139 H=1139:00 +CPU= 13.621 17:41:16 STEP 1140 H=1140:00 +CPU= 26.164 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001140 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001140 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001140 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:08:13 STEP 1141 H=1141:00 +CPU= 15.705 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:08:47 STEP 1142 H=1142:00 +CPU= 16.389 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:36:50 STEP 1143 H=1143:00 +CPU= 25.416 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:58:17 STEP 1144 H=1144:00 +CPU= 16.574 18:58:32 STEP 1145 H=1145:00 +CPU= 14.268 18:58:56 STEP 1146 H=1146:00 +CPU= 23.399 18:59:10 STEP 1147 H=1147:00 +CPU= 14.217 18:59:25 STEP 1148 H=1148:00 +CPU= 14.671 18:59:48 STEP 1149 H=1149:00 +CPU= 22.906 19:00:03 STEP 1150 H=1150:00 +CPU= 14.646 19:00:18 STEP 1151 H=1151:00 +CPU= 14.632 19:00:46 STEP 1152 H=1152:00 +CPU= 26.572 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMGGhq0f+001152 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMSHhq0f+001152 Moving to projects directory: /media/nathan/512gb/boinc/slots/0/ICMUAhq0f+001152 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001128 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001128 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001128 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMGGhq0f+001140 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMSHhq0f+001140 Adding to the zip: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_12183575/ICMUAhq0f+001140 Zipping up the intermediate file: /media/nathan/512gb/boinc/projects/climateprediction.net/oifs_43r3_ps_0931_1997050100_123_966_12183575_0_r544234623_47.zip Uploading the intermediate file: upload_file_47.zip 19:01:01 STEP 1153 H=1153:00 +CPU= 14.184 19:01:16 STEP 1154 H=1154:00 +CPU= 14.637 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:24:58 | 1507457 | 22268760 | oifs_43r3_ps_0931_1997050100_123_966_12183575_0 | 2,419,200 | 15,135 | 0.0063 |
09 May 2023 16:24:41 | 1507457 | 22268760 | oifs_43r3_ps_0931_1997050100_123_966_12183575_0 | 1,555,200 | 9,067 | 0.0058 |
©2024 cpdn.org