climateprediction.net home page
Task 22292929

Task 22292929

Name oifs_43r3_ps_0691_2020050100_123_989_12206335_0
Workunit 12206335
Created 25 Dec 2022, 2:38:23 UTC
Sent 25 Dec 2022, 4:50:40 UTC
Report deadline 24 Jan 2023, 4:50:40 UTC
Received 4 Jan 2023, 16:58:58 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1511650
Run time 3 days 15 hours 54 min 22 sec
CPU time 3 days 2 hours 39 min 41 sec
Validate state Valid
Credit 13,305.60
Device peak FLOPS 2.42 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.21 MB
Peak swap size 4,978.07 MB
Peak disk usage 5,376.84 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
s directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002724
  13:54:43 STEP 2725 H=2725:00 +CPU= 69.279
  13:56:09 STEP 2726 H=2726:00 +CPU= 72.169
  13:58:24 STEP 2727 H=2727:00 +CPU=112.617
  13:59:48 STEP 2728 H=2728:00 +CPU= 72.068
  14:01:10 STEP 2729 H=2729:00 +CPU= 69.323
  14:03:25 STEP 2730 H=2730:00 +CPU=115.462
  14:04:46 STEP 2731 H=2731:00 +CPU= 68.974
  14:06:11 STEP 2732 H=2732:00 +CPU= 71.823
  14:08:16 STEP 2733 H=2733:00 +CPU=111.917
  14:09:38 STEP 2734 H=2734:00 +CPU= 72.216
  14:11:01 STEP 2735 H=2735:00 +CPU= 72.004
  14:13:34 STEP 2736 H=2736:00 +CPU=131.762
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_113.zip
Uploading the intermediate file: upload_file_113.zip
  14:14:58 STEP 2737 H=2737:00 +CPU= 69.015
  14:16:24 STEP 2738 H=2738:00 +CPU= 71.939
  14:18:37 STEP 2739 H=2739:00 +CPU=112.628
  14:20:02 STEP 2740 H=2740:00 +CPU= 71.960
  14:21:23 STEP 2741 H=2741:00 +CPU= 69.236
  14:23:37 STEP 2742 H=2742:00 +CPU=115.315
  14:24:53 STEP 2743 H=2743:00 +CPU= 68.954
  14:26:18 STEP 2744 H=2744:00 +CPU= 72.200
  14:28:31 STEP 2745 H=2745:00 +CPU=113.061
  14:29:51 STEP 2746 H=2746:00 +CPU= 71.896
  14:31:14 STEP 2747 H=2747:00 +CPU= 69.614
  14:33:42 STEP 2748 H=2748:00 +CPU=126.242
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002748
  14:35:04 STEP 2749 H=2749:00 +CPU= 69.538
  14:36:29 STEP 2750 H=2750:00 +CPU= 72.225
  14:38:38 STEP 2751 H=2751:00 +CPU=112.397
  14:40:00 STEP 2752 H=2752:00 +CPU= 71.989
  14:41:23 STEP 2753 H=2753:00 +CPU= 69.331
  14:43:34 STEP 2754 H=2754:00 +CPU=115.436
  14:44:55 STEP 2755 H=2755:00 +CPU= 69.492
  14:46:19 STEP 2756 H=2756:00 +CPU= 71.828
  14:48:29 STEP 2757 H=2757:00 +CPU=112.673
  14:49:54 STEP 2758 H=2758:00 +CPU= 72.201
  14:51:18 STEP 2759 H=2759:00 +CPU= 71.779
  14:53:53 STEP 2760 H=2760:00 +CPU=131.692
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_114.zip
Uploading the intermediate file: upload_file_114.zip
  14:55:12 STEP 2761 H=2761:00 +CPU= 69.174
  14:56:33 STEP 2762 H=2762:00 +CPU= 71.705
  14:58:42 STEP 2763 H=2763:00 +CPU=111.987
  15:00:03 STEP 2764 H=2764:00 +CPU= 71.541
  15:01:20 STEP 2765 H=2765:00 +CPU= 69.329
  15:03:31 STEP 2766 H=2766:00 +CPU=115.733
  15:04:53 STEP 2767 H=2767:00 +CPU= 69.631
  15:06:18 STEP 2768 H=2768:00 +CPU= 72.203
  15:08:27 STEP 2769 H=2769:00 +CPU=112.606
  15:09:49 STEP 2770 H=2770:00 +CPU= 72.122
  15:11:12 STEP 2771 H=2771:00 +CPU= 69.366
  15:13:38 STEP 2772 H=2772:00 +CPU=125.995
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002772
  15:14:56 STEP 2773 H=2773:00 +CPU= 69.090
  15:16:22 STEP 2774 H=2774:00 +CPU= 72.320
  15:18:30 STEP 2775 H=2775:00 +CPU=112.560
  15:19:55 STEP 2776 H=2776:00 +CPU= 72.777
  15:21:18 STEP 2777 H=2777:00 +CPU= 69.451
  15:23:29 STEP 2778 H=2778:00 +CPU=115.595
  15:24:52 STEP 2779 H=2779:00 +CPU= 69.316
  15:26:19 STEP 2780 H=2780:00 +CPU= 73.512
  15:28:41 STEP 2781 H=2781:00 +CPU=119.869
  15:30:13 STEP 2782 H=2782:00 +CPU= 76.520
  15:31:39 STEP 2783 H=2783:00 +CPU= 76.251
  15:34:21 STEP 2784 H=2784:00 +CPU=139.797
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_115.zip
Uploading the intermediate file: upload_file_115.zip
  15:35:48 STEP 2785 H=2785:00 +CPU= 73.630
  15:37:17 STEP 2786 H=2786:00 +CPU= 77.020
  15:39:40 STEP 2787 H=2787:00 +CPU=120.084
  15:41:09 STEP 2788 H=2788:00 +CPU= 76.782
  15:42:33 STEP 2789 H=2789:00 +CPU= 73.548
  15:44:56 STEP 2790 H=2790:00 +CPU=122.688
  15:46:23 STEP 2791 H=2791:00 +CPU= 73.917
  15:47:51 STEP 2792 H=2792:00 +CPU= 76.757
  15:50:07 STEP 2793 H=2793:00 +CPU=119.485
  15:51:30 STEP 2794 H=2794:00 +CPU= 76.423
  15:52:56 STEP 2795 H=2795:00 +CPU= 73.999
  15:55:35 STEP 2796 H=2796:00 +CPU=133.849
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002796
  15:57:05 STEP 2797 H=2797:00 +CPU= 74.034
  15:58:37 STEP 2798 H=2798:00 +CPU= 76.896
  16:00:57 STEP 2799 H=2799:00 +CPU=119.669
  16:02:26 STEP 2800 H=2800:00 +CPU= 76.630
  16:03:52 STEP 2801 H=2801:00 +CPU= 73.752
  16:06:14 STEP 2802 H=2802:00 +CPU=122.497
  16:07:37 STEP 2803 H=2803:00 +CPU= 73.615
  16:09:06 STEP 2804 H=2804:00 +CPU= 76.315
  16:11:21 STEP 2805 H=2805:00 +CPU=119.850
  16:12:52 STEP 2806 H=2806:00 +CPU= 76.870
  16:14:22 STEP 2807 H=2807:00 +CPU= 76.614
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0691
(argv4) batchid: 989
(argv5) wuid: 12206335
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6bc6639fa8c1a2f4707d51dad5bfe3c5 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0691_2020050100_123_989_12206335.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0691_2020050100_123_989_12206335.zip
ic_ancil_file: ic_ancil_12206335
ifsdata_file: ifsdata_12206335
climate_data_file: clim_data_12206335
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_517fd04b1f155054a971849706955cdc to: /var/lib/boinc-client/slots/3/ic_ancil_12206335.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12206335.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d8f8639bfddfcb1bb6bc1b039957815a to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12206335.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12206335.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12206335.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12206335.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12206335.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12206335.xml
-- Model is restarting --
Adjusting last_iter, 2808, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12206335.xml
last_cpu_time: 253360
upload_file_number: 116
last_iter: 2807
last_upload: 10022400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366
The child process has been launched with process id: 1053919
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1053919:1053919] [20221228:213500:1672263300:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:37:44 STEP 2808 H=2808:00 +CPU=136.314
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_116.zip
Uploading the intermediate file: upload_file_116.zip
  21:39:11 STEP 2809 H=2809:00 +CPU= 73.596
  21:40:40 STEP 2810 H=2810:00 +CPU= 76.180
  21:42:57 STEP 2811 H=2811:00 +CPU=119.084
  21:44:27 STEP 2812 H=2812:00 +CPU= 76.515
  21:45:55 STEP 2813 H=2813:00 +CPU= 73.666
  21:48:15 STEP 2814 H=2814:00 +CPU=122.132
  21:49:43 STEP 2815 H=2815:00 +CPU= 73.450
  21:51:11 STEP 2816 H=2816:00 +CPU= 76.578
  21:53:26 STEP 2817 H=2817:00 +CPU=118.849
  21:54:55 STEP 2818 H=2818:00 +CPU= 76.666
  21:56:20 STEP 2819 H=2819:00 +CPU= 73.765
  21:58:55 STEP 2820 H=2820:00 +CPU=133.114
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002820
  22:00:22 STEP 2821 H=2821:00 +CPU= 73.858
  22:01:50 STEP 2822 H=2822:00 +CPU= 76.468
  22:04:08 STEP 2823 H=2823:00 +CPU=119.169
  22:05:35 STEP 2824 H=2824:00 +CPU= 76.385
  22:06:58 STEP 2825 H=2825:00 +CPU= 73.345
  22:09:17 STEP 2826 H=2826:00 +CPU=122.113
  22:10:44 STEP 2827 H=2827:00 +CPU= 74.039
  22:12:11 STEP 2828 H=2828:00 +CPU= 76.767
  22:14:30 STEP 2829 H=2829:00 +CPU=119.412
  22:16:01 STEP 2830 H=2830:00 +CPU= 76.531
  22:17:32 STEP 2831 H=2831:00 +CPU= 76.505
  22:20:16 STEP 2832 H=2832:00 +CPU=139.430
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_117.zip
Uploading the intermediate file: upload_file_117.zip
  22:21:41 STEP 2833 H=2833:00 +CPU= 73.428
  22:23:12 STEP 2834 H=2834:00 +CPU= 76.484
  22:25:32 STEP 2835 H=2835:00 +CPU=118.945
  22:27:00 STEP 2836 H=2836:00 +CPU= 76.470
  22:28:21 STEP 2837 H=2837:00 +CPU= 73.340
  22:30:46 STEP 2838 H=2838:00 +CPU=122.494
  22:32:09 STEP 2839 H=2839:00 +CPU= 73.793
  22:33:39 STEP 2840 H=2840:00 +CPU= 76.724
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0691
(argv4) batchid: 989
(argv5) wuid: 12206335
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6bc6639fa8c1a2f4707d51dad5bfe3c5 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0691_2020050100_123_989_12206335.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0691_2020050100_123_989_12206335.zip
ic_ancil_file: ic_ancil_12206335
ifsdata_file: ifsdata_12206335
climate_data_file: clim_data_12206335
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_517fd04b1f155054a971849706955cdc to: /var/lib/boinc-client/slots/3/ic_ancil_12206335.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12206335.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d8f8639bfddfcb1bb6bc1b039957815a to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12206335.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12206335.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12206335.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12206335.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12206335.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12206335.xml
-- Model is restarting --
Adjusting last_iter, 2841, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12206335.xml
last_cpu_time: 256484
upload_file_number: 118
last_iter: 2831
last_upload: 10195200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366
The child process has been launched with process id: 1060551
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1060551:1060551] [20221228:234337:1672271017:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:46:27 STEP 2832 H=2832:00 +CPU=135.731
  23:47:52 STEP 2833 H=2833:00 +CPU= 73.154
  23:49:21 STEP 2834 H=2834:00 +CPU= 76.322
  23:51:34 STEP 2835 H=2835:00 +CPU=118.424
  23:53:01 STEP 2836 H=2836:00 +CPU= 76.571
  23:54:27 STEP 2837 H=2837:00 +CPU= 73.632
  23:56:51 STEP 2838 H=2838:00 +CPU=122.376
  23:58:16 STEP 2839 H=2839:00 +CPU= 73.899
  23:59:46 STEP 2840 H=2840:00 +CPU= 76.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:06:14 STEP 2841 H=2841:00 +CPU=119.238
  00:07:45 STEP 2842 H=2842:00 +CPU= 76.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:11:22 STEP 2843 H=2843:00 +CPU= 73.483
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:14:22 STEP 2844 H=2844:00 +CPU=133.492
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002844
  00:15:49 STEP 2845 H=2845:00 +CPU= 73.581
  00:17:19 STEP 2846 H=2846:00 +CPU= 76.631
  00:19:40 STEP 2847 H=2847:00 +CPU=118.938
  00:21:10 STEP 2848 H=2848:00 +CPU= 76.531
  00:22:34 STEP 2849 H=2849:00 +CPU= 73.551
  00:24:53 STEP 2850 H=2850:00 +CPU=121.798
  00:26:19 STEP 2851 H=2851:00 +CPU= 73.625
  00:27:47 STEP 2852 H=2852:00 +CPU= 76.532
  00:30:06 STEP 2853 H=2853:00 +CPU=119.241
  00:31:37 STEP 2854 H=2854:00 +CPU= 76.507
  00:33:09 STEP 2855 H=2855:00 +CPU= 76.540
  00:35:56 STEP 2856 H=2856:00 +CPU=139.354
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_118.zip
Uploading the intermediate file: upload_file_118.zip
  00:37:25 STEP 2857 H=2857:00 +CPU= 73.545
  00:39:02 STEP 2858 H=2858:00 +CPU= 76.234
  00:41:20 STEP 2859 H=2859:00 +CPU=118.849
  00:42:48 STEP 2860 H=2860:00 +CPU= 76.286
  00:44:15 STEP 2861 H=2861:00 +CPU= 73.868
  00:46:41 STEP 2862 H=2862:00 +CPU=122.635
  00:48:07 STEP 2863 H=2863:00 +CPU= 73.748
  00:49:35 STEP 2864 H=2864:00 +CPU= 76.634
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:52:34 STEP 2865 H=2865:00 +CPU=119.436
  00:54:09 STEP 2866 H=2866:00 +CPU= 76.893
  00:55:54 STEP 2867 H=2867:00 +CPU= 74.059
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0691
(argv4) batchid: 989
(argv5) wuid: 12206335
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6bc6639fa8c1a2f4707d51dad5bfe3c5 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0691_2020050100_123_989_12206335.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0691_2020050100_123_989_12206335.zip
ic_ancil_file: ic_ancil_12206335
ifsdata_file: ifsdata_12206335
climate_data_file: clim_data_12206335
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_517fd04b1f155054a971849706955cdc to: /var/lib/boinc-client/slots/3/ic_ancil_12206335.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12206335.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d8f8639bfddfcb1bb6bc1b039957815a to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12206335.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12206335.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12206335.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12206335.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12206335.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12206335.xml
-- Model is restarting --
Adjusting last_iter, 2868, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12206335.xml
last_cpu_time: 259794
upload_file_number: 119
last_iter: 2855
last_upload: 10281600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366
The child process has been launched with process id: 1068110
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1068110:1068110] [20221229:013822:1672277902:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:41:16 STEP 2856 H=2856:00 +CPU=136.344
  01:42:43 STEP 2857 H=2857:00 +CPU= 73.662
  01:44:16 STEP 2858 H=2858:00 +CPU= 76.592
  01:46:37 STEP 2859 H=2859:00 +CPU=118.703
  01:48:08 STEP 2860 H=2860:00 +CPU= 76.400
  01:49:31 STEP 2861 H=2861:00 +CPU= 73.400
  01:51:55 STEP 2862 H=2862:00 +CPU=122.444
  01:53:21 STEP 2863 H=2863:00 +CPU= 73.937
  01:54:49 STEP 2864 H=2864:00 +CPU= 76.598
  01:57:12 STEP 2865 H=2865:00 +CPU=119.465
  01:58:42 STEP 2866 H=2866:00 +CPU= 76.637
  02:00:07 STEP 2867 H=2867:00 +CPU= 73.591
  02:02:42 STEP 2868 H=2868:00 +CPU=133.240
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002868
  02:04:09 STEP 2869 H=2869:00 +CPU= 73.659
  02:05:40 STEP 2870 H=2870:00 +CPU= 76.615
  02:07:58 STEP 2871 H=2871:00 +CPU=119.100
  02:09:28 STEP 2872 H=2872:00 +CPU= 76.617
  02:10:56 STEP 2873 H=2873:00 +CPU= 73.954
  02:13:24 STEP 2874 H=2874:00 +CPU=122.555
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:15:01 STEP 2875 H=2875:00 +CPU= 73.681
  02:16:30 STEP 2876 H=2876:00 +CPU= 76.442
  02:18:51 STEP 2877 H=2877:00 +CPU=118.958
  02:20:20 STEP 2878 H=2878:00 +CPU= 76.427
  02:21:48 STEP 2879 H=2879:00 +CPU= 76.253
  02:24:31 STEP 2880 H=2880:00 +CPU=138.946
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_119.zip
Uploading the intermediate file: upload_file_119.zip
  02:25:59 STEP 2881 H=2881:00 +CPU= 73.598
  02:27:27 STEP 2882 H=2882:00 +CPU= 76.582
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:29:58 STEP 2883 H=2883:00 +CPU=119.189
  02:31:31 STEP 2884 H=2884:00 +CPU= 76.774
  02:32:54 STEP 2885 H=2885:00 +CPU= 73.540
  02:35:17 STEP 2886 H=2886:00 +CPU=122.135
  02:36:41 STEP 2887 H=2887:00 +CPU= 73.398
  02:38:08 STEP 2888 H=2888:00 +CPU= 76.429
  02:40:23 STEP 2889 H=2889:00 +CPU=119.054
  02:41:52 STEP 2890 H=2890:00 +CPU= 76.648
  02:43:18 STEP 2891 H=2891:00 +CPU= 73.858
  02:45:55 STEP 2892 H=2892:00 +CPU=133.494
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002892
  02:47:23 STEP 2893 H=2893:00 +CPU= 73.703
  02:48:55 STEP 2894 H=2894:00 +CPU= 76.555
  02:51:14 STEP 2895 H=2895:00 +CPU=119.064
  02:52:46 STEP 2896 H=2896:00 +CPU= 76.622
  02:54:10 STEP 2897 H=2897:00 +CPU= 73.676
  02:56:31 STEP 2898 H=2898:00 +CPU=122.020
  02:57:57 STEP 2899 H=2899:00 +CPU= 73.744
  02:59:24 STEP 2900 H=2900:00 +CPU= 76.367
  03:01:41 STEP 2901 H=2901:00 +CPU=119.451
  03:03:09 STEP 2902 H=2902:00 +CPU= 76.523
  03:04:37 STEP 2903 H=2903:00 +CPU= 76.254
  03:07:18 STEP 2904 H=2904:00 +CPU=139.520
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_120.zip
Uploading the intermediate file: upload_file_120.zip
  03:08:45 STEP 2905 H=2905:00 +CPU= 73.397
  03:10:16 STEP 2906 H=2906:00 +CPU= 76.446
  03:12:37 STEP 2907 H=2907:00 +CPU=119.107
  03:14:04 STEP 2908 H=2908:00 +CPU= 76.306
  03:15:33 STEP 2909 H=2909:00 +CPU= 74.077
  03:17:55 STEP 2910 H=2910:00 +CPU=122.659
  03:19:22 STEP 2911 H=2911:00 +CPU= 73.838
  03:20:51 STEP 2912 H=2912:00 +CPU= 76.463
  03:23:11 STEP 2913 H=2913:00 +CPU=119.404
  03:24:40 STEP 2914 H=2914:00 +CPU= 76.525
  03:26:09 STEP 2915 H=2915:00 +CPU= 73.773
  03:28:47 STEP 2916 H=2916:00 +CPU=133.173
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002916
  03:30:12 STEP 2917 H=2917:00 +CPU= 73.491
  03:31:41 STEP 2918 H=2918:00 +CPU= 76.858
  03:34:00 STEP 2919 H=2919:00 +CPU=119.314
  03:35:30 STEP 2920 H=2920:00 +CPU= 76.670
  03:36:58 STEP 2921 H=2921:00 +CPU= 73.902
  03:39:21 STEP 2922 H=2922:00 +CPU=122.526
  03:40:49 STEP 2923 H=2923:00 +CPU= 74.030
  03:42:19 STEP 2924 H=2924:00 +CPU= 76.664
  03:44:39 STEP 2925 H=2925:00 +CPU=119.077
  03:46:08 STEP 2926 H=2926:00 +CPU= 76.378
  03:47:39 STEP 2927 H=2927:00 +CPU= 76.216
  03:50:21 STEP 2928 H=2928:00 +CPU=139.598
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_121.zip
Uploading the intermediate file: upload_file_121.zip
  03:51:50 STEP 2929 H=2929:00 +CPU= 73.742
  03:53:19 STEP 2930 H=2930:00 +CPU= 76.535
  03:55:38 STEP 2931 H=2931:00 +CPU=119.250
  03:57:08 STEP 2932 H=2932:00 +CPU= 76.555
  03:58:34 STEP 2933 H=2933:00 +CPU= 73.439
  04:01:00 STEP 2934 H=2934:00 +CPU=122.217
  04:02:24 STEP 2935 H=2935:00 +CPU= 73.523
  04:03:54 STEP 2936 H=2936:00 +CPU= 76.475
  04:06:15 STEP 2937 H=2937:00 +CPU=119.629
  04:07:44 STEP 2938 H=2938:00 +CPU= 76.771
  04:09:13 STEP 2939 H=2939:00 +CPU= 73.987
  04:11:48 STEP 2940 H=2940:00 +CPU=133.642
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002940
  04:13:16 STEP 2941 H=2941:00 +CPU= 73.893
  04:14:46 STEP 2942 H=2942:00 +CPU= 76.659
  04:17:03 STEP 2943 H=2943:00 +CPU=119.057
  04:18:34 STEP 2944 H=2944:00 +CPU= 76.460
  04:20:01 STEP 2945 H=2945:00 +CPU= 73.836
  04:22:16 STEP 2946 H=2946:00 +CPU=122.163
  04:23:44 STEP 2947 H=2947:00 +CPU= 73.602
  04:25:13 STEP 2948 H=2948:00 +CPU= 76.555
  04:27:33 STEP 2949 H=2949:00 +CPU=119.354
  04:29:04 STEP 2950 H=2950:00 +CPU= 76.194
  04:30:35 STEP 2951 H=2951:00 +CPU= 76.503
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0691
(argv4) batchid: 989
(argv5) wuid: 12206335
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6bc6639fa8c1a2f4707d51dad5bfe3c5 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0691_2020050100_123_989_12206335.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0691_2020050100_123_989_12206335.zip
ic_ancil_file: ic_ancil_12206335
ifsdata_file: ifsdata_12206335
climate_data_file: clim_data_12206335
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_517fd04b1f155054a971849706955cdc to: /var/lib/boinc-client/slots/3/ic_ancil_12206335.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12206335.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d8f8639bfddfcb1bb6bc1b039957815a to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12206335.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12206335.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12206335.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12206335.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12206335.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12206335.xml
-- Model is restarting --
Adjusting last_iter, 2952, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12206335.xml
last_cpu_time: 268652
upload_file_number: 122
last_iter: 2951
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366
The child process has been launched with process id: 1080322
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1080322:1080322] [20221229:053329:1672292009:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:35:56 STEP 2952 H=2952:00 +CPU=124.079
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206335/ICMUAhq0f+002940
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0691_2020050100_123_989_12206335_0_r497661366_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: -3596400
05:39:21 (1080300): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:00:01 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 9,504,000 238,672 0.0251
09 May 2023 15:59:37 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 8,640,000 217,965 0.0252
09 May 2023 15:59:14 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 7,776,000 197,195 0.0254
09 May 2023 15:58:48 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 6,912,000 175,309 0.0254
09 May 2023 15:58:20 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 6,048,000 153,306 0.0253
09 May 2023 15:57:50 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 5,184,000 131,321 0.0253
09 May 2023 15:57:18 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 4,320,000 109,317 0.0253
09 May 2023 15:56:43 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 3,456,000 87,325 0.0253
09 May 2023 15:56:06 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 2,592,000 65,336 0.0252
09 May 2023 15:55:30 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 1,728,000 43,346 0.0251
09 May 2023 15:54:50 1511650 22292929 oifs_43r3_ps_0691_2020050100_123_989_12206335_0 864,000 21,601 0.0250


©2024 cpdn.org