climateprediction.net home page
Task 22299554

Task 22299554

Name oifs_43r3_ps_0315_2002050100_123_971_12187959_1
Workunit 12187959
Created 18 Jan 2023, 20:50:10 UTC
Sent 18 Jan 2023, 20:52:37 UTC
Report deadline 19 Mar 2023, 20:52:37 UTC
Received 19 Jan 2023, 14:04:37 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1532237
Run time 16 hours 14 min 42 sec
CPU time 15 hours 58 min 52 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,413.51 MB
Peak swap size 4,978.08 MB
Peak disk usage 6,061.96 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
51:00 +CPU= 18.036
  04:03:50 STEP 2652 H=2652:00 +CPU= 33.563
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002652
  04:04:11 STEP 2653 H=2653:00 +CPU= 19.871
  04:04:31 STEP 2654 H=2654:00 +CPU= 20.096
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0315
(argv4) batchid: 971
(argv5) wuid: 12187959
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5701fb94b96cc242319fb9dac33d2dc to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0315_2002050100_123_971_12187959.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0315_2002050100_123_971_12187959.zip
ic_ancil_file: ic_ancil_12187959
ifsdata_file: ifsdata_12187959
climate_data_file: clim_data_12187959
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2cbd7f649367332f90b9f547dcc31a2a to: /var/lib/boinc-client/slots/0/ic_ancil_12187959.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12187959.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_86f388c690ddc2891e1b57b105f43e2f to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187959.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187959.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187959.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187959.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12187959.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12187959.xml
-- Model is restarting --
Adjusting last_iter, 2654, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12187959.xml
last_cpu_time: 49880
upload_file_number: 110
last_iter: 2639
last_upload: 9504000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740
The child process has been launched with process id: 3543753
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3543753:3543753] [20230119:040535:1674126335:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:06:12 STEP 2640 H=2640:00 +CPU= 33.034
  04:06:32 STEP 2641 H=2641:00 +CPU= 19.673
  04:06:50 STEP 2642 H=2642:00 +CPU= 17.975
  04:07:18 STEP 2643 H=2643:00 +CPU= 27.286
  04:07:38 STEP 2644 H=2644:00 +CPU= 19.724
  04:07:57 STEP 2645 H=2645:00 +CPU= 18.997
  04:08:27 STEP 2646 H=2646:00 +CPU= 29.982
  04:08:46 STEP 2647 H=2647:00 +CPU= 18.932
  04:09:05 STEP 2648 H=2648:00 +CPU= 18.626
  04:09:35 STEP 2649 H=2649:00 +CPU= 28.782
  04:09:55 STEP 2650 H=2650:00 +CPU= 20.017
  04:10:13 STEP 2651 H=2651:00 +CPU= 17.447
  04:10:43 STEP 2652 H=2652:00 +CPU= 30.264
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002652
  04:11:03 STEP 2653 H=2653:00 +CPU= 19.365
  04:11:23 STEP 2654 H=2654:00 +CPU= 19.287
  04:11:50 STEP 2655 H=2655:00 +CPU= 27.028
  04:12:08 STEP 2656 H=2656:00 +CPU= 17.353
  04:12:25 STEP 2657 H=2657:00 +CPU= 17.582
  04:12:56 STEP 2658 H=2658:00 +CPU= 29.318
  04:13:13 STEP 2659 H=2659:00 +CPU= 17.478
  04:13:32 STEP 2660 H=2660:00 +CPU= 18.214
  04:14:01 STEP 2661 H=2661:00 +CPU= 28.673
  04:14:22 STEP 2662 H=2662:00 +CPU= 20.191
  04:14:39 STEP 2663 H=2663:00 +CPU= 17.624
  04:15:12 STEP 2664 H=2664:00 +CPU= 32.218
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_110.zip
Uploading the intermediate file: upload_file_110.zip
  04:15:29 STEP 2665 H=2665:00 +CPU= 16.581
  04:15:49 STEP 2666 H=2666:00 +CPU= 19.509
  04:16:19 STEP 2667 H=2667:00 +CPU= 29.559
  04:16:36 STEP 2668 H=2668:00 +CPU= 16.550
  04:16:55 STEP 2669 H=2669:00 +CPU= 18.732
  04:17:24 STEP 2670 H=2670:00 +CPU= 28.446
  04:17:41 STEP 2671 H=2671:00 +CPU= 16.906
  04:17:58 STEP 2672 H=2672:00 +CPU= 17.188
  04:18:29 STEP 2673 H=2673:00 +CPU= 29.676
  04:18:48 STEP 2674 H=2674:00 +CPU= 19.118
  04:19:08 STEP 2675 H=2675:00 +CPU= 20.038
  04:19:40 STEP 2676 H=2676:00 +CPU= 31.301
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002676
  04:19:58 STEP 2677 H=2677:00 +CPU= 18.120
  04:20:19 STEP 2678 H=2678:00 +CPU= 20.198
  04:20:46 STEP 2679 H=2679:00 +CPU= 27.033
  04:21:06 STEP 2680 H=2680:00 +CPU= 18.863
  04:21:24 STEP 2681 H=2681:00 +CPU= 18.205
  04:21:54 STEP 2682 H=2682:00 +CPU= 30.091
  04:22:12 STEP 2683 H=2683:00 +CPU= 17.000
  04:22:29 STEP 2684 H=2684:00 +CPU= 16.976
  04:22:59 STEP 2685 H=2685:00 +CPU= 29.288
  04:23:18 STEP 2686 H=2686:00 +CPU= 19.064
  04:23:38 STEP 2687 H=2687:00 +CPU= 19.033
  04:24:09 STEP 2688 H=2688:00 +CPU= 31.440
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_111.zip
Uploading the intermediate file: upload_file_111.zip
  04:24:29 STEP 2689 H=2689:00 +CPU= 18.677
  04:24:47 STEP 2690 H=2690:00 +CPU= 18.220
  04:25:17 STEP 2691 H=2691:00 +CPU= 29.005
  04:25:35 STEP 2692 H=2692:00 +CPU= 18.104
  04:25:54 STEP 2693 H=2693:00 +CPU= 19.155
  04:26:21 STEP 2694 H=2694:00 +CPU= 26.576
  04:26:38 STEP 2695 H=2695:00 +CPU= 16.616
  04:26:57 STEP 2696 H=2696:00 +CPU= 18.484
  04:27:29 STEP 2697 H=2697:00 +CPU= 31.294
  04:27:49 STEP 2698 H=2698:00 +CPU= 19.764
  04:28:07 STEP 2699 H=2699:00 +CPU= 17.944
  04:28:37 STEP 2700 H=2700:00 +CPU= 29.591
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002700
  04:28:55 STEP 2701 H=2701:00 +CPU= 17.277
  04:29:11 STEP 2702 H=2702:00 +CPU= 16.288
  04:29:40 STEP 2703 H=2703:00 +CPU= 28.135
  04:29:58 STEP 2704 H=2704:00 +CPU= 17.985
  04:30:17 STEP 2705 H=2705:00 +CPU= 19.249
  04:30:48 STEP 2706 H=2706:00 +CPU= 30.359
  04:31:07 STEP 2707 H=2707:00 +CPU= 18.624
  04:31:26 STEP 2708 H=2708:00 +CPU= 18.238
  04:31:56 STEP 2709 H=2709:00 +CPU= 29.252
  04:32:14 STEP 2710 H=2710:00 +CPU= 17.815
  04:32:34 STEP 2711 H=2711:00 +CPU= 19.423
  04:33:06 STEP 2712 H=2712:00 +CPU= 32.209
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_112.zip
Uploading the intermediate file: upload_file_112.zip
  04:33:25 STEP 2713 H=2713:00 +CPU= 18.197
  04:33:42 STEP 2714 H=2714:00 +CPU= 17.442
  04:34:11 STEP 2715 H=2715:00 +CPU= 28.153
  04:34:31 STEP 2716 H=2716:00 +CPU= 19.363
  04:34:50 STEP 2717 H=2717:00 +CPU= 18.561
  04:35:19 STEP 2718 H=2718:00 +CPU= 28.709
  04:35:37 STEP 2719 H=2719:00 +CPU= 17.554
  04:35:54 STEP 2720 H=2720:00 +CPU= 17.198
  04:36:22 STEP 2721 H=2721:00 +CPU= 27.628
  04:36:40 STEP 2722 H=2722:00 +CPU= 17.512
  04:36:58 STEP 2723 H=2723:00 +CPU= 17.820
  04:37:31 STEP 2724 H=2724:00 +CPU= 32.081
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002724
  04:37:49 STEP 2725 H=2725:00 +CPU= 17.918
  04:38:06 STEP 2726 H=2726:00 +CPU= 16.353
  04:38:34 STEP 2727 H=2727:00 +CPU= 27.829
  04:38:54 STEP 2728 H=2728:00 +CPU= 19.441
  04:39:12 STEP 2729 H=2729:00 +CPU= 18.688
  04:39:42 STEP 2730 H=2730:00 +CPU= 29.267
  04:40:01 STEP 2731 H=2731:00 +CPU= 18.859
  04:40:18 STEP 2732 H=2732:00 +CPU= 16.809
  04:40:46 STEP 2733 H=2733:00 +CPU= 27.058
  04:41:04 STEP 2734 H=2734:00 +CPU= 17.869
  04:41:26 STEP 2735 H=2735:00 +CPU= 21.113
  04:41:59 STEP 2736 H=2736:00 +CPU= 32.920
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_113.zip
Uploading the intermediate file: upload_file_113.zip
  04:42:15 STEP 2737 H=2737:00 +CPU= 16.084
  04:42:35 STEP 2738 H=2738:00 +CPU= 19.144
  04:43:05 STEP 2739 H=2739:00 +CPU= 29.407
  04:43:24 STEP 2740 H=2740:00 +CPU= 19.091
  04:43:41 STEP 2741 H=2741:00 +CPU= 16.233
  04:44:10 STEP 2742 H=2742:00 +CPU= 28.838
  04:44:30 STEP 2743 H=2743:00 +CPU= 19.335
  04:44:47 STEP 2744 H=2744:00 +CPU= 17.764
  04:45:18 STEP 2745 H=2745:00 +CPU= 29.664
  04:45:39 STEP 2746 H=2746:00 +CPU= 20.756
  04:45:59 STEP 2747 H=2747:00 +CPU= 19.410
  04:46:31 STEP 2748 H=2748:00 +CPU= 31.324
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002748
  04:46:49 STEP 2749 H=2749:00 +CPU= 18.191
  04:47:10 STEP 2750 H=2750:00 +CPU= 20.273
  04:47:39 STEP 2751 H=2751:00 +CPU= 28.435
  04:47:57 STEP 2752 H=2752:00 +CPU= 18.415
  04:48:15 STEP 2753 H=2753:00 +CPU= 17.725
  04:48:47 STEP 2754 H=2754:00 +CPU= 30.918
  04:49:05 STEP 2755 H=2755:00 +CPU= 18.219
  04:49:22 STEP 2756 H=2756:00 +CPU= 16.718
  04:49:50 STEP 2757 H=2757:00 +CPU= 27.429
  04:50:09 STEP 2758 H=2758:00 +CPU= 18.344
  04:50:29 STEP 2759 H=2759:00 +CPU= 20.089
  04:51:05 STEP 2760 H=2760:00 +CPU= 35.425
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_114.zip
Uploading the intermediate file: upload_file_114.zip
  04:51:25 STEP 2761 H=2761:00 +CPU= 19.138
  04:51:45 STEP 2762 H=2762:00 +CPU= 20.197
  04:52:14 STEP 2763 H=2763:00 +CPU= 27.917
  04:52:32 STEP 2764 H=2764:00 +CPU= 17.885
  04:52:48 STEP 2765 H=2765:00 +CPU= 16.140
  04:53:18 STEP 2766 H=2766:00 +CPU= 29.145
  04:53:36 STEP 2767 H=2767:00 +CPU= 18.210
  04:53:56 STEP 2768 H=2768:00 +CPU= 19.803
  04:54:24 STEP 2769 H=2769:00 +CPU= 27.675
  04:54:43 STEP 2770 H=2770:00 +CPU= 18.398
  04:55:01 STEP 2771 H=2771:00 +CPU= 17.601
  04:55:32 STEP 2772 H=2772:00 +CPU= 29.910
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002772
  04:55:49 STEP 2773 H=2773:00 +CPU= 17.202
  04:56:07 STEP 2774 H=2774:00 +CPU= 17.663
  04:56:34 STEP 2775 H=2775:00 +CPU= 26.772
  04:56:52 STEP 2776 H=2776:00 +CPU= 18.079
  04:57:10 STEP 2777 H=2777:00 +CPU= 17.665
  04:57:38 STEP 2778 H=2778:00 +CPU= 27.152
  04:57:56 STEP 2779 H=2779:00 +CPU= 17.550
  04:58:14 STEP 2780 H=2780:00 +CPU= 18.235
  04:58:43 STEP 2781 H=2781:00 +CPU= 28.613
  04:59:04 STEP 2782 H=2782:00 +CPU= 19.940
  04:59:22 STEP 2783 H=2783:00 +CPU= 18.833
  04:59:54 STEP 2784 H=2784:00 +CPU= 31.574
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_115.zip
Uploading the intermediate file: upload_file_115.zip
  05:00:13 STEP 2785 H=2785:00 +CPU= 17.964
  05:00:33 STEP 2786 H=2786:00 +CPU= 20.124
  05:01:03 STEP 2787 H=2787:00 +CPU= 29.088
  05:01:21 STEP 2788 H=2788:00 +CPU= 18.106
  05:01:41 STEP 2789 H=2789:00 +CPU= 19.406
  05:02:10 STEP 2790 H=2790:00 +CPU= 28.666
  05:02:28 STEP 2791 H=2791:00 +CPU= 17.883
  05:02:49 STEP 2792 H=2792:00 +CPU= 20.177
  05:03:18 STEP 2793 H=2793:00 +CPU= 29.582
  05:03:38 STEP 2794 H=2794:00 +CPU= 19.629
  05:03:56 STEP 2795 H=2795:00 +CPU= 17.724
  05:04:28 STEP 2796 H=2796:00 +CPU= 31.443
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002796
  05:04:47 STEP 2797 H=2797:00 +CPU= 18.534
  05:05:05 STEP 2798 H=2798:00 +CPU= 17.888
  05:05:36 STEP 2799 H=2799:00 +CPU= 29.958
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0315
(argv4) batchid: 971
(argv5) wuid: 12187959
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5701fb94b96cc242319fb9dac33d2dc to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0315_2002050100_123_971_12187959.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0315_2002050100_123_971_12187959.zip
ic_ancil_file: ic_ancil_12187959
ifsdata_file: ifsdata_12187959
climate_data_file: clim_data_12187959
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2cbd7f649367332f90b9f547dcc31a2a to: /var/lib/boinc-client/slots/0/ic_ancil_12187959.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12187959.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_86f388c690ddc2891e1b57b105f43e2f to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187959.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187959.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187959.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187959.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12187959.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12187959.xml
-- Model is restarting --
Adjusting last_iter, 2800, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12187959.xml
last_cpu_time: 53423
upload_file_number: 116
last_iter: 2783
last_upload: 10022400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740
The child process has been launched with process id: 3555407
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3555407:3555407] [20230119:050646:1674130006:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:07:21 STEP 2784 H=2784:00 +CPU= 31.557
  05:07:40 STEP 2785 H=2785:00 +CPU= 18.555
  05:07:59 STEP 2786 H=2786:00 +CPU= 19.409
  05:08:29 STEP 2787 H=2787:00 +CPU= 29.616
  05:08:48 STEP 2788 H=2788:00 +CPU= 18.451
  05:09:06 STEP 2789 H=2789:00 +CPU= 18.143
  05:09:34 STEP 2790 H=2790:00 +CPU= 27.074
  05:09:52 STEP 2791 H=2791:00 +CPU= 17.553
  05:10:10 STEP 2792 H=2792:00 +CPU= 17.759
  05:10:40 STEP 2793 H=2793:00 +CPU= 29.495
  05:11:00 STEP 2794 H=2794:00 +CPU= 19.560
  05:11:17 STEP 2795 H=2795:00 +CPU= 17.318
  05:11:49 STEP 2796 H=2796:00 +CPU= 30.854
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002796
  05:12:06 STEP 2797 H=2797:00 +CPU= 17.746
  05:12:26 STEP 2798 H=2798:00 +CPU= 18.879
  05:12:53 STEP 2799 H=2799:00 +CPU= 26.750
  05:13:11 STEP 2800 H=2800:00 +CPU= 17.990
  05:13:30 STEP 2801 H=2801:00 +CPU= 18.851
  05:13:57 STEP 2802 H=2802:00 +CPU= 26.867
  05:14:15 STEP 2803 H=2803:00 +CPU= 17.433
  05:14:33 STEP 2804 H=2804:00 +CPU= 17.399
  05:15:02 STEP 2805 H=2805:00 +CPU= 28.958
  05:15:22 STEP 2806 H=2806:00 +CPU= 19.516
  05:15:42 STEP 2807 H=2807:00 +CPU= 19.569
  05:16:14 STEP 2808 H=2808:00 +CPU= 32.123
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_116.zip
Uploading the intermediate file: upload_file_116.zip
  05:16:33 STEP 2809 H=2809:00 +CPU= 18.676
  05:16:51 STEP 2810 H=2810:00 +CPU= 17.714
  05:17:17 STEP 2811 H=2811:00 +CPU= 25.450
  05:17:36 STEP 2812 H=2812:00 +CPU= 17.959
  05:17:54 STEP 2813 H=2813:00 +CPU= 18.017
  05:18:21 STEP 2814 H=2814:00 +CPU= 27.663
  05:18:39 STEP 2815 H=2815:00 +CPU= 17.256
  05:18:58 STEP 2816 H=2816:00 +CPU= 18.416
  05:19:25 STEP 2817 H=2817:00 +CPU= 26.307
  05:19:43 STEP 2818 H=2818:00 +CPU= 18.609
  05:20:02 STEP 2819 H=2819:00 +CPU= 18.602
  05:20:32 STEP 2820 H=2820:00 +CPU= 29.633
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002820
  05:20:50 STEP 2821 H=2821:00 +CPU= 17.490
  05:21:08 STEP 2822 H=2822:00 +CPU= 18.229
  05:21:35 STEP 2823 H=2823:00 +CPU= 26.517
  05:21:52 STEP 2824 H=2824:00 +CPU= 17.111
  05:22:09 STEP 2825 H=2825:00 +CPU= 16.751
  05:22:38 STEP 2826 H=2826:00 +CPU= 28.036
  05:22:55 STEP 2827 H=2827:00 +CPU= 17.028
  05:23:14 STEP 2828 H=2828:00 +CPU= 18.665
  05:23:42 STEP 2829 H=2829:00 +CPU= 27.902
  05:24:01 STEP 2830 H=2830:00 +CPU= 18.688
  05:24:19 STEP 2831 H=2831:00 +CPU= 18.091
  05:24:49 STEP 2832 H=2832:00 +CPU= 29.771
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_117.zip
Uploading the intermediate file: upload_file_117.zip
  05:25:08 STEP 2833 H=2833:00 +CPU= 18.130
  05:25:27 STEP 2834 H=2834:00 +CPU= 19.317
  05:25:53 STEP 2835 H=2835:00 +CPU= 25.436
  05:26:12 STEP 2836 H=2836:00 +CPU= 18.412
  05:26:31 STEP 2837 H=2837:00 +CPU= 19.221
  05:26:59 STEP 2838 H=2838:00 +CPU= 27.108
  05:27:16 STEP 2839 H=2839:00 +CPU= 16.993
  05:27:34 STEP 2840 H=2840:00 +CPU= 17.747
  05:28:01 STEP 2841 H=2841:00 +CPU= 27.091
  05:28:20 STEP 2842 H=2842:00 +CPU= 17.917
  05:28:37 STEP 2843 H=2843:00 +CPU= 17.153
  05:29:06 STEP 2844 H=2844:00 +CPU= 28.708
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002844
  05:29:25 STEP 2845 H=2845:00 +CPU= 18.625
  05:29:43 STEP 2846 H=2846:00 +CPU= 17.632
  05:30:10 STEP 2847 H=2847:00 +CPU= 26.624
  05:30:28 STEP 2848 H=2848:00 +CPU= 18.505
  05:30:47 STEP 2849 H=2849:00 +CPU= 18.637
  05:31:14 STEP 2850 H=2850:00 +CPU= 26.165
  05:31:32 STEP 2851 H=2851:00 +CPU= 18.384
  05:31:51 STEP 2852 H=2852:00 +CPU= 18.509
  05:32:18 STEP 2853 H=2853:00 +CPU= 26.470
  05:32:35 STEP 2854 H=2854:00 +CPU= 16.624
  05:32:53 STEP 2855 H=2855:00 +CPU= 17.917
  05:33:26 STEP 2856 H=2856:00 +CPU= 33.131
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_118.zip
Uploading the intermediate file: upload_file_118.zip
  05:33:43 STEP 2857 H=2857:00 +CPU= 16.071
  05:34:00 STEP 2858 H=2858:00 +CPU= 16.911
  05:34:27 STEP 2859 H=2859:00 +CPU= 26.865
  05:34:45 STEP 2860 H=2860:00 +CPU= 18.120
  05:35:01 STEP 2861 H=2861:00 +CPU= 15.681
  05:35:24 STEP 2862 H=2862:00 +CPU= 23.052
  05:35:44 STEP 2863 H=2863:00 +CPU= 18.977
  05:36:04 STEP 2864 H=2864:00 +CPU= 20.021
  05:36:31 STEP 2865 H=2865:00 +CPU= 26.566
  05:36:49 STEP 2866 H=2866:00 +CPU= 17.769
  05:37:08 STEP 2867 H=2867:00 +CPU= 19.018
  05:37:39 STEP 2868 H=2868:00 +CPU= 30.268
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002868
  05:37:56 STEP 2869 H=2869:00 +CPU= 17.014
  05:38:13 STEP 2870 H=2870:00 +CPU= 16.346
  05:38:40 STEP 2871 H=2871:00 +CPU= 27.084
  05:38:59 STEP 2872 H=2872:00 +CPU= 18.751
  05:39:17 STEP 2873 H=2873:00 +CPU= 17.487
  05:39:43 STEP 2874 H=2874:00 +CPU= 26.459
  05:40:03 STEP 2875 H=2875:00 +CPU= 18.940
  05:40:24 STEP 2876 H=2876:00 +CPU= 20.750
  05:40:52 STEP 2877 H=2877:00 +CPU= 28.090
  05:41:10 STEP 2878 H=2878:00 +CPU= 17.903
  05:41:29 STEP 2879 H=2879:00 +CPU= 18.567
  05:42:00 STEP 2880 H=2880:00 +CPU= 30.586
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_119.zip
Uploading the intermediate file: upload_file_119.zip
  05:42:17 STEP 2881 H=2881:00 +CPU= 16.598
  05:42:34 STEP 2882 H=2882:00 +CPU= 16.637
  05:43:01 STEP 2883 H=2883:00 +CPU= 26.716
  05:43:20 STEP 2884 H=2884:00 +CPU= 18.994
  05:43:39 STEP 2885 H=2885:00 +CPU= 18.610
  05:44:07 STEP 2886 H=2886:00 +CPU= 27.012
  05:44:27 STEP 2887 H=2887:00 +CPU= 20.190
  05:44:46 STEP 2888 H=2888:00 +CPU= 19.426
  05:45:14 STEP 2889 H=2889:00 +CPU= 27.494
  05:45:32 STEP 2890 H=2890:00 +CPU= 17.329
  05:45:49 STEP 2891 H=2891:00 +CPU= 16.808
  05:46:21 STEP 2892 H=2892:00 +CPU= 31.154
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002892
  05:46:39 STEP 2893 H=2893:00 +CPU= 18.076
  05:46:57 STEP 2894 H=2894:00 +CPU= 17.784
  05:47:24 STEP 2895 H=2895:00 +CPU= 26.452
  05:47:43 STEP 2896 H=2896:00 +CPU= 18.997
  05:48:02 STEP 2897 H=2897:00 +CPU= 18.492
  05:48:27 STEP 2898 H=2898:00 +CPU= 24.976
  05:48:42 STEP 2899 H=2899:00 +CPU= 15.614
  05:49:01 STEP 2900 H=2900:00 +CPU= 18.750
  05:49:31 STEP 2901 H=2901:00 +CPU= 28.863
  05:49:48 STEP 2902 H=2902:00 +CPU= 17.025
  05:50:06 STEP 2903 H=2903:00 +CPU= 17.227
  05:50:37 STEP 2904 H=2904:00 +CPU= 31.394
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_120.zip
Uploading the intermediate file: upload_file_120.zip
  05:50:56 STEP 2905 H=2905:00 +CPU= 18.120
  05:51:12 STEP 2906 H=2906:00 +CPU= 16.434
  05:51:36 STEP 2907 H=2907:00 +CPU= 23.594
  05:51:55 STEP 2908 H=2908:00 +CPU= 18.794
  05:52:16 STEP 2909 H=2909:00 +CPU= 19.865
  05:52:43 STEP 2910 H=2910:00 +CPU= 27.358
  05:53:01 STEP 2911 H=2911:00 +CPU= 17.512
  05:53:20 STEP 2912 H=2912:00 +CPU= 18.511
  05:53:46 STEP 2913 H=2913:00 +CPU= 26.419
  05:54:05 STEP 2914 H=2914:00 +CPU= 18.230
  05:54:24 STEP 2915 H=2915:00 +CPU= 18.696
  05:54:55 STEP 2916 H=2916:00 +CPU= 31.302
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002916
  05:55:12 STEP 2917 H=2917:00 +CPU= 16.463
  05:55:30 STEP 2918 H=2918:00 +CPU= 17.915
  05:55:57 STEP 2919 H=2919:00 +CPU= 27.055
  05:56:16 STEP 2920 H=2920:00 +CPU= 18.716
  05:56:33 STEP 2921 H=2921:00 +CPU= 16.402
  05:56:57 STEP 2922 H=2922:00 +CPU= 23.660
  05:57:14 STEP 2923 H=2923:00 +CPU= 17.214
  05:57:33 STEP 2924 H=2924:00 +CPU= 19.165
  05:58:01 STEP 2925 H=2925:00 +CPU= 27.302
  05:58:19 STEP 2926 H=2926:00 +CPU= 17.681
  05:58:36 STEP 2927 H=2927:00 +CPU= 17.087
  05:59:08 STEP 2928 H=2928:00 +CPU= 31.040
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_121.zip
Uploading the intermediate file: upload_file_121.zip
  05:59:28 STEP 2929 H=2929:00 +CPU= 19.404
  05:59:47 STEP 2930 H=2930:00 +CPU= 19.182
  06:00:13 STEP 2931 H=2931:00 +CPU= 25.535
  06:00:32 STEP 2932 H=2932:00 +CPU= 18.802
  06:00:50 STEP 2933 H=2933:00 +CPU= 18.129
  06:01:18 STEP 2934 H=2934:00 +CPU= 27.611
  06:01:34 STEP 2935 H=2935:00 +CPU= 15.624
  06:01:51 STEP 2936 H=2936:00 +CPU= 16.664
  06:02:19 STEP 2937 H=2937:00 +CPU= 27.232
  06:02:38 STEP 2938 H=2938:00 +CPU= 19.088
  06:02:56 STEP 2939 H=2939:00 +CPU= 17.668
  06:03:25 STEP 2940 H=2940:00 +CPU= 28.444
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002940
  06:03:43 STEP 2941 H=2941:00 +CPU= 18.510
  06:04:03 STEP 2942 H=2942:00 +CPU= 19.403
  06:04:28 STEP 2943 H=2943:00 +CPU= 24.196
  06:04:44 STEP 2944 H=2944:00 +CPU= 16.086
  06:05:03 STEP 2945 H=2945:00 +CPU= 18.762
  06:05:33 STEP 2946 H=2946:00 +CPU= 29.818
  06:05:49 STEP 2947 H=2947:00 +CPU= 16.100
  06:06:06 STEP 2948 H=2948:00 +CPU= 16.892
  06:06:36 STEP 2949 H=2949:00 +CPU= 29.123
  06:06:55 STEP 2950 H=2950:00 +CPU= 18.377
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0315
(argv4) batchid: 971
(argv5) wuid: 12187959
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5701fb94b96cc242319fb9dac33d2dc to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0315_2002050100_123_971_12187959.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0315_2002050100_123_971_12187959.zip
ic_ancil_file: ic_ancil_12187959
ifsdata_file: ifsdata_12187959
climate_data_file: clim_data_12187959
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2cbd7f649367332f90b9f547dcc31a2a to: /var/lib/boinc-client/slots/0/ic_ancil_12187959.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12187959.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_86f388c690ddc2891e1b57b105f43e2f to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187959.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187959.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187959.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187959.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12187959.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12187959.xml
-- Model is restarting --
Adjusting last_iter, 2951, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12187959.xml
last_cpu_time: 56985
upload_file_number: 122
last_iter: 2927
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740
The child process has been launched with process id: 3565968
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3565968:3565968] [20230119:060808:1674133688:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:08:42 STEP 2928 H=2928:00 +CPU= 31.021
  06:09:01 STEP 2929 H=2929:00 +CPU= 18.550
  06:09:20 STEP 2930 H=2930:00 +CPU= 18.640
  06:09:47 STEP 2931 H=2931:00 +CPU= 27.129
  06:10:05 STEP 2932 H=2932:00 +CPU= 17.385
  06:10:25 STEP 2933 H=2933:00 +CPU= 19.439
  06:10:53 STEP 2934 H=2934:00 +CPU= 27.682
  06:11:10 STEP 2935 H=2935:00 +CPU= 17.321
  06:11:28 STEP 2936 H=2936:00 +CPU= 17.574
  06:11:57 STEP 2937 H=2937:00 +CPU= 28.910
  06:12:15 STEP 2938 H=2938:00 +CPU= 17.496
  06:12:32 STEP 2939 H=2939:00 +CPU= 16.852
  06:13:02 STEP 2940 H=2940:00 +CPU= 29.112
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002940
  06:13:21 STEP 2941 H=2941:00 +CPU= 18.909
  06:13:41 STEP 2942 H=2942:00 +CPU= 19.521
  06:14:06 STEP 2943 H=2943:00 +CPU= 25.269
  06:14:24 STEP 2944 H=2944:00 +CPU= 17.712
  06:14:44 STEP 2945 H=2945:00 +CPU= 19.351
  06:15:13 STEP 2946 H=2946:00 +CPU= 28.400
  06:15:28 STEP 2947 H=2947:00 +CPU= 15.704
  06:15:47 STEP 2948 H=2948:00 +CPU= 18.311
  06:16:16 STEP 2949 H=2949:00 +CPU= 28.208
  06:16:32 STEP 2950 H=2950:00 +CPU= 16.156
  06:16:49 STEP 2951 H=2951:00 +CPU= 17.022
  06:17:23 STEP 2952 H=2952:00 +CPU= 32.797
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187959/ICMGGhq0f+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0315_2002050100_123_971_12187959_1_r1370360740_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
06:20:46 (3565964): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:28:40 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 10,623,600 57,532 0.0054
09 May 2023 16:28:12 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 7,862,400 39,873 0.0051
09 May 2023 16:28:06 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 6,998,400 35,660 0.0051
09 May 2023 16:27:55 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 6,134,400 31,448 0.0051
09 May 2023 16:27:49 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 5,270,400 27,246 0.0052
09 May 2023 16:27:43 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 4,406,400 23,033 0.0052
09 May 2023 16:27:39 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 3,542,400 18,811 0.0053
09 May 2023 16:27:33 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 2,678,400 14,598 0.0055
09 May 2023 16:27:28 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 1,814,400 10,341 0.0057
09 May 2023 16:27:18 1532237 22299554 oifs_43r3_ps_0315_2002050100_123_971_12187959_1 950,400 5,919 0.0062


©2024 cpdn.org