climateprediction.net home page
Task 22266388

Task 22266388

Name oifs_43r3_ps_0715_1995050100_123_964_12181359_0
Workunit 12181359
Created 22 Dec 2022, 9:04:42 UTC
Sent 22 Dec 2022, 9:08:48 UTC
Report deadline 21 Jan 2023, 9:08:48 UTC
Received 1 Jan 2023, 19:52:13 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1521081
Run time 1 days 8 hours 21 min 59 sec
CPU time 1 days 6 hours 21 min 9 sec
Validate state Invalid
Credit 13,184.64
Device peak FLOPS 5.44 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.39 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,289.56 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
672578124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:56383:56383] [20230101:140204:1672578124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:56383:56383] [20230101:140204:1672578124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:56383:56383] [20230101:140204:1672578124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:56383:56383] [20230101:140204:1672578124:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:03:19 STEP 2616 H=2616:00 +CPU= 67.292
  14:04:05 STEP 2617 H=2617:00 +CPU= 44.686
  14:04:56 STEP 2618 H=2618:00 +CPU= 49.924
  14:06:15 STEP 2619 H=2619:00 +CPU= 77.610
  14:07:02 STEP 2620 H=2620:00 +CPU= 45.325
  14:07:45 STEP 2621 H=2621:00 +CPU= 42.326
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:14:08 STEP 2622 H=2622:00 +CPU= 57.591
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:16:03 STEP 2623 H=2623:00 +CPU= 34.340
  14:16:37 STEP 2624 H=2624:00 +CPU= 33.718
  14:17:35 STEP 2625 H=2625:00 +CPU= 56.934
  14:18:16 STEP 2626 H=2626:00 +CPU= 40.481
  14:18:52 STEP 2627 H=2627:00 +CPU= 36.036
  14:19:46 STEP 2628 H=2628:00 +CPU= 53.430
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002628
  14:20:20 STEP 2629 H=2629:00 +CPU= 33.379
  14:21:07 STEP 2630 H=2630:00 +CPU= 45.912
  14:22:28 STEP 2631 H=2631:00 +CPU= 79.144
  14:23:15 STEP 2632 H=2632:00 +CPU= 46.735
  14:24:03 STEP 2633 H=2633:00 +CPU= 46.750
  14:25:12 STEP 2634 H=2634:00 +CPU= 68.788
  14:25:59 STEP 2635 H=2635:00 +CPU= 46.403
  14:26:46 STEP 2636 H=2636:00 +CPU= 46.182
  14:27:56 STEP 2637 H=2637:00 +CPU= 68.249
  14:28:48 STEP 2638 H=2638:00 +CPU= 51.480
  14:29:39 STEP 2639 H=2639:00 +CPU= 49.090
  14:31:02 STEP 2640 H=2640:00 +CPU= 81.602
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_109.zip
Uploading the intermediate file: upload_file_109.zip
  14:31:44 STEP 2641 H=2641:00 +CPU= 41.299
  14:32:24 STEP 2642 H=2642:00 +CPU= 39.583
  14:33:22 STEP 2643 H=2643:00 +CPU= 57.537
  14:34:02 STEP 2644 H=2644:00 +CPU= 38.750
  14:34:40 STEP 2645 H=2645:00 +CPU= 36.426
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:36:45 STEP 2646 H=2646:00 +CPU= 51.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:42:19 STEP 2647 H=2647:00 +CPU= 31.487
  14:42:58 STEP 2648 H=2648:00 +CPU= 30.892
  14:43:58 STEP 2649 H=2649:00 +CPU= 47.128
  14:44:41 STEP 2650 H=2650:00 +CPU= 34.099
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:45:34 STEP 2651 H=2651:00 +CPU= 34.549
  14:46:38 STEP 2652 H=2652:00 +CPU= 60.106
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002652
  14:47:15 STEP 2653 H=2653:00 +CPU= 35.445
  14:47:54 STEP 2654 H=2654:00 +CPU= 37.784
  14:48:56 STEP 2655 H=2655:00 +CPU= 57.907
  14:49:35 STEP 2656 H=2656:00 +CPU= 37.983
  14:50:09 STEP 2657 H=2657:00 +CPU= 32.518
  14:51:00 STEP 2658 H=2658:00 +CPU= 47.931
  14:51:27 STEP 2659 H=2659:00 +CPU= 26.818
  14:51:55 STEP 2660 H=2660:00 +CPU= 26.764
  14:52:36 STEP 2661 H=2661:00 +CPU= 39.875
  14:53:05 STEP 2662 H=2662:00 +CPU= 26.859
  14:53:33 STEP 2663 H=2663:00 +CPU= 27.240
  14:54:25 STEP 2664 H=2664:00 +CPU= 47.791
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_110.zip
Uploading the intermediate file: upload_file_110.zip
  14:54:49 STEP 2665 H=2665:00 +CPU= 19.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:57:24 STEP 2666 H=2666:00 +CPU= 27.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:58:26 STEP 2667 H=2667:00 +CPU= 44.750
  14:59:02 STEP 2668 H=2668:00 +CPU= 30.639
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:12:24 STEP 2669 H=2669:00 +CPU= 28.897
  15:13:20 STEP 2670 H=2670:00 +CPU= 47.995
  15:13:53 STEP 2671 H=2671:00 +CPU= 28.197
  15:14:25 STEP 2672 H=2672:00 +CPU= 28.255
  15:15:14 STEP 2673 H=2673:00 +CPU= 42.520
  15:15:46 STEP 2674 H=2674:00 +CPU= 28.052
  15:16:18 STEP 2675 H=2675:00 +CPU= 27.513
  15:17:12 STEP 2676 H=2676:00 +CPU= 47.116
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002676
  15:17:44 STEP 2677 H=2677:00 +CPU= 27.632
  15:18:17 STEP 2678 H=2678:00 +CPU= 28.631
  15:19:05 STEP 2679 H=2679:00 +CPU= 42.181
  15:19:40 STEP 2680 H=2680:00 +CPU= 29.758
  15:20:16 STEP 2681 H=2681:00 +CPU= 30.703
  15:21:12 STEP 2682 H=2682:00 +CPU= 48.249
  15:21:48 STEP 2683 H=2683:00 +CPU= 30.881
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:26:08 STEP 2684 H=2684:00 +CPU= 31.085
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:30:36 STEP 2685 H=2685:00 +CPU= 45.081
  15:31:10 STEP 2686 H=2686:00 +CPU= 29.863
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0715
(argv4) batchid: 964
(argv5) wuid: 12181359
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_bf8807d3ab1da18b8cda64fdcb91ea4d to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip
ic_ancil_file: ic_ancil_12181359
ifsdata_file: ifsdata_12181359
climate_data_file: clim_data_12181359
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e33f194d446fdf5176278169e33b28eb to: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ec633de182e05f42f6929883a755e89f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12181359.xml
-- Model is restarting --
Adjusting last_iter, 2687, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml
last_cpu_time: 99715
upload_file_number: 111
last_iter: 2663
last_upload: 9590400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501
The child process has been launched with process id: 69144
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:01:06 STEP 2664 H=2664:00 +CPU= 56.735
  16:01:41 STEP 2665 H=2665:00 +CPU= 31.450
  16:02:21 STEP 2666 H=2666:00 +CPU= 35.154
  16:03:20 STEP 2667 H=2667:00 +CPU= 51.585
  16:03:55 STEP 2668 H=2668:00 +CPU= 31.127
  16:04:29 STEP 2669 H=2669:00 +CPU= 31.735
  16:05:22 STEP 2670 H=2670:00 +CPU= 48.671
  16:05:56 STEP 2671 H=2671:00 +CPU= 31.199
  16:06:29 STEP 2672 H=2672:00 +CPU= 31.304
  16:07:21 STEP 2673 H=2673:00 +CPU= 47.343
  16:07:53 STEP 2674 H=2674:00 +CPU= 29.961
  16:08:28 STEP 2675 H=2675:00 +CPU= 32.320
  16:09:27 STEP 2676 H=2676:00 +CPU= 53.905
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002676
  16:10:00 STEP 2677 H=2677:00 +CPU= 31.197
  16:10:35 STEP 2678 H=2678:00 +CPU= 32.534
  16:11:27 STEP 2679 H=2679:00 +CPU= 47.398
  16:11:58 STEP 2680 H=2680:00 +CPU= 29.020
  16:12:29 STEP 2681 H=2681:00 +CPU= 28.818
  16:13:16 STEP 2682 H=2682:00 +CPU= 44.066
  16:13:48 STEP 2683 H=2683:00 +CPU= 28.830
  16:14:22 STEP 2684 H=2684:00 +CPU= 31.614
  16:15:09 STEP 2685 H=2685:00 +CPU= 43.795
  16:15:41 STEP 2686 H=2686:00 +CPU= 29.479
  16:16:12 STEP 2687 H=2687:00 +CPU= 28.789
  16:17:08 STEP 2688 H=2688:00 +CPU= 51.993
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_111.zip
Uploading the intermediate file: upload_file_111.zip
  16:17:43 STEP 2689 H=2689:00 +CPU= 33.367
  16:18:20 STEP 2690 H=2690:00 +CPU= 34.126
  16:19:13 STEP 2691 H=2691:00 +CPU= 49.168
  16:19:49 STEP 2692 H=2692:00 +CPU= 33.782
  16:20:21 STEP 2693 H=2693:00 +CPU= 29.265
  16:21:13 STEP 2694 H=2694:00 +CPU= 48.942
  16:21:47 STEP 2695 H=2695:00 +CPU= 31.756
  16:22:24 STEP 2696 H=2696:00 +CPU= 34.180
  16:23:16 STEP 2697 H=2697:00 +CPU= 48.918
  16:23:52 STEP 2698 H=2698:00 +CPU= 33.914
  16:24:27 STEP 2699 H=2699:00 +CPU= 31.681
  16:25:19 STEP 2700 H=2700:00 +CPU= 49.504
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002700
  16:25:49 STEP 2701 H=2701:00 +CPU= 27.815
  16:26:22 STEP 2702 H=2702:00 +CPU= 31.194
  16:27:12 STEP 2703 H=2703:00 +CPU= 46.221
  16:27:45 STEP 2704 H=2704:00 +CPU= 30.630
  16:28:14 STEP 2705 H=2705:00 +CPU= 28.006
  16:29:01 STEP 2706 H=2706:00 +CPU= 44.265
  16:29:30 STEP 2707 H=2707:00 +CPU= 27.005
  16:30:01 STEP 2708 H=2708:00 +CPU= 28.781
  16:30:46 STEP 2709 H=2709:00 +CPU= 42.562
  16:31:17 STEP 2710 H=2710:00 +CPU= 29.188
  16:31:48 STEP 2711 H=2711:00 +CPU= 28.967
  16:32:41 STEP 2712 H=2712:00 +CPU= 49.978
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_112.zip
Uploading the intermediate file: upload_file_112.zip
  16:33:11 STEP 2713 H=2713:00 +CPU= 27.982
  16:33:41 STEP 2714 H=2714:00 +CPU= 28.268
  16:34:27 STEP 2715 H=2715:00 +CPU= 42.824
  16:34:58 STEP 2716 H=2716:00 +CPU= 29.336
  16:35:28 STEP 2717 H=2717:00 +CPU= 28.158
  16:36:13 STEP 2718 H=2718:00 +CPU= 42.700
  16:36:43 STEP 2719 H=2719:00 +CPU= 28.035
  16:37:13 STEP 2720 H=2720:00 +CPU= 27.804
  16:37:56 STEP 2721 H=2721:00 +CPU= 40.565
  16:38:25 STEP 2722 H=2722:00 +CPU= 27.177
  16:38:52 STEP 2723 H=2723:00 +CPU= 26.142
  16:39:39 STEP 2724 H=2724:00 +CPU= 44.102
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002724
  16:40:08 STEP 2725 H=2725:00 +CPU= 26.983
  16:40:36 STEP 2726 H=2726:00 +CPU= 26.876
  16:41:19 STEP 2727 H=2727:00 +CPU= 40.627
  16:41:48 STEP 2728 H=2728:00 +CPU= 27.346
  16:42:16 STEP 2729 H=2729:00 +CPU= 26.454
  16:43:00 STEP 2730 H=2730:00 +CPU= 40.945
  16:43:29 STEP 2731 H=2731:00 +CPU= 26.159
  16:44:01 STEP 2732 H=2732:00 +CPU= 27.663
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:47:30 STEP 2733 H=2733:00 +CPU= 41.484
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:01:07 STEP 2734 H=2734:00 +CPU= 28.190
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:07:18 STEP 2735 H=2735:00 +CPU= 29.059
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0715
(argv4) batchid: 964
(argv5) wuid: 12181359
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_bf8807d3ab1da18b8cda64fdcb91ea4d to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip
ic_ancil_file: ic_ancil_12181359
ifsdata_file: ifsdata_12181359
climate_data_file: clim_data_12181359
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e33f194d446fdf5176278169e33b28eb to: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ec633de182e05f42f6929883a755e89f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12181359.xml
-- Model is restarting --
Adjusting last_iter, 2736, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml
last_cpu_time: 102283
upload_file_number: 113
last_iter: 2735
last_upload: 9763200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501
The child process has been launched with process id: 72744
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:22:38 STEP 2736 H=2736:00 +CPU= 40.134
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_113.zip
Uploading the intermediate file: upload_file_113.zip
  17:23:01 STEP 2737 H=2737:00 +CPU= 22.590
  17:23:25 STEP 2738 H=2738:00 +CPU= 22.642
  17:24:02 STEP 2739 H=2739:00 +CPU= 36.421
  17:24:30 STEP 2740 H=2740:00 +CPU= 25.872
  17:25:02 STEP 2741 H=2741:00 +CPU= 29.869
  17:25:55 STEP 2742 H=2742:00 +CPU= 48.441
  17:26:28 STEP 2743 H=2743:00 +CPU= 29.418
  17:27:04 STEP 2744 H=2744:00 +CPU= 32.716
  17:27:57 STEP 2745 H=2745:00 +CPU= 47.827
  17:28:32 STEP 2746 H=2746:00 +CPU= 31.968
  17:29:05 STEP 2747 H=2747:00 +CPU= 29.657
  17:30:03 STEP 2748 H=2748:00 +CPU= 53.783
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002748
  17:30:37 STEP 2749 H=2749:00 +CPU= 30.137
  17:31:10 STEP 2750 H=2750:00 +CPU= 30.439
  17:32:02 STEP 2751 H=2751:00 +CPU= 46.802
  17:32:34 STEP 2752 H=2752:00 +CPU= 29.078
  17:33:08 STEP 2753 H=2753:00 +CPU= 30.747
  17:34:02 STEP 2754 H=2754:00 +CPU= 49.474
  17:34:36 STEP 2755 H=2755:00 +CPU= 31.289
  17:35:11 STEP 2756 H=2756:00 +CPU= 32.174
  17:36:01 STEP 2757 H=2757:00 +CPU= 46.128
  17:36:34 STEP 2758 H=2758:00 +CPU= 29.347
  17:37:05 STEP 2759 H=2759:00 +CPU= 28.872
  17:38:00 STEP 2760 H=2760:00 +CPU= 50.465
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_114.zip
Uploading the intermediate file: upload_file_114.zip
  17:38:31 STEP 2761 H=2761:00 +CPU= 27.968
  17:39:03 STEP 2762 H=2762:00 +CPU= 30.252
  17:39:51 STEP 2763 H=2763:00 +CPU= 43.966
  17:40:22 STEP 2764 H=2764:00 +CPU= 29.564
  17:40:53 STEP 2765 H=2765:00 +CPU= 27.758
  17:41:41 STEP 2766 H=2766:00 +CPU= 44.158
  17:42:15 STEP 2767 H=2767:00 +CPU= 31.158
  17:42:50 STEP 2768 H=2768:00 +CPU= 32.325
  17:43:44 STEP 2769 H=2769:00 +CPU= 49.064
  17:44:21 STEP 2770 H=2770:00 +CPU= 33.477
  17:44:54 STEP 2771 H=2771:00 +CPU= 30.578
  17:45:51 STEP 2772 H=2772:00 +CPU= 52.494
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002772
  17:46:25 STEP 2773 H=2773:00 +CPU= 30.761
  17:47:01 STEP 2774 H=2774:00 +CPU= 32.710
  17:47:54 STEP 2775 H=2775:00 +CPU= 47.714
  17:48:29 STEP 2776 H=2776:00 +CPU= 32.349
  17:49:04 STEP 2777 H=2777:00 +CPU= 31.795
  17:49:54 STEP 2778 H=2778:00 +CPU= 45.598
  17:50:24 STEP 2779 H=2779:00 +CPU= 27.822
  17:50:57 STEP 2780 H=2780:00 +CPU= 30.354
  17:51:50 STEP 2781 H=2781:00 +CPU= 48.727
  17:52:24 STEP 2782 H=2782:00 +CPU= 30.782
  17:52:53 STEP 2783 H=2783:00 +CPU= 27.956
  17:53:49 STEP 2784 H=2784:00 +CPU= 49.914
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_115.zip
Uploading the intermediate file: upload_file_115.zip
  17:54:18 STEP 2785 H=2785:00 +CPU= 26.468
  17:54:48 STEP 2786 H=2786:00 +CPU= 27.733
  17:55:35 STEP 2787 H=2787:00 +CPU= 43.079
  17:56:04 STEP 2788 H=2788:00 +CPU= 27.950
  17:56:34 STEP 2789 H=2789:00 +CPU= 27.204
  17:57:21 STEP 2790 H=2790:00 +CPU= 43.610
  17:57:50 STEP 2791 H=2791:00 +CPU= 27.580
  17:58:20 STEP 2792 H=2792:00 +CPU= 27.747
  17:59:06 STEP 2793 H=2793:00 +CPU= 42.635
  17:59:37 STEP 2794 H=2794:00 +CPU= 28.437
  18:00:07 STEP 2795 H=2795:00 +CPU= 28.052
  18:00:57 STEP 2796 H=2796:00 +CPU= 46.709
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002796
  18:01:27 STEP 2797 H=2797:00 +CPU= 27.639
  18:01:56 STEP 2798 H=2798:00 +CPU= 27.583
  18:02:40 STEP 2799 H=2799:00 +CPU= 40.567
  18:03:08 STEP 2800 H=2800:00 +CPU= 26.342
  18:03:36 STEP 2801 H=2801:00 +CPU= 25.883
  18:04:20 STEP 2802 H=2802:00 +CPU= 40.873
  18:04:48 STEP 2803 H=2803:00 +CPU= 25.596
  18:05:17 STEP 2804 H=2804:00 +CPU= 26.792
  18:06:00 STEP 2805 H=2805:00 +CPU= 40.566
  18:06:29 STEP 2806 H=2806:00 +CPU= 26.390
  18:06:57 STEP 2807 H=2807:00 +CPU= 26.564
  18:07:53 STEP 2808 H=2808:00 +CPU= 51.504
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_116.zip
Uploading the intermediate file: upload_file_116.zip
  18:08:24 STEP 2809 H=2809:00 +CPU= 28.281
  18:08:57 STEP 2810 H=2810:00 +CPU= 30.548
  18:09:45 STEP 2811 H=2811:00 +CPU= 43.619
  18:10:14 STEP 2812 H=2812:00 +CPU= 26.657
  18:10:41 STEP 2813 H=2813:00 +CPU= 25.933
  18:11:26 STEP 2814 H=2814:00 +CPU= 41.443
  18:11:53 STEP 2815 H=2815:00 +CPU= 26.124
  18:12:22 STEP 2816 H=2816:00 +CPU= 26.948
  18:13:05 STEP 2817 H=2817:00 +CPU= 39.711
  18:13:34 STEP 2818 H=2818:00 +CPU= 27.011
  18:14:02 STEP 2819 H=2819:00 +CPU= 25.964
  18:14:50 STEP 2820 H=2820:00 +CPU= 44.485
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002820
  18:15:20 STEP 2821 H=2821:00 +CPU= 26.963
  18:15:52 STEP 2822 H=2822:00 +CPU= 29.328
  18:16:40 STEP 2823 H=2823:00 +CPU= 44.299
  18:17:12 STEP 2824 H=2824:00 +CPU= 29.817
  18:17:42 STEP 2825 H=2825:00 +CPU= 28.364
  18:18:26 STEP 2826 H=2826:00 +CPU= 41.019
  18:18:54 STEP 2827 H=2827:00 +CPU= 26.001
  18:19:23 STEP 2828 H=2828:00 +CPU= 26.415
  18:20:06 STEP 2829 H=2829:00 +CPU= 40.236
  18:20:34 STEP 2830 H=2830:00 +CPU= 26.863
  18:21:03 STEP 2831 H=2831:00 +CPU= 26.913
  18:21:54 STEP 2832 H=2832:00 +CPU= 47.296
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_117.zip
Uploading the intermediate file: upload_file_117.zip
  18:22:22 STEP 2833 H=2833:00 +CPU= 25.701
  18:22:50 STEP 2834 H=2834:00 +CPU= 26.687
  18:23:36 STEP 2835 H=2835:00 +CPU= 42.392
  18:24:08 STEP 2836 H=2836:00 +CPU= 29.526
  18:24:39 STEP 2837 H=2837:00 +CPU= 28.381
  18:25:26 STEP 2838 H=2838:00 +CPU= 44.852
  18:25:56 STEP 2839 H=2839:00 +CPU= 27.286
  18:26:25 STEP 2840 H=2840:00 +CPU= 27.082
  18:27:07 STEP 2841 H=2841:00 +CPU= 40.097
  18:27:36 STEP 2842 H=2842:00 +CPU= 27.130
  18:28:04 STEP 2843 H=2843:00 +CPU= 26.181
  18:28:53 STEP 2844 H=2844:00 +CPU= 45.039
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002844
  18:29:20 STEP 2845 H=2845:00 +CPU= 26.059
  18:29:50 STEP 2846 H=2846:00 +CPU= 26.827
  18:30:34 STEP 2847 H=2847:00 +CPU= 40.573
  18:31:03 STEP 2848 H=2848:00 +CPU= 26.860
  18:31:33 STEP 2849 H=2849:00 +CPU= 27.445
  18:32:22 STEP 2850 H=2850:00 +CPU= 45.242
  18:32:51 STEP 2851 H=2851:00 +CPU= 26.971
  18:33:20 STEP 2852 H=2852:00 +CPU= 26.708
  18:34:04 STEP 2853 H=2853:00 +CPU= 40.518
  18:34:32 STEP 2854 H=2854:00 +CPU= 26.850
  18:35:01 STEP 2855 H=2855:00 +CPU= 26.640
  18:35:51 STEP 2856 H=2856:00 +CPU= 47.548
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_118.zip
Uploading the intermediate file: upload_file_118.zip
  18:36:19 STEP 2857 H=2857:00 +CPU= 25.977
  18:36:48 STEP 2858 H=2858:00 +CPU= 26.717
  18:37:31 STEP 2859 H=2859:00 +CPU= 40.265
  18:38:00 STEP 2860 H=2860:00 +CPU= 26.697
  18:38:30 STEP 2861 H=2861:00 +CPU= 28.354
  18:39:19 STEP 2862 H=2862:00 +CPU= 44.716
  18:39:51 STEP 2863 H=2863:00 +CPU= 28.980
  18:40:24 STEP 2864 H=2864:00 +CPU= 29.980
  18:41:08 STEP 2865 H=2865:00 +CPU= 40.942
  18:41:37 STEP 2866 H=2866:00 +CPU= 26.487
  18:42:05 STEP 2867 H=2867:00 +CPU= 26.007
  18:42:53 STEP 2868 H=2868:00 +CPU= 44.906
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002868
  18:43:21 STEP 2869 H=2869:00 +CPU= 26.201
  18:43:51 STEP 2870 H=2870:00 +CPU= 26.939
  18:44:34 STEP 2871 H=2871:00 +CPU= 40.115
  18:45:03 STEP 2872 H=2872:00 +CPU= 26.657
  18:45:31 STEP 2873 H=2873:00 +CPU= 26.269
  18:46:17 STEP 2874 H=2874:00 +CPU= 43.201
  18:46:48 STEP 2875 H=2875:00 +CPU= 28.940
  18:47:20 STEP 2876 H=2876:00 +CPU= 29.735
  18:48:03 STEP 2877 H=2877:00 +CPU= 40.426
  18:48:32 STEP 2878 H=2878:00 +CPU= 26.492
  18:49:00 STEP 2879 H=2879:00 +CPU= 26.696
  18:49:52 STEP 2880 H=2880:00 +CPU= 47.087
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_119.zip
Uploading the intermediate file: upload_file_119.zip
  18:50:20 STEP 2881 H=2881:00 +CPU= 25.752
  18:50:48 STEP 2882 H=2882:00 +CPU= 26.429
  18:51:32 STEP 2883 H=2883:00 +CPU= 40.126
  18:52:00 STEP 2884 H=2884:00 +CPU= 26.920
  18:52:28 STEP 2885 H=2885:00 +CPU= 26.053
  18:53:14 STEP 2886 H=2886:00 +CPU= 42.345
  18:53:45 STEP 2887 H=2887:00 +CPU= 29.014
  18:54:16 STEP 2888 H=2888:00 +CPU= 29.062
  18:54:59 STEP 2889 H=2889:00 +CPU= 40.318
  18:55:27 STEP 2890 H=2890:00 +CPU= 26.568
  18:55:55 STEP 2891 H=2891:00 +CPU= 26.090
  18:56:44 STEP 2892 H=2892:00 +CPU= 45.190
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002892
  18:57:14 STEP 2893 H=2893:00 +CPU= 27.053
  18:57:46 STEP 2894 H=2894:00 +CPU= 29.473
  18:58:34 STEP 2895 H=2895:00 +CPU= 43.398
  18:59:05 STEP 2896 H=2896:00 +CPU= 29.007
  18:59:36 STEP 2897 H=2897:00 +CPU= 28.349
  19:00:29 STEP 2898 H=2898:00 +CPU= 48.265
  19:01:04 STEP 2899 H=2899:00 +CPU= 31.682
  19:01:40 STEP 2900 H=2900:00 +CPU= 32.961
  19:02:34 STEP 2901 H=2901:00 +CPU= 48.875
  19:03:11 STEP 2902 H=2902:00 +CPU= 33.353
  19:03:44 STEP 2903 H=2903:00 +CPU= 29.966
  19:04:40 STEP 2904 H=2904:00 +CPU= 51.461
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_120.zip
Uploading the intermediate file: upload_file_120.zip
  19:05:07 STEP 2905 H=2905:00 +CPU= 26.681
  19:05:35 STEP 2906 H=2906:00 +CPU= 26.266
  19:06:20 STEP 2907 H=2907:00 +CPU= 42.898
  19:07:04 STEP 2908 H=2908:00 +CPU= 41.235
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0715
(argv4) batchid: 964
(argv5) wuid: 12181359
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/10
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_bf8807d3ab1da18b8cda64fdcb91ea4d to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip
ic_ancil_file: ic_ancil_12181359
ifsdata_file: ifsdata_12181359
climate_data_file: clim_data_12181359
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e33f194d446fdf5176278169e33b28eb to: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ec633de182e05f42f6929883a755e89f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip
Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml
Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12181359.xml
-- Model is restarting --
Adjusting last_iter, 2909, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml
last_cpu_time: 108134
upload_file_number: 121
last_iter: 2903
last_upload: 10454400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501
The child process has been launched with process id: 73100
Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:32:18 STEP 2904 H=2904:00 +CPU= 76.504
  19:33:02 STEP 2905 H=2905:00 +CPU= 39.519
  19:33:48 STEP 2906 H=2906:00 +CPU= 41.755
  19:35:00 STEP 2907 H=2907:00 +CPU= 63.211
  19:35:53 STEP 2908 H=2908:00 +CPU= 46.490
  19:36:42 STEP 2909 H=2909:00 +CPU= 43.729
  19:37:58 STEP 2910 H=2910:00 +CPU= 66.632
  19:38:45 STEP 2911 H=2911:00 +CPU= 41.712
  19:39:24 STEP 2912 H=2912:00 +CPU= 35.545
  19:40:28 STEP 2913 H=2913:00 +CPU= 57.220
  19:41:13 STEP 2914 H=2914:00 +CPU= 40.724
  19:41:55 STEP 2915 H=2915:00 +CPU= 37.571
  19:43:01 STEP 2916 H=2916:00 +CPU= 59.100
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002916
  19:43:37 STEP 2917 H=2917:00 +CPU= 33.036
  19:44:15 STEP 2918 H=2918:00 +CPU= 34.007
  19:45:09 STEP 2919 H=2919:00 +CPU= 49.514
  19:45:48 STEP 2920 H=2920:00 +CPU= 36.092
  19:46:26 STEP 2921 H=2921:00 +CPU= 34.219
  19:47:23 STEP 2922 H=2922:00 +CPU= 52.337
  19:48:00 STEP 2923 H=2923:00 +CPU= 33.949
  19:48:38 STEP 2924 H=2924:00 +CPU= 34.679
  19:49:37 STEP 2925 H=2925:00 +CPU= 53.596
  19:50:19 STEP 2926 H=2926:00 +CPU= 37.252
  19:50:57 STEP 2927 H=2927:00 +CPU= 34.881
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:05:02 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 9,417,600 95,344 0.0101
09 May 2023 16:04:10 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 8,553,600 87,089 0.0102
09 May 2023 16:04:02 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 7,689,600 78,789 0.0102
09 May 2023 16:03:54 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 6,825,600 70,770 0.0104
09 May 2023 16:03:49 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 5,961,600 62,342 0.0105
09 May 2023 16:03:40 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 5,097,600 53,469 0.0105
09 May 2023 16:03:34 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 4,233,600 44,356 0.0105
09 May 2023 15:48:22 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 2,592,000 24,751 0.0095
09 May 2023 15:48:09 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 1,728,000 16,147 0.0093
09 May 2023 15:47:58 1521081 22266388 oifs_43r3_ps_0715_1995050100_123_964_12181359_0 864,000 8,298 0.0096


©2024 cpdn.org