climateprediction.net home page
Task 22300714

Task 22300714

Name oifs_43r3_ps_0728_1990050100_123_959_12176372_1
Workunit 12176372
Created 20 Jan 2023, 20:03:25 UTC
Sent 20 Jan 2023, 20:11:36 UTC
Report deadline 21 Mar 2023, 20:11:36 UTC
Received 26 Jan 2023, 1:41:33 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1538243
Run time 22 hours 22 min 35 sec
CPU time 22 hours 21 min 56 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.52 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,416.69 MB
Peak swap size 4,978.08 MB
Peak disk usage 4,574.89 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
OK:w1:1:1:1408:1408] [20230123:221216:1674511936:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1408:1408] [20230123:221216:1674511936:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1408:1408] [20230123:221216:1674511936:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1408:1408] [20230123:221216:1674511936:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1408:1408] [20230123:221216:1674511936:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1408:1408] [20230123:221216:1674511936:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1408:1408] [20230123:221216:1674511936:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1408:1408] [20230123:221216:1674511936:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:13:37 STEP 2544 H=2544:00 +CPU= 38.990
  22:14:00 STEP 2545 H=2545:00 +CPU= 23.874
  22:14:26 STEP 2546 H=2546:00 +CPU= 25.304
  22:14:56 STEP 2547 H=2547:00 +CPU= 30.411
  22:15:18 STEP 2548 H=2548:00 +CPU= 21.937
  22:15:43 STEP 2549 H=2549:00 +CPU= 24.408
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1990050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0728
(argv4) batchid: 959
(argv5) wuid: 12176372
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/5
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_68a822b3e96a2e0d0744927567b2198e to: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_0728_1990050100_123_959_12176372.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_0728_1990050100_123_959_12176372.zip
ic_ancil_file: ic_ancil_12176372
ifsdata_file: ifsdata_12176372
climate_data_file: clim_data_12176372
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cd554a23f0f6a11c29fe766e11531bf2 to: /var/lib/boinc-client_40000/slots/5/ic_ancil_12176372.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/5/ic_ancil_12176372.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c6670e955278f36595614049abf1c4ec to: /var/lib/boinc-client_40000/slots/5/ifsdata/ifsdata_12176372.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/5/ifsdata/ifsdata_12176372.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/5/159l_2/clim_data_12176372.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/5/159l_2/clim_data_12176372.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/5/progress_file_12176372.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/5/progress_file_12176372.xml
-- Model is restarting --
Adjusting last_iter, 2550, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/5/progress_file_12176372.xml
last_cpu_time: 69540
upload_file_number: 106
last_iter: 2543
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839
The child process has been launched with process id: 1413
Executing the command: /var/lib/boinc-client_40000/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1413:1413] [20230123:222929:1674512969:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:30:45 STEP 2544 H=2544:00 +CPU= 36.508
  22:31:08 STEP 2545 H=2545:00 +CPU= 22.255
  22:31:32 STEP 2546 H=2546:00 +CPU= 24.003
  22:32:06 STEP 2547 H=2547:00 +CPU= 34.774
  22:32:29 STEP 2548 H=2548:00 +CPU= 22.375
  22:32:48 STEP 2549 H=2549:00 +CPU= 19.731
  22:33:19 STEP 2550 H=2550:00 +CPU= 30.964
  22:33:42 STEP 2551 H=2551:00 +CPU= 22.588
  22:34:07 STEP 2552 H=2552:00 +CPU= 25.025
  22:34:43 STEP 2553 H=2553:00 +CPU= 35.543
  22:35:06 STEP 2554 H=2554:00 +CPU= 23.261
  22:35:28 STEP 2555 H=2555:00 +CPU= 22.626
  22:36:05 STEP 2556 H=2556:00 +CPU= 36.069
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002556
  22:36:24 STEP 2557 H=2557:00 +CPU= 19.765
  22:36:44 STEP 2558 H=2558:00 +CPU= 20.115
  22:37:18 STEP 2559 H=2559:00 +CPU= 33.986
  22:37:41 STEP 2560 H=2560:00 +CPU= 22.847
  22:38:01 STEP 2561 H=2561:00 +CPU= 19.575
  22:38:32 STEP 2562 H=2562:00 +CPU= 31.195
  22:38:51 STEP 2563 H=2563:00 +CPU= 18.801
  22:39:12 STEP 2564 H=2564:00 +CPU= 20.932
  22:39:46 STEP 2565 H=2565:00 +CPU= 33.747
  22:40:04 STEP 2566 H=2566:00 +CPU= 18.489
  22:40:20 STEP 2567 H=2567:00 +CPU= 16.080
  22:40:55 STEP 2568 H=2568:00 +CPU= 35.161
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_106.zip
Uploading the intermediate file: upload_file_106.zip
  22:41:16 STEP 2569 H=2569:00 +CPU= 20.851
  22:41:38 STEP 2570 H=2570:00 +CPU= 21.662
  22:42:12 STEP 2571 H=2571:00 +CPU= 34.318
  22:42:32 STEP 2572 H=2572:00 +CPU= 20.225
  22:42:53 STEP 2573 H=2573:00 +CPU= 20.560
  22:43:26 STEP 2574 H=2574:00 +CPU= 33.142
  22:43:45 STEP 2575 H=2575:00 +CPU= 18.869
  22:44:04 STEP 2576 H=2576:00 +CPU= 19.404
  22:44:36 STEP 2577 H=2577:00 +CPU= 31.824
  22:44:57 STEP 2578 H=2578:00 +CPU= 20.792
  22:45:16 STEP 2579 H=2579:00 +CPU= 19.350
  22:45:50 STEP 2580 H=2580:00 +CPU= 34.074
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002580
  22:46:08 STEP 2581 H=2581:00 +CPU= 17.779
  22:46:31 STEP 2582 H=2582:00 +CPU= 22.447
  22:47:02 STEP 2583 H=2583:00 +CPU= 31.313
  22:47:24 STEP 2584 H=2584:00 +CPU= 21.991
  22:47:43 STEP 2585 H=2585:00 +CPU= 19.295
  22:48:07 STEP 2586 H=2586:00 +CPU= 24.144
  22:48:26 STEP 2587 H=2587:00 +CPU= 18.775
  22:48:48 STEP 2588 H=2588:00 +CPU= 21.390
  22:49:21 STEP 2589 H=2589:00 +CPU= 33.131
  22:49:40 STEP 2590 H=2590:00 +CPU= 19.157
  22:49:59 STEP 2591 H=2591:00 +CPU= 19.127
  22:50:36 STEP 2592 H=2592:00 +CPU= 37.406
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_107.zip
Uploading the intermediate file: upload_file_107.zip
  22:50:55 STEP 2593 H=2593:00 +CPU= 18.232
  22:51:09 STEP 2594 H=2594:00 +CPU= 14.111
  22:51:33 STEP 2595 H=2595:00 +CPU= 23.966
  22:51:47 STEP 2596 H=2596:00 +CPU= 14.653
  22:52:04 STEP 2597 H=2597:00 +CPU= 17.021
  22:52:39 STEP 2598 H=2598:00 +CPU= 34.078
  22:53:00 STEP 2599 H=2599:00 +CPU= 21.280
  22:53:21 STEP 2600 H=2600:00 +CPU= 21.417
  22:53:49 STEP 2601 H=2601:00 +CPU= 27.417
  22:54:05 STEP 2602 H=2602:00 +CPU= 16.179
  22:54:23 STEP 2603 H=2603:00 +CPU= 18.055
  22:55:00 STEP 2604 H=2604:00 +CPU= 37.052
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002604
  22:55:21 STEP 2605 H=2605:00 +CPU= 20.586
  22:55:41 STEP 2606 H=2606:00 +CPU= 20.140
  22:56:10 STEP 2607 H=2607:00 +CPU= 29.704
  22:56:26 STEP 2608 H=2608:00 +CPU= 15.728
  22:56:44 STEP 2609 H=2609:00 +CPU= 17.811
  22:57:15 STEP 2610 H=2610:00 +CPU= 31.048
  22:57:34 STEP 2611 H=2611:00 +CPU= 19.433
  22:57:51 STEP 2612 H=2612:00 +CPU= 16.497
  22:58:20 STEP 2613 H=2613:00 +CPU= 29.079
  22:58:36 STEP 2614 H=2614:00 +CPU= 15.848
  22:58:55 STEP 2615 H=2615:00 +CPU= 19.131
  22:59:32 STEP 2616 H=2616:00 +CPU= 37.078
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_108.zip
Uploading the intermediate file: upload_file_108.zip
  22:59:50 STEP 2617 H=2617:00 +CPU= 17.443
  23:00:05 STEP 2618 H=2618:00 +CPU= 15.359
  23:00:39 STEP 2619 H=2619:00 +CPU= 34.259
  23:00:59 STEP 2620 H=2620:00 +CPU= 20.046
  23:01:20 STEP 2621 H=2621:00 +CPU= 20.629
  23:01:51 STEP 2622 H=2622:00 +CPU= 31.120
  23:02:10 STEP 2623 H=2623:00 +CPU= 19.030
  23:02:30 STEP 2624 H=2624:00 +CPU= 20.308
  23:03:03 STEP 2625 H=2625:00 +CPU= 32.623
  23:03:25 STEP 2626 H=2626:00 +CPU= 22.547
  23:03:47 STEP 2627 H=2627:00 +CPU= 21.251
  23:04:20 STEP 2628 H=2628:00 +CPU= 33.466
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002628
  23:04:42 STEP 2629 H=2629:00 +CPU= 21.743
  23:05:02 STEP 2630 H=2630:00 +CPU= 20.026
  23:05:35 STEP 2631 H=2631:00 +CPU= 32.799
  23:05:54 STEP 2632 H=2632:00 +CPU= 19.162
  23:06:15 STEP 2633 H=2633:00 +CPU= 21.143
  23:06:46 STEP 2634 H=2634:00 +CPU= 31.377
  23:07:07 STEP 2635 H=2635:00 +CPU= 20.820
  23:07:29 STEP 2636 H=2636:00 +CPU= 21.528
  23:07:57 STEP 2637 H=2637:00 +CPU= 27.988
  23:08:17 STEP 2638 H=2638:00 +CPU= 20.088
  23:08:38 STEP 2639 H=2639:00 +CPU= 21.498
  23:09:15 STEP 2640 H=2640:00 +CPU= 36.981
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_109.zip
Uploading the intermediate file: upload_file_109.zip
  23:09:35 STEP 2641 H=2641:00 +CPU= 19.700
  23:09:53 STEP 2642 H=2642:00 +CPU= 17.817
  23:10:23 STEP 2643 H=2643:00 +CPU= 29.878
  23:10:44 STEP 2644 H=2644:00 +CPU= 21.133
  23:11:00 STEP 2645 H=2645:00 +CPU= 16.118
  23:11:32 STEP 2646 H=2646:00 +CPU= 31.744
  23:11:53 STEP 2647 H=2647:00 +CPU= 20.912
  23:12:11 STEP 2648 H=2648:00 +CPU= 18.219
  23:12:38 STEP 2649 H=2649:00 +CPU= 27.385
  23:13:01 STEP 2650 H=2650:00 +CPU= 22.401
  23:13:20 STEP 2651 H=2651:00 +CPU= 18.954
  23:13:49 STEP 2652 H=2652:00 +CPU= 29.140
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002652
  23:14:10 STEP 2653 H=2653:00 +CPU= 21.333
  23:14:30 STEP 2654 H=2654:00 +CPU= 20.187
  23:14:58 STEP 2655 H=2655:00 +CPU= 27.353
  23:15:20 STEP 2656 H=2656:00 +CPU= 21.728
  23:15:37 STEP 2657 H=2657:00 +CPU= 17.501
  23:16:08 STEP 2658 H=2658:00 +CPU= 31.205
  23:16:29 STEP 2659 H=2659:00 +CPU= 20.503
  23:16:49 STEP 2660 H=2660:00 +CPU= 19.869
  23:17:21 STEP 2661 H=2661:00 +CPU= 32.570
  23:17:41 STEP 2662 H=2662:00 +CPU= 20.060
  23:18:01 STEP 2663 H=2663:00 +CPU= 19.940
  23:18:34 STEP 2664 H=2664:00 +CPU= 32.671
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_110.zip
Uploading the intermediate file: upload_file_110.zip
  23:18:49 STEP 2665 H=2665:00 +CPU= 15.063
  23:19:04 STEP 2666 H=2666:00 +CPU= 15.228
  23:19:33 STEP 2667 H=2667:00 +CPU= 29.076
  23:19:54 STEP 2668 H=2668:00 +CPU= 20.717
  23:20:16 STEP 2669 H=2669:00 +CPU= 21.903
  23:20:50 STEP 2670 H=2670:00 +CPU= 34.192
  23:21:06 STEP 2671 H=2671:00 +CPU= 16.318
  23:21:22 STEP 2672 H=2672:00 +CPU= 15.735
  23:21:52 STEP 2673 H=2673:00 +CPU= 30.122
  23:22:14 STEP 2674 H=2674:00 +CPU= 21.619
  23:22:33 STEP 2675 H=2675:00 +CPU= 18.817
  23:23:01 STEP 2676 H=2676:00 +CPU= 28.270
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002676
  23:23:22 STEP 2677 H=2677:00 +CPU= 20.889
  23:23:41 STEP 2678 H=2678:00 +CPU= 19.078
  23:24:12 STEP 2679 H=2679:00 +CPU= 31.441
  23:24:33 STEP 2680 H=2680:00 +CPU= 20.542
  23:24:54 STEP 2681 H=2681:00 +CPU= 20.967
  23:25:24 STEP 2682 H=2682:00 +CPU= 30.391
  23:25:41 STEP 2683 H=2683:00 +CPU= 16.720
  23:25:57 STEP 2684 H=2684:00 +CPU= 15.864
  23:26:28 STEP 2685 H=2685:00 +CPU= 30.906
  23:26:49 STEP 2686 H=2686:00 +CPU= 21.530
  23:27:12 STEP 2687 H=2687:00 +CPU= 22.536
  23:27:46 STEP 2688 H=2688:00 +CPU= 34.052
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_111.zip
Uploading the intermediate file: upload_file_111.zip
  23:28:06 STEP 2689 H=2689:00 +CPU= 19.958
  23:28:24 STEP 2690 H=2690:00 +CPU= 17.727
  23:28:53 STEP 2691 H=2691:00 +CPU= 29.132
  23:29:10 STEP 2692 H=2692:00 +CPU= 17.413
  23:29:27 STEP 2693 H=2693:00 +CPU= 16.763
  23:29:58 STEP 2694 H=2694:00 +CPU= 31.468
  23:30:14 STEP 2695 H=2695:00 +CPU= 15.781
  23:30:29 STEP 2696 H=2696:00 +CPU= 14.645
  23:31:02 STEP 2697 H=2697:00 +CPU= 32.575
  23:31:24 STEP 2698 H=2698:00 +CPU= 22.513
  23:31:44 STEP 2699 H=2699:00 +CPU= 19.612
  23:32:14 STEP 2700 H=2700:00 +CPU= 30.332
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002700
  23:32:32 STEP 2701 H=2701:00 +CPU= 18.484
  23:32:47 STEP 2702 H=2702:00 +CPU= 14.832
  23:33:21 STEP 2703 H=2703:00 +CPU= 33.369
  23:33:43 STEP 2704 H=2704:00 +CPU= 21.870
  23:34:02 STEP 2705 H=2705:00 +CPU= 19.228
  23:34:36 STEP 2706 H=2706:00 +CPU= 33.895
  23:34:55 STEP 2707 H=2707:00 +CPU= 19.388
  23:35:16 STEP 2708 H=2708:00 +CPU= 20.802
  23:35:45 STEP 2709 H=2709:00 +CPU= 28.867
  23:36:04 STEP 2710 H=2710:00 +CPU= 19.263
  23:36:24 STEP 2711 H=2711:00 +CPU= 19.975
  23:37:03 STEP 2712 H=2712:00 +CPU= 39.243
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_112.zip
Uploading the intermediate file: upload_file_112.zip
  23:37:25 STEP 2713 H=2713:00 +CPU= 21.514
  23:37:46 STEP 2714 H=2714:00 +CPU= 21.591
  23:38:15 STEP 2715 H=2715:00 +CPU= 28.968
  23:38:36 STEP 2716 H=2716:00 +CPU= 20.870
  23:38:55 STEP 2717 H=2717:00 +CPU= 18.902
  23:39:26 STEP 2718 H=2718:00 +CPU= 30.817
  23:39:47 STEP 2719 H=2719:00 +CPU= 21.450
  23:40:04 STEP 2720 H=2720:00 +CPU= 16.684
  23:40:35 STEP 2721 H=2721:00 +CPU= 30.622
  23:40:55 STEP 2722 H=2722:00 +CPU= 20.381
  23:41:16 STEP 2723 H=2723:00 +CPU= 21.238
  23:41:53 STEP 2724 H=2724:00 +CPU= 36.188
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002724
  23:42:14 STEP 2725 H=2725:00 +CPU= 21.898
  23:42:34 STEP 2726 H=2726:00 +CPU= 19.714
  23:43:06 STEP 2727 H=2727:00 +CPU= 32.030
  23:43:24 STEP 2728 H=2728:00 +CPU= 18.012
  23:43:44 STEP 2729 H=2729:00 +CPU= 19.901
  23:44:17 STEP 2730 H=2730:00 +CPU= 32.655
  23:44:37 STEP 2731 H=2731:00 +CPU= 20.362
  23:44:56 STEP 2732 H=2732:00 +CPU= 18.571
  23:45:27 STEP 2733 H=2733:00 +CPU= 31.643
  23:45:49 STEP 2734 H=2734:00 +CPU= 21.528
  23:46:10 STEP 2735 H=2735:00 +CPU= 21.493
  23:46:50 STEP 2736 H=2736:00 +CPU= 39.597
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_113.zip
Uploading the intermediate file: upload_file_113.zip
  23:47:12 STEP 2737 H=2737:00 +CPU= 21.677
  23:47:34 STEP 2738 H=2738:00 +CPU= 21.957
  23:47:56 STEP 2739 H=2739:00 +CPU= 22.626
  23:48:18 STEP 2740 H=2740:00 +CPU= 21.576
  23:48:33 STEP 2741 H=2741:00 +CPU= 14.804
  23:49:00 STEP 2742 H=2742:00 +CPU= 27.387
  23:49:22 STEP 2743 H=2743:00 +CPU= 21.677
  23:49:44 STEP 2744 H=2744:00 +CPU= 22.750
  23:50:17 STEP 2745 H=2745:00 +CPU= 32.863
  23:50:34 STEP 2746 H=2746:00 +CPU= 16.602
  23:50:52 STEP 2747 H=2747:00 +CPU= 18.482
  23:51:26 STEP 2748 H=2748:00 +CPU= 33.574
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002748
  23:51:46 STEP 2749 H=2749:00 +CPU= 19.722
  23:52:09 STEP 2750 H=2750:00 +CPU= 23.064
  23:52:40 STEP 2751 H=2751:00 +CPU= 30.939
  23:53:01 STEP 2752 H=2752:00 +CPU= 21.586
  23:53:22 STEP 2753 H=2753:00 +CPU= 20.483
  23:53:57 STEP 2754 H=2754:00 +CPU= 35.489
  23:54:20 STEP 2755 H=2755:00 +CPU= 23.083
  23:54:42 STEP 2756 H=2756:00 +CPU= 21.703
  23:55:11 STEP 2757 H=2757:00 +CPU= 29.091
  23:55:28 STEP 2758 H=2758:00 +CPU= 16.664
  23:55:43 STEP 2759 H=2759:00 +CPU= 14.842
  23:56:11 STEP 2760 H=2760:00 +CPU= 28.769
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_114.zip
Uploading the intermediate file: upload_file_114.zip
  23:56:28 STEP 2761 H=2761:00 +CPU= 16.487
  23:56:49 STEP 2762 H=2762:00 +CPU= 20.575
  23:57:21 STEP 2763 H=2763:00 +CPU= 32.181
  23:57:42 STEP 2764 H=2764:00 +CPU= 21.578
  23:58:02 STEP 2765 H=2765:00 +CPU= 19.939
  23:58:36 STEP 2766 H=2766:00 +CPU= 33.734
  23:58:56 STEP 2767 H=2767:00 +CPU= 20.270
  23:59:20 STEP 2768 H=2768:00 +CPU= 23.574
  23:59:54 STEP 2769 H=2769:00 +CPU= 33.776
  00:00:17 STEP 2770 H=2770:00 +CPU= 23.218
  00:00:33 STEP 2771 H=2771:00 +CPU= 16.572
  00:01:08 STEP 2772 H=2772:00 +CPU= 35.019
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002772
  00:01:28 STEP 2773 H=2773:00 +CPU= 19.894
  00:01:48 STEP 2774 H=2774:00 +CPU= 19.500
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1990050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0728
(argv4) batchid: 959
(argv5) wuid: 12176372
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/5
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_68a822b3e96a2e0d0744927567b2198e to: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_0728_1990050100_123_959_12176372.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_0728_1990050100_123_959_12176372.zip
ic_ancil_file: ic_ancil_12176372
ifsdata_file: ifsdata_12176372
climate_data_file: clim_data_12176372
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cd554a23f0f6a11c29fe766e11531bf2 to: /var/lib/boinc-client_40000/slots/5/ic_ancil_12176372.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/5/ic_ancil_12176372.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c6670e955278f36595614049abf1c4ec to: /var/lib/boinc-client_40000/slots/5/ifsdata/ifsdata_12176372.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/5/ifsdata/ifsdata_12176372.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/5/159l_2/clim_data_12176372.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/5/159l_2/clim_data_12176372.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/5/progress_file_12176372.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/5/progress_file_12176372.xml
-- Model is restarting --
Adjusting last_iter, 2775, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/5/progress_file_12176372.xml
last_cpu_time: 75047
upload_file_number: 115
last_iter: 2759
last_upload: 9936000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839
The child process has been launched with process id: 1417
Executing the command: /var/lib/boinc-client_40000/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:w1:1:1:1417:1417] [20230124:001208:1674519128:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:13:33 STEP 2760 H=2760:00 +CPU= 41.102
  00:13:56 STEP 2761 H=2761:00 +CPU= 22.532
  00:14:20 STEP 2762 H=2762:00 +CPU= 24.601
  00:14:50 STEP 2763 H=2763:00 +CPU= 29.630
  00:15:13 STEP 2764 H=2764:00 +CPU= 22.849
  00:15:36 STEP 2765 H=2765:00 +CPU= 22.936
  00:16:11 STEP 2766 H=2766:00 +CPU= 35.087
  00:16:33 STEP 2767 H=2767:00 +CPU= 22.105
  00:16:56 STEP 2768 H=2768:00 +CPU= 23.239
  00:17:29 STEP 2769 H=2769:00 +CPU= 32.760
  00:17:54 STEP 2770 H=2770:00 +CPU= 25.134
  00:18:19 STEP 2771 H=2771:00 +CPU= 24.572
  00:18:55 STEP 2772 H=2772:00 +CPU= 36.554
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002772
  00:19:19 STEP 2773 H=2773:00 +CPU= 24.025
  00:19:44 STEP 2774 H=2774:00 +CPU= 25.022
  00:20:21 STEP 2775 H=2775:00 +CPU= 36.343
  00:20:45 STEP 2776 H=2776:00 +CPU= 24.691
  00:21:10 STEP 2777 H=2777:00 +CPU= 24.208
  00:21:47 STEP 2778 H=2778:00 +CPU= 37.084
  00:22:11 STEP 2779 H=2779:00 +CPU= 24.409
  00:22:34 STEP 2780 H=2780:00 +CPU= 22.972
  00:23:07 STEP 2781 H=2781:00 +CPU= 33.041
  00:23:32 STEP 2782 H=2782:00 +CPU= 24.417
  00:23:56 STEP 2783 H=2783:00 +CPU= 24.662
  00:24:40 STEP 2784 H=2784:00 +CPU= 43.759
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_115.zip
Uploading the intermediate file: upload_file_115.zip
  00:25:01 STEP 2785 H=2785:00 +CPU= 20.999
  00:25:22 STEP 2786 H=2786:00 +CPU= 21.350
  00:26:01 STEP 2787 H=2787:00 +CPU= 38.428
  00:26:26 STEP 2788 H=2788:00 +CPU= 24.898
  00:26:50 STEP 2789 H=2789:00 +CPU= 23.961
  00:27:27 STEP 2790 H=2790:00 +CPU= 37.373
  00:27:51 STEP 2791 H=2791:00 +CPU= 24.343
  00:28:14 STEP 2792 H=2792:00 +CPU= 22.104
  00:28:49 STEP 2793 H=2793:00 +CPU= 35.001
  00:29:10 STEP 2794 H=2794:00 +CPU= 21.185
  00:29:33 STEP 2795 H=2795:00 +CPU= 23.578
  00:30:14 STEP 2796 H=2796:00 +CPU= 40.994
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002796
  00:30:39 STEP 2797 H=2797:00 +CPU= 24.774
  00:31:05 STEP 2798 H=2798:00 +CPU= 25.573
  00:31:41 STEP 2799 H=2799:00 +CPU= 36.167
  00:32:05 STEP 2800 H=2800:00 +CPU= 24.094
  00:32:28 STEP 2801 H=2801:00 +CPU= 23.572
  00:33:08 STEP 2802 H=2802:00 +CPU= 39.161
  00:33:32 STEP 2803 H=2803:00 +CPU= 24.474
  00:33:58 STEP 2804 H=2804:00 +CPU= 25.710
  00:34:36 STEP 2805 H=2805:00 +CPU= 38.407
  00:35:01 STEP 2806 H=2806:00 +CPU= 25.172
  00:35:22 STEP 2807 H=2807:00 +CPU= 20.864
  00:36:04 STEP 2808 H=2808:00 +CPU= 42.046
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_116.zip
Uploading the intermediate file: upload_file_116.zip
  00:36:27 STEP 2809 H=2809:00 +CPU= 22.849
  00:36:53 STEP 2810 H=2810:00 +CPU= 25.282
  00:37:22 STEP 2811 H=2811:00 +CPU= 29.608
  00:37:44 STEP 2812 H=2812:00 +CPU= 21.434
  00:38:06 STEP 2813 H=2813:00 +CPU= 21.959
  00:38:44 STEP 2814 H=2814:00 +CPU= 38.770
  00:39:03 STEP 2815 H=2815:00 +CPU= 18.544
  00:39:29 STEP 2816 H=2816:00 +CPU= 25.659
  00:40:08 STEP 2817 H=2817:00 +CPU= 39.253
  00:40:34 STEP 2818 H=2818:00 +CPU= 26.041
  00:40:59 STEP 2819 H=2819:00 +CPU= 25.165
  00:41:40 STEP 2820 H=2820:00 +CPU= 40.936
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002820
  00:42:05 STEP 2821 H=2821:00 +CPU= 24.935
  00:42:31 STEP 2822 H=2822:00 +CPU= 26.034
  00:43:10 STEP 2823 H=2823:00 +CPU= 38.738
  00:43:36 STEP 2824 H=2824:00 +CPU= 25.882
  00:43:59 STEP 2825 H=2825:00 +CPU= 23.788
  00:44:35 STEP 2826 H=2826:00 +CPU= 35.701
  00:44:52 STEP 2827 H=2827:00 +CPU= 16.908
  00:45:12 STEP 2828 H=2828:00 +CPU= 19.931
  00:45:49 STEP 2829 H=2829:00 +CPU= 37.262
  00:46:13 STEP 2830 H=2830:00 +CPU= 23.541
  00:46:38 STEP 2831 H=2831:00 +CPU= 24.846
  00:47:21 STEP 2832 H=2832:00 +CPU= 43.257
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_117.zip
Uploading the intermediate file: upload_file_117.zip
  00:47:45 STEP 2833 H=2833:00 +CPU= 24.203
  00:48:11 STEP 2834 H=2834:00 +CPU= 25.577
  00:48:48 STEP 2835 H=2835:00 +CPU= 37.307
  00:49:13 STEP 2836 H=2836:00 +CPU= 25.104
  00:49:36 STEP 2837 H=2837:00 +CPU= 23.097
  00:50:11 STEP 2838 H=2838:00 +CPU= 35.365
  00:50:35 STEP 2839 H=2839:00 +CPU= 23.269
  00:51:00 STEP 2840 H=2840:00 +CPU= 25.589
  00:51:31 STEP 2841 H=2841:00 +CPU= 30.762
  00:51:49 STEP 2842 H=2842:00 +CPU= 17.608
  00:52:13 STEP 2843 H=2843:00 +CPU= 24.563
  00:52:53 STEP 2844 H=2844:00 +CPU= 40.136
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002844
  00:53:18 STEP 2845 H=2845:00 +CPU= 24.231
  00:53:42 STEP 2846 H=2846:00 +CPU= 24.664
  00:54:20 STEP 2847 H=2847:00 +CPU= 37.648
  00:54:44 STEP 2848 H=2848:00 +CPU= 23.800
  00:55:03 STEP 2849 H=2849:00 +CPU= 18.874
  00:55:42 STEP 2850 H=2850:00 +CPU= 39.497
  00:56:05 STEP 2851 H=2851:00 +CPU= 22.441
  00:56:26 STEP 2852 H=2852:00 +CPU= 21.025
  00:57:02 STEP 2853 H=2853:00 +CPU= 36.765
  00:57:28 STEP 2854 H=2854:00 +CPU= 25.270
  00:57:54 STEP 2855 H=2855:00 +CPU= 26.046
  00:58:37 STEP 2856 H=2856:00 +CPU= 43.507
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_118.zip
Uploading the intermediate file: upload_file_118.zip
  00:59:01 STEP 2857 H=2857:00 +CPU= 24.184
  00:59:27 STEP 2858 H=2858:00 +CPU= 25.246
  00:59:59 STEP 2859 H=2859:00 +CPU= 32.376
  01:00:17 STEP 2860 H=2860:00 +CPU= 17.976
  01:00:35 STEP 2861 H=2861:00 +CPU= 18.119
  01:01:03 STEP 2862 H=2862:00 +CPU= 28.288
  01:01:23 STEP 2863 H=2863:00 +CPU= 20.018
  01:01:41 STEP 2864 H=2864:00 +CPU= 17.415
  01:02:13 STEP 2865 H=2865:00 +CPU= 32.188
  01:02:32 STEP 2866 H=2866:00 +CPU= 19.057
  01:02:57 STEP 2867 H=2867:00 +CPU= 25.041
  01:03:34 STEP 2868 H=2868:00 +CPU= 36.686
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002868
  01:03:54 STEP 2869 H=2869:00 +CPU= 20.049
  01:04:20 STEP 2870 H=2870:00 +CPU= 26.361
  01:05:00 STEP 2871 H=2871:00 +CPU= 39.563
  01:05:26 STEP 2872 H=2872:00 +CPU= 26.013
  01:05:47 STEP 2873 H=2873:00 +CPU= 20.905
  01:06:26 STEP 2874 H=2874:00 +CPU= 38.875
  01:06:51 STEP 2875 H=2875:00 +CPU= 25.276
  01:07:17 STEP 2876 H=2876:00 +CPU= 26.131
  01:07:55 STEP 2877 H=2877:00 +CPU= 37.761
  01:08:18 STEP 2878 H=2878:00 +CPU= 23.510
  01:08:40 STEP 2879 H=2879:00 +CPU= 21.216
  01:09:19 STEP 2880 H=2880:00 +CPU= 39.619
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_119.zip
Uploading the intermediate file: upload_file_119.zip
  01:09:44 STEP 2881 H=2881:00 +CPU= 24.560
  01:10:10 STEP 2882 H=2882:00 +CPU= 26.283
  01:10:48 STEP 2883 H=2883:00 +CPU= 37.865
  01:11:13 STEP 2884 H=2884:00 +CPU= 24.994
  01:11:37 STEP 2885 H=2885:00 +CPU= 24.322
  01:12:16 STEP 2886 H=2886:00 +CPU= 38.909
  01:12:41 STEP 2887 H=2887:00 +CPU= 24.507
  01:12:59 STEP 2888 H=2888:00 +CPU= 18.252
  01:13:36 STEP 2889 H=2889:00 +CPU= 37.295
  01:14:02 STEP 2890 H=2890:00 +CPU= 25.360
  01:14:28 STEP 2891 H=2891:00 +CPU= 25.990
  01:15:11 STEP 2892 H=2892:00 +CPU= 43.644
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002892
  01:15:37 STEP 2893 H=2893:00 +CPU= 25.374
  01:16:02 STEP 2894 H=2894:00 +CPU= 25.185
  01:16:31 STEP 2895 H=2895:00 +CPU= 29.413
  01:16:57 STEP 2896 H=2896:00 +CPU= 25.706
  01:17:22 STEP 2897 H=2897:00 +CPU= 25.260
  01:18:00 STEP 2898 H=2898:00 +CPU= 37.990
  01:18:25 STEP 2899 H=2899:00 +CPU= 24.477
  01:18:50 STEP 2900 H=2900:00 +CPU= 25.055
  01:19:28 STEP 2901 H=2901:00 +CPU= 37.793
  01:19:53 STEP 2902 H=2902:00 +CPU= 25.155
  01:20:19 STEP 2903 H=2903:00 +CPU= 25.855
  01:20:58 STEP 2904 H=2904:00 +CPU= 39.631
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_120.zip
Uploading the intermediate file: upload_file_120.zip
  01:21:23 STEP 2905 H=2905:00 +CPU= 24.506
  01:21:47 STEP 2906 H=2906:00 +CPU= 24.603
  01:22:25 STEP 2907 H=2907:00 +CPU= 37.287
  01:22:49 STEP 2908 H=2908:00 +CPU= 24.639
  01:23:12 STEP 2909 H=2909:00 +CPU= 23.242
  01:23:47 STEP 2910 H=2910:00 +CPU= 34.791
  01:24:11 STEP 2911 H=2911:00 +CPU= 24.186
  01:24:37 STEP 2912 H=2912:00 +CPU= 25.151
  01:25:09 STEP 2913 H=2913:00 +CPU= 32.436
  01:25:35 STEP 2914 H=2914:00 +CPU= 26.375
  01:26:01 STEP 2915 H=2915:00 +CPU= 25.262
  01:26:44 STEP 2916 H=2916:00 +CPU= 43.422
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002916
  01:27:09 STEP 2917 H=2917:00 +CPU= 24.967
  01:27:34 STEP 2918 H=2918:00 +CPU= 25.038
  01:28:12 STEP 2919 H=2919:00 +CPU= 38.099
  01:28:36 STEP 2920 H=2920:00 +CPU= 23.299
  01:28:58 STEP 2921 H=2921:00 +CPU= 22.849
  01:29:37 STEP 2922 H=2922:00 +CPU= 38.573
  01:30:00 STEP 2923 H=2923:00 +CPU= 23.552
  01:30:18 STEP 2924 H=2924:00 +CPU= 17.861
  01:30:53 STEP 2925 H=2925:00 +CPU= 34.586
  01:31:16 STEP 2926 H=2926:00 +CPU= 23.466
  01:31:42 STEP 2927 H=2927:00 +CPU= 25.556
  01:32:27 STEP 2928 H=2928:00 +CPU= 44.610
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_121.zip
Uploading the intermediate file: upload_file_121.zip
  01:32:51 STEP 2929 H=2929:00 +CPU= 24.532
  01:33:17 STEP 2930 H=2930:00 +CPU= 25.427
  01:33:56 STEP 2931 H=2931:00 +CPU= 38.961
  01:34:21 STEP 2932 H=2932:00 +CPU= 25.741
  01:34:46 STEP 2933 H=2933:00 +CPU= 24.294
  01:35:20 STEP 2934 H=2934:00 +CPU= 34.833
  01:35:39 STEP 2935 H=2935:00 +CPU= 18.735
  01:36:04 STEP 2936 H=2936:00 +CPU= 24.760
  01:36:42 STEP 2937 H=2937:00 +CPU= 37.713
  01:37:07 STEP 2938 H=2938:00 +CPU= 25.676
  01:37:32 STEP 2939 H=2939:00 +CPU= 24.276
  01:38:12 STEP 2940 H=2940:00 +CPU= 40.807
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002940
  01:38:32 STEP 2941 H=2941:00 +CPU= 19.648
  01:38:55 STEP 2942 H=2942:00 +CPU= 22.796
  01:39:33 STEP 2943 H=2943:00 +CPU= 38.145
  01:39:59 STEP 2944 H=2944:00 +CPU= 25.910
  01:40:24 STEP 2945 H=2945:00 +CPU= 24.697
  01:41:03 STEP 2946 H=2946:00 +CPU= 38.973
  01:41:27 STEP 2947 H=2947:00 +CPU= 24.242
  01:41:51 STEP 2948 H=2948:00 +CPU= 24.324
  01:42:30 STEP 2949 H=2949:00 +CPU= 38.887
  01:42:56 STEP 2950 H=2950:00 +CPU= 25.577
  01:43:17 STEP 2951 H=2951:00 +CPU= 21.546
  01:44:00 STEP 2952 H=2952:00 +CPU= 42.634
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12176372/ICMSHhq0f+002928
Zipping up the final file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0728_1990050100_123_959_12176372_1_r1991168839_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
01:47:23 (1319): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:38:36 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 10,623,600 80,516 0.0076
09 May 2023 16:38:36 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 10,623,600 80,516 0.0076
09 May 2023 16:38:18 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 8,640,000 64,740 0.0075
09 May 2023 16:38:18 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 8,640,000 64,740 0.0075
09 May 2023 16:38:09 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 7,776,000 58,226 0.0075
09 May 2023 16:38:09 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 7,776,000 58,226 0.0075
09 May 2023 16:37:59 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 6,912,000 51,640 0.0075
09 May 2023 16:37:59 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 6,912,000 51,640 0.0075
09 May 2023 16:37:54 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 6,048,000 45,064 0.0075
09 May 2023 16:37:54 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 6,048,000 45,064 0.0075
09 May 2023 16:37:43 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 5,184,000 38,599 0.0074
09 May 2023 16:37:34 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 4,320,000 32,094 0.0074
09 May 2023 16:37:24 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 3,456,000 25,551 0.0074
09 May 2023 16:37:24 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 3,456,000 25,551 0.0074
09 May 2023 16:37:14 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 2,592,000 19,067 0.0074
09 May 2023 16:37:09 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 1,728,000 12,593 0.0073
09 May 2023 16:37:09 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 1,728,000 12,593 0.0073
09 May 2023 16:36:58 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 864,000 6,280 0.0073
09 May 2023 16:36:58 1538243 22300714 oifs_43r3_ps_0728_1990050100_123_959_12176372_1 864,000 6,280 0.0073


©2024 cpdn.org