climateprediction.net home page
Task 22285298

Task 22285298

Name oifs_43r3_ps_0501_2013050100_123_982_12199145_0
Workunit 12199145
Created 24 Dec 2022, 7:44:31 UTC
Sent 8 Jan 2023, 18:29:30 UTC
Report deadline 7 Feb 2023, 18:29:30 UTC
Received 12 Jan 2023, 17:02:00 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1509499
Run time 13 hours 43 min 50 sec
CPU time 13 hours 26 min 51 sec
Validate state Invalid
Credit 9,192.96
Device peak FLOPS 5.06 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,414.58 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,189.68 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ub/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:164619:1673451979:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:164619:1673451979:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:164619:1673451979:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:46:57 STEP 1584 H=1584:00 +CPU= 32.400
  16:47:16 STEP 1585 H=1585:00 +CPU= 18.341
  16:47:35 STEP 1586 H=1586:00 +CPU= 18.979
  16:48:05 STEP 1587 H=1587:00 +CPU= 28.910
  16:48:24 STEP 1588 H=1588:00 +CPU= 18.612
  16:48:42 STEP 1589 H=1589:00 +CPU= 17.870
  16:49:11 STEP 1590 H=1590:00 +CPU= 28.873
  16:49:29 STEP 1591 H=1591:00 +CPU= 17.891
  16:49:47 STEP 1592 H=1592:00 +CPU= 18.403
  16:50:15 STEP 1593 H=1593:00 +CPU= 28.275
  16:50:34 STEP 1594 H=1594:00 +CPU= 18.455
  16:50:52 STEP 1595 H=1595:00 +CPU= 17.705
  16:51:24 STEP 1596 H=1596:00 +CPU= 31.165
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001596
  16:51:41 STEP 1597 H=1597:00 +CPU= 16.891
  16:51:59 STEP 1598 H=1598:00 +CPU= 17.854
  16:52:30 STEP 1599 H=1599:00 +CPU= 29.227
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.408] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4394MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.408] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.408] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.408] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.408] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.408] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4394 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.408] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.408] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.411] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.411] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165241:1673452361:381.412] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               POSTPHY_LAYER 
double free or corruption (top)
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165242:1673452362:382.412] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:6778:6778] [20230111:165242:1673452362:382.412] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000E99CCD  postphy_layer_            144  postphy_layer.F90
oifs_43r3_model.e  0000000000E5022E  ec_phys_                  945  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0501
(argv4) batchid: 982
(argv5) wuid: 12199145
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a4068be3472cc28e00e0cc835b5e45ed to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0501_2013050100_123_982_12199145.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0501_2013050100_123_982_12199145.zip
ic_ancil_file: ic_ancil_12199145
ifsdata_file: ifsdata_12199145
climate_data_file: clim_data_12199145
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a15338638cb4f138ee73eac0eb71f0cb to: /var/lib/boinc-client/slots/1/ic_ancil_12199145.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12199145.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fc7cee412b4c912150c99db18400b8cb to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12199145.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12199145.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12199145.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12199145.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12199145.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12199145.xml
-- Model is restarting --
Adjusting last_iter, 1600, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12199145.xml
last_cpu_time: 38019
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025
The child process has been launched with process id: 2946
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2946:2946] [20230112:140114:1673528474:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:02:02 STEP 1584 H=1584:00 +CPU= 31.657
  14:02:23 STEP 1585 H=1585:00 +CPU= 17.725
  14:02:41 STEP 1586 H=1586:00 +CPU= 18.190
  14:03:12 STEP 1587 H=1587:00 +CPU= 29.108
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:03:53 STEP 1588 H=1588:00 +CPU= 19.157
  14:04:11 STEP 1589 H=1589:00 +CPU= 18.028
  14:04:41 STEP 1590 H=1590:00 +CPU= 29.278
  14:05:01 STEP 1591 H=1591:00 +CPU= 18.844
  14:05:19 STEP 1592 H=1592:00 +CPU= 17.905
  14:05:46 STEP 1593 H=1593:00 +CPU= 26.992
  14:06:04 STEP 1594 H=1594:00 +CPU= 17.789
  14:06:21 STEP 1595 H=1595:00 +CPU= 17.194
  14:06:52 STEP 1596 H=1596:00 +CPU= 30.960
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001596
  14:07:10 STEP 1597 H=1597:00 +CPU= 17.859
  14:07:29 STEP 1598 H=1598:00 +CPU= 18.376
  14:07:57 STEP 1599 H=1599:00 +CPU= 28.206
  14:08:15 STEP 1600 H=1600:00 +CPU= 18.283
  14:08:33 STEP 1601 H=1601:00 +CPU= 17.812
  14:09:02 STEP 1602 H=1602:00 +CPU= 28.771
  14:09:20 STEP 1603 H=1603:00 +CPU= 17.822
  14:09:39 STEP 1604 H=1604:00 +CPU= 18.373
  14:10:07 STEP 1605 H=1605:00 +CPU= 28.316
  14:10:26 STEP 1606 H=1606:00 +CPU= 18.330
  14:10:44 STEP 1607 H=1607:00 +CPU= 18.331
  14:11:18 STEP 1608 H=1608:00 +CPU= 33.096
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_66.zip
Uploading the intermediate file: upload_file_66.zip
  14:11:39 STEP 1609 H=1609:00 +CPU= 18.904
  14:11:58 STEP 1610 H=1610:00 +CPU= 18.812
  14:12:29 STEP 1611 H=1611:00 +CPU= 29.059
  14:12:49 STEP 1612 H=1612:00 +CPU= 18.863
  14:13:07 STEP 1613 H=1613:00 +CPU= 17.890
  14:13:39 STEP 1614 H=1614:00 +CPU= 29.835
  14:13:59 STEP 1615 H=1615:00 +CPU= 18.949
  14:14:19 STEP 1616 H=1616:00 +CPU= 18.929
  14:14:49 STEP 1617 H=1617:00 +CPU= 28.942
  14:15:08 STEP 1618 H=1618:00 +CPU= 18.667
  14:15:28 STEP 1619 H=1619:00 +CPU= 18.507
  14:16:03 STEP 1620 H=1620:00 +CPU= 32.108
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001620
  14:16:21 STEP 1621 H=1621:00 +CPU= 17.986
  14:16:40 STEP 1622 H=1622:00 +CPU= 18.662
  14:17:12 STEP 1623 H=1623:00 +CPU= 29.256
  14:17:33 STEP 1624 H=1624:00 +CPU= 19.215
  14:17:52 STEP 1625 H=1625:00 +CPU= 18.305
  14:18:22 STEP 1626 H=1626:00 +CPU= 29.789
  14:18:41 STEP 1627 H=1627:00 +CPU= 18.319
  14:19:00 STEP 1628 H=1628:00 +CPU= 18.528
  14:19:31 STEP 1629 H=1629:00 +CPU= 29.336
  14:19:51 STEP 1630 H=1630:00 +CPU= 19.571
  14:20:11 STEP 1631 H=1631:00 +CPU= 18.905
  14:20:51 STEP 1632 H=1632:00 +CPU= 34.084
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_67.zip
Uploading the intermediate file: upload_file_67.zip
  14:21:10 STEP 1633 H=1633:00 +CPU= 18.345
  14:21:29 STEP 1634 H=1634:00 +CPU= 18.818
  14:21:59 STEP 1635 H=1635:00 +CPU= 28.893
  14:22:18 STEP 1636 H=1636:00 +CPU= 18.687
  14:22:36 STEP 1637 H=1637:00 +CPU= 18.294
  14:23:06 STEP 1638 H=1638:00 +CPU= 28.942
  14:23:24 STEP 1639 H=1639:00 +CPU= 17.928
  14:23:42 STEP 1640 H=1640:00 +CPU= 18.368
  14:24:11 STEP 1641 H=1641:00 +CPU= 28.337
  14:24:29 STEP 1642 H=1642:00 +CPU= 18.270
  14:24:47 STEP 1643 H=1643:00 +CPU= 17.749
  14:25:18 STEP 1644 H=1644:00 +CPU= 31.421
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001644
  14:25:36 STEP 1645 H=1645:00 +CPU= 17.752
  14:25:54 STEP 1646 H=1646:00 +CPU= 18.230
  14:26:23 STEP 1647 H=1647:00 +CPU= 28.124
  14:26:41 STEP 1648 H=1648:00 +CPU= 18.396
  14:26:59 STEP 1649 H=1649:00 +CPU= 17.694
  14:27:28 STEP 1650 H=1650:00 +CPU= 28.754
  14:27:45 STEP 1651 H=1651:00 +CPU= 17.750
  14:28:04 STEP 1652 H=1652:00 +CPU= 18.246
  14:28:32 STEP 1653 H=1653:00 +CPU= 28.226
  14:28:50 STEP 1654 H=1654:00 +CPU= 18.263
  14:29:09 STEP 1655 H=1655:00 +CPU= 18.304
  14:29:42 STEP 1656 H=1656:00 +CPU= 33.009
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_68.zip
Uploading the intermediate file: upload_file_68.zip
  14:30:00 STEP 1657 H=1657:00 +CPU= 17.734
  14:30:18 STEP 1658 H=1658:00 +CPU= 18.250
  14:30:46 STEP 1659 H=1659:00 +CPU= 28.204
  14:31:05 STEP 1660 H=1660:00 +CPU= 18.275
  14:31:23 STEP 1661 H=1661:00 +CPU= 17.876
  14:31:52 STEP 1662 H=1662:00 +CPU= 28.991
  14:32:10 STEP 1663 H=1663:00 +CPU= 17.937
  14:32:29 STEP 1664 H=1664:00 +CPU= 18.517
  14:32:57 STEP 1665 H=1665:00 +CPU= 28.447
  14:33:16 STEP 1666 H=1666:00 +CPU= 18.584
  14:33:34 STEP 1667 H=1667:00 +CPU= 17.898
  14:34:06 STEP 1668 H=1668:00 +CPU= 31.691
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001668
  14:34:24 STEP 1669 H=1669:00 +CPU= 18.037
  14:34:43 STEP 1670 H=1670:00 +CPU= 18.479
  14:35:12 STEP 1671 H=1671:00 +CPU= 28.662
  14:35:31 STEP 1672 H=1672:00 +CPU= 18.446
  14:35:49 STEP 1673 H=1673:00 +CPU= 17.989
  14:36:18 STEP 1674 H=1674:00 +CPU= 29.130
  14:36:37 STEP 1675 H=1675:00 +CPU= 18.216
  14:36:55 STEP 1676 H=1676:00 +CPU= 18.354
  14:37:23 STEP 1677 H=1677:00 +CPU= 28.332
  14:37:42 STEP 1678 H=1678:00 +CPU= 18.524
  14:38:01 STEP 1679 H=1679:00 +CPU= 18.440
  14:38:34 STEP 1680 H=1680:00 +CPU= 33.280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_69.zip
Uploading the intermediate file: upload_file_69.zip
  14:38:52 STEP 1681 H=1681:00 +CPU= 17.876
  14:39:11 STEP 1682 H=1682:00 +CPU= 18.456
  14:39:40 STEP 1683 H=1683:00 +CPU= 28.429
  14:39:58 STEP 1684 H=1684:00 +CPU= 18.561
  14:40:16 STEP 1685 H=1685:00 +CPU= 17.800
  14:40:46 STEP 1686 H=1686:00 +CPU= 28.941
  14:41:04 STEP 1687 H=1687:00 +CPU= 17.889
  14:41:22 STEP 1688 H=1688:00 +CPU= 18.323
  14:41:51 STEP 1689 H=1689:00 +CPU= 28.519
  14:42:09 STEP 1690 H=1690:00 +CPU= 18.370
  14:42:27 STEP 1691 H=1691:00 +CPU= 17.890
  14:42:59 STEP 1692 H=1692:00 +CPU= 31.372
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001692
  14:43:17 STEP 1693 H=1693:00 +CPU= 18.084
  14:43:36 STEP 1694 H=1694:00 +CPU= 18.549
  14:44:06 STEP 1695 H=1695:00 +CPU= 29.998
  14:44:25 STEP 1696 H=1696:00 +CPU= 18.811
  14:44:43 STEP 1697 H=1697:00 +CPU= 17.872
  14:45:12 STEP 1698 H=1698:00 +CPU= 28.910
  14:45:30 STEP 1699 H=1699:00 +CPU= 17.944
  14:45:49 STEP 1700 H=1700:00 +CPU= 18.303
  14:46:17 STEP 1701 H=1701:00 +CPU= 28.239
  14:46:35 STEP 1702 H=1702:00 +CPU= 18.340
  14:46:54 STEP 1703 H=1703:00 +CPU= 18.340
  14:47:27 STEP 1704 H=1704:00 +CPU= 33.088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_70.zip
Uploading the intermediate file: upload_file_70.zip
  14:47:45 STEP 1705 H=1705:00 +CPU= 17.816
  14:48:03 STEP 1706 H=1706:00 +CPU= 18.218
  14:48:32 STEP 1707 H=1707:00 +CPU= 28.361
  14:48:50 STEP 1708 H=1708:00 +CPU= 18.281
  14:49:08 STEP 1709 H=1709:00 +CPU= 17.819
  14:49:37 STEP 1710 H=1710:00 +CPU= 28.840
  14:49:55 STEP 1711 H=1711:00 +CPU= 18.045
  14:50:13 STEP 1712 H=1712:00 +CPU= 18.293
  14:50:41 STEP 1713 H=1713:00 +CPU= 28.243
  14:51:00 STEP 1714 H=1714:00 +CPU= 18.454
  14:51:18 STEP 1715 H=1715:00 +CPU= 17.809
  14:51:49 STEP 1716 H=1716:00 +CPU= 31.257
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001716
  14:52:07 STEP 1717 H=1717:00 +CPU= 17.783
  14:52:25 STEP 1718 H=1718:00 +CPU= 18.361
  14:52:54 STEP 1719 H=1719:00 +CPU= 28.447
  14:53:12 STEP 1720 H=1720:00 +CPU= 18.302
  14:53:30 STEP 1721 H=1721:00 +CPU= 17.896
  14:53:59 STEP 1722 H=1722:00 +CPU= 28.799
  14:54:17 STEP 1723 H=1723:00 +CPU= 17.761
  14:54:35 STEP 1724 H=1724:00 +CPU= 18.291
  14:55:03 STEP 1725 H=1725:00 +CPU= 28.137
  14:55:22 STEP 1726 H=1726:00 +CPU= 18.263
  14:55:40 STEP 1727 H=1727:00 +CPU= 18.277
  14:56:13 STEP 1728 H=1728:00 +CPU= 33.203
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_71.zip
Uploading the intermediate file: upload_file_71.zip
  14:56:31 STEP 1729 H=1729:00 +CPU= 17.944
  14:56:50 STEP 1730 H=1730:00 +CPU= 18.227
  14:57:18 STEP 1731 H=1731:00 +CPU= 28.209
  14:57:36 STEP 1732 H=1732:00 +CPU= 18.285
  14:57:54 STEP 1733 H=1733:00 +CPU= 17.731
  14:58:23 STEP 1734 H=1734:00 +CPU= 28.817
  14:58:41 STEP 1735 H=1735:00 +CPU= 17.834
  14:58:59 STEP 1736 H=1736:00 +CPU= 18.291
  14:59:28 STEP 1737 H=1737:00 +CPU= 28.257
  14:59:46 STEP 1738 H=1738:00 +CPU= 18.344
  15:00:04 STEP 1739 H=1739:00 +CPU= 18.145
  15:00:36 STEP 1740 H=1740:00 +CPU= 31.332
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001740
  15:00:54 STEP 1741 H=1741:00 +CPU= 17.846
  15:01:13 STEP 1742 H=1742:00 +CPU= 18.579
  15:01:41 STEP 1743 H=1743:00 +CPU= 28.247
  15:02:00 STEP 1744 H=1744:00 +CPU= 18.324
  15:02:18 STEP 1745 H=1745:00 +CPU= 18.070
  15:02:47 STEP 1746 H=1746:00 +CPU= 28.997
  15:03:05 STEP 1747 H=1747:00 +CPU= 17.802
  15:03:24 STEP 1748 H=1748:00 +CPU= 18.341
  15:03:53 STEP 1749 H=1749:00 +CPU= 28.757
  15:04:11 STEP 1750 H=1750:00 +CPU= 18.309
  15:04:30 STEP 1751 H=1751:00 +CPU= 18.349
  15:05:04 STEP 1752 H=1752:00 +CPU= 33.502
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_72.zip
Uploading the intermediate file: upload_file_72.zip
  15:05:22 STEP 1753 H=1753:00 +CPU= 17.920
  15:05:40 STEP 1754 H=1754:00 +CPU= 18.375
  15:06:09 STEP 1755 H=1755:00 +CPU= 28.533
  15:06:28 STEP 1756 H=1756:00 +CPU= 18.520
  15:06:46 STEP 1757 H=1757:00 +CPU= 17.869
  15:07:15 STEP 1758 H=1758:00 +CPU= 28.996
  15:07:33 STEP 1759 H=1759:00 +CPU= 17.935
  15:07:51 STEP 1760 H=1760:00 +CPU= 18.398
  15:08:20 STEP 1761 H=1761:00 +CPU= 28.510
  15:08:39 STEP 1762 H=1762:00 +CPU= 18.417
  15:08:57 STEP 1763 H=1763:00 +CPU= 17.999
  15:09:29 STEP 1764 H=1764:00 +CPU= 31.439
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001764
  15:09:47 STEP 1765 H=1765:00 +CPU= 17.910
  15:10:05 STEP 1766 H=1766:00 +CPU= 18.495
  15:10:34 STEP 1767 H=1767:00 +CPU= 28.576
  15:10:53 STEP 1768 H=1768:00 +CPU= 18.499
  15:11:11 STEP 1769 H=1769:00 +CPU= 18.016
  15:11:40 STEP 1770 H=1770:00 +CPU= 29.108
  15:11:58 STEP 1771 H=1771:00 +CPU= 17.963
  15:12:17 STEP 1772 H=1772:00 +CPU= 18.392
  15:12:45 STEP 1773 H=1773:00 +CPU= 28.490
  15:13:04 STEP 1774 H=1774:00 +CPU= 18.361
  15:13:22 STEP 1775 H=1775:00 +CPU= 18.301
  15:13:55 STEP 1776 H=1776:00 +CPU= 33.020
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_73.zip
Uploading the intermediate file: upload_file_73.zip
  15:14:13 STEP 1777 H=1777:00 +CPU= 17.813
  15:14:32 STEP 1778 H=1778:00 +CPU= 18.240
  15:15:00 STEP 1779 H=1779:00 +CPU= 28.302
  15:15:18 STEP 1780 H=1780:00 +CPU= 18.305
  15:15:36 STEP 1781 H=1781:00 +CPU= 17.772
  15:16:05 STEP 1782 H=1782:00 +CPU= 28.811
  15:16:23 STEP 1783 H=1783:00 +CPU= 17.796
  15:16:41 STEP 1784 H=1784:00 +CPU= 18.286
  15:17:10 STEP 1785 H=1785:00 +CPU= 28.150
  15:17:28 STEP 1786 H=1786:00 +CPU= 18.480
  15:17:46 STEP 1787 H=1787:00 +CPU= 17.800
  15:18:17 STEP 1788 H=1788:00 +CPU= 31.105
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001788
  15:18:35 STEP 1789 H=1789:00 +CPU= 17.927
  15:18:54 STEP 1790 H=1790:00 +CPU= 18.362
  15:19:22 STEP 1791 H=1791:00 +CPU= 28.323
  15:19:41 STEP 1792 H=1792:00 +CPU= 18.311
  15:19:58 STEP 1793 H=1793:00 +CPU= 17.874
  15:20:27 STEP 1794 H=1794:00 +CPU= 28.769
  15:20:45 STEP 1795 H=1795:00 +CPU= 17.816
  15:21:04 STEP 1796 H=1796:00 +CPU= 18.339
  15:21:32 STEP 1797 H=1797:00 +CPU= 28.181
  15:21:50 STEP 1798 H=1798:00 +CPU= 18.264
  15:22:09 STEP 1799 H=1799:00 +CPU= 18.337
  15:22:42 STEP 1800 H=1800:00 +CPU= 32.961
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_74.zip
Uploading the intermediate file: upload_file_74.zip
  15:23:00 STEP 1801 H=1801:00 +CPU= 17.812
  15:23:18 STEP 1802 H=1802:00 +CPU= 18.215
  15:23:46 STEP 1803 H=1803:00 +CPU= 28.149
  15:24:04 STEP 1804 H=1804:00 +CPU= 18.241
  15:24:22 STEP 1805 H=1805:00 +CPU= 17.722
  15:24:51 STEP 1806 H=1806:00 +CPU= 28.885
  15:25:09 STEP 1807 H=1807:00 +CPU= 17.762
  15:25:27 STEP 1808 H=1808:00 +CPU= 18.335
  15:25:55 STEP 1809 H=1809:00 +CPU= 28.159
  15:26:14 STEP 1810 H=1810:00 +CPU= 18.304
  15:26:32 STEP 1811 H=1811:00 +CPU= 17.737
  15:27:03 STEP 1812 H=1812:00 +CPU= 31.164
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001812
  15:27:21 STEP 1813 H=1813:00 +CPU= 17.824
  15:27:39 STEP 1814 H=1814:00 +CPU= 18.404
  15:28:08 STEP 1815 H=1815:00 +CPU= 28.305
  15:28:27 STEP 1816 H=1816:00 +CPU= 18.400
  15:28:45 STEP 1817 H=1817:00 +CPU= 17.887
  15:29:13 STEP 1818 H=1818:00 +CPU= 28.715
  15:29:31 STEP 1819 H=1819:00 +CPU= 17.773
  15:29:50 STEP 1820 H=1820:00 +CPU= 18.286
  15:30:19 STEP 1821 H=1821:00 +CPU= 29.018
  15:30:39 STEP 1822 H=1822:00 +CPU= 19.044
  15:30:58 STEP 1823 H=1823:00 +CPU= 18.690
  15:31:32 STEP 1824 H=1824:00 +CPU= 33.400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_75.zip
Uploading the intermediate file: upload_file_75.zip
  15:31:51 STEP 1825 H=1825:00 +CPU= 18.655
Uploading trickle at timestep: 6566400
  15:32:12 STEP 1826 H=1826:00 +CPU= 19.417
  15:32:44 STEP 1827 H=1827:00 +CPU= 29.736
  15:33:03 STEP 1828 H=1828:00 +CPU= 18.952
  15:33:22 STEP 1829 H=1829:00 +CPU= 18.393
  15:33:53 STEP 1830 H=1830:00 +CPU= 29.487
  15:34:12 STEP 1831 H=1831:00 +CPU= 18.336
  15:34:31 STEP 1832 H=1832:00 +CPU= 18.878
  15:35:01 STEP 1833 H=1833:00 +CPU= 28.797
  15:35:20 STEP 1834 H=1834:00 +CPU= 19.067
  15:35:39 STEP 1835 H=1835:00 +CPU= 18.190
  15:36:11 STEP 1836 H=1836:00 +CPU= 31.457
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001836
  15:36:29 STEP 1837 H=1837:00 +CPU= 17.937
  15:36:48 STEP 1838 H=1838:00 +CPU= 18.449
  15:37:16 STEP 1839 H=1839:00 +CPU= 28.435
  15:37:35 STEP 1840 H=1840:00 +CPU= 18.404
  15:37:53 STEP 1841 H=1841:00 +CPU= 17.918
  15:38:22 STEP 1842 H=1842:00 +CPU= 29.021
  15:38:40 STEP 1843 H=1843:00 +CPU= 17.941
  15:38:59 STEP 1844 H=1844:00 +CPU= 18.601
  15:39:29 STEP 1845 H=1845:00 +CPU= 28.882
  15:39:49 STEP 1846 H=1846:00 +CPU= 19.062
  15:40:08 STEP 1847 H=1847:00 +CPU= 18.619
  15:40:42 STEP 1848 H=1848:00 +CPU= 33.595
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_76.zip
Uploading the intermediate file: upload_file_76.zip
  15:41:02 STEP 1849 H=1849:00 +CPU= 18.532
  15:41:21 STEP 1850 H=1850:00 +CPU= 18.558
  15:41:50 STEP 1851 H=1851:00 +CPU= 28.338
  15:42:10 STEP 1852 H=1852:00 +CPU= 19.088
  15:42:29 STEP 1853 H=1853:00 +CPU= 18.686
  15:43:01 STEP 1854 H=1854:00 +CPU= 30.045
  15:43:21 STEP 1855 H=1855:00 +CPU= 18.588
  15:43:41 STEP 1856 H=1856:00 +CPU= 19.179
  15:44:11 STEP 1857 H=1857:00 +CPU= 29.265
  15:44:30 STEP 1858 H=1858:00 +CPU= 18.831
  15:44:51 STEP 1859 H=1859:00 +CPU= 18.954
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:45:37 STEP 1860 H=1860:00 +CPU= 32.900
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001860
  15:45:56 STEP 1861 H=1861:00 +CPU= 18.580
  15:46:16 STEP 1862 H=1862:00 +CPU= 19.116
  15:46:47 STEP 1863 H=1863:00 +CPU= 29.612
  15:47:08 STEP 1864 H=1864:00 +CPU= 19.408
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:47:48 STEP 1865 H=1865:00 +CPU= 18.794
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:49:31 STEP 1866 H=1866:00 +CPU= 30.147
  15:49:50 STEP 1867 H=1867:00 +CPU= 18.245
  15:50:09 STEP 1868 H=1868:00 +CPU= 18.797
  15:50:38 STEP 1869 H=1869:00 +CPU= 28.776
  15:50:57 STEP 1870 H=1870:00 +CPU= 18.699
  15:51:17 STEP 1871 H=1871:00 +CPU= 18.963
  15:51:52 STEP 1872 H=1872:00 +CPU= 33.985
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_77.zip
Uploading the intermediate file: upload_file_77.zip
  15:52:10 STEP 1873 H=1873:00 +CPU= 18.163
  15:52:29 STEP 1874 H=1874:00 +CPU= 18.502
  15:52:58 STEP 1875 H=1875:00 +CPU= 28.754
  15:53:17 STEP 1876 H=1876:00 +CPU= 18.898
  15:53:36 STEP 1877 H=1877:00 +CPU= 18.007
  15:54:05 STEP 1878 H=1878:00 +CPU= 29.307
  15:54:24 STEP 1879 H=1879:00 +CPU= 18.280
  15:54:44 STEP 1880 H=1880:00 +CPU= 18.997
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:55:56 STEP 1881 H=1881:00 +CPU= 29.405
  15:56:16 STEP 1882 H=1882:00 +CPU= 19.067
  15:56:35 STEP 1883 H=1883:00 +CPU= 18.446
  15:57:08 STEP 1884 H=1884:00 +CPU= 31.897
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001884
  15:57:26 STEP 1885 H=1885:00 +CPU= 18.157
  15:57:45 STEP 1886 H=1886:00 +CPU= 18.533
  15:58:14 STEP 1887 H=1887:00 +CPU= 28.413
  15:58:34 STEP 1888 H=1888:00 +CPU= 18.987
  15:58:52 STEP 1889 H=1889:00 +CPU= 18.241
  15:59:22 STEP 1890 H=1890:00 +CPU= 29.049
  15:59:40 STEP 1891 H=1891:00 +CPU= 17.881
  15:59:59 STEP 1892 H=1892:00 +CPU= 18.661
  16:00:27 STEP 1893 H=1893:00 +CPU= 28.493
  16:00:46 STEP 1894 H=1894:00 +CPU= 18.386
  16:01:05 STEP 1895 H=1895:00 +CPU= 18.663
  16:01:39 STEP 1896 H=1896:00 +CPU= 33.603
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_78.zip
Uploading the intermediate file: upload_file_78.zip
  16:01:58 STEP 1897 H=1897:00 +CPU= 18.186
  16:02:17 STEP 1898 H=1898:00 +CPU= 19.210
  16:02:47 STEP 1899 H=1899:00 +CPU= 28.887
  16:03:06 STEP 1900 H=1900:00 +CPU= 18.541
  16:03:24 STEP 1901 H=1901:00 +CPU= 18.195
  16:03:55 STEP 1902 H=1902:00 +CPU= 29.722
  16:04:14 STEP 1903 H=1903:00 +CPU= 18.428
  16:04:33 STEP 1904 H=1904:00 +CPU= 18.719
  16:05:02 STEP 1905 H=1905:00 +CPU= 28.842
  16:05:21 STEP 1906 H=1906:00 +CPU= 18.557
  16:05:39 STEP 1907 H=1907:00 +CPU= 18.142
  16:06:11 STEP 1908 H=1908:00 +CPU= 31.578
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001908
  16:06:30 STEP 1909 H=1909:00 +CPU= 17.990
  16:06:48 STEP 1910 H=1910:00 +CPU= 18.565
  16:07:17 STEP 1911 H=1911:00 +CPU= 28.490
  16:07:36 STEP 1912 H=1912:00 +CPU= 18.513
  16:07:54 STEP 1913 H=1913:00 +CPU= 18.072
  16:08:24 STEP 1914 H=1914:00 +CPU= 29.122
  16:08:42 STEP 1915 H=1915:00 +CPU= 18.034
  16:09:01 STEP 1916 H=1916:00 +CPU= 18.637
  16:09:30 STEP 1917 H=1917:00 +CPU= 28.736
  16:09:49 STEP 1918 H=1918:00 +CPU= 18.708
  16:10:08 STEP 1919 H=1919:00 +CPU= 18.872
  16:10:44 STEP 1920 H=1920:00 +CPU= 34.255
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_79.zip
Uploading the intermediate file: upload_file_79.zip
  16:11:03 STEP 1921 H=1921:00 +CPU= 18.317
  16:11:22 STEP 1922 H=1922:00 +CPU= 18.830
  16:11:51 STEP 1923 H=1923:00 +CPU= 28.802
  16:12:10 STEP 1924 H=1924:00 +CPU= 18.665
  16:12:29 STEP 1925 H=1925:00 +CPU= 18.097
  16:12:59 STEP 1926 H=1926:00 +CPU= 29.570
  16:13:17 STEP 1927 H=1927:00 +CPU= 18.008
  16:13:36 STEP 1928 H=1928:00 +CPU= 18.835
  16:14:06 STEP 1929 H=1929:00 +CPU= 29.172
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:15:07 STEP 1930 H=1930:00 +CPU= 19.130
  16:15:26 STEP 1931 H=1931:00 +CPU= 18.383
  16:15:58 STEP 1932 H=1932:00 +CPU= 31.608
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001932
  16:16:17 STEP 1933 H=1933:00 +CPU= 18.056
  16:16:36 STEP 1934 H=1934:00 +CPU= 18.778
  16:17:05 STEP 1935 H=1935:00 +CPU= 28.724
  16:17:24 STEP 1936 H=1936:00 +CPU= 18.563
  16:17:42 STEP 1937 H=1937:00 +CPU= 18.228
  16:18:12 STEP 1938 H=1938:00 +CPU= 29.361
  16:18:31 STEP 1939 H=1939:00 +CPU= 18.198
  16:18:50 STEP 1940 H=1940:00 +CPU= 18.670
  16:19:18 STEP 1941 H=1941:00 +CPU= 28.474
  16:19:38 STEP 1942 H=1942:00 +CPU= 18.903
  16:19:57 STEP 1943 H=1943:00 +CPU= 18.553
  16:20:31 STEP 1944 H=1944:00 +CPU= 33.638
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_80.zip
Uploading the intermediate file: upload_file_80.zip
  16:20:49 STEP 1945 H=1945:00 +CPU= 18.066
  16:21:08 STEP 1946 H=1946:00 +CPU= 18.595
  16:21:38 STEP 1947 H=1947:00 +CPU= 28.996
  16:21:58 STEP 1948 H=1948:00 +CPU= 19.020
  16:22:17 STEP 1949 H=1949:00 +CPU= 18.326
  16:22:48 STEP 1950 H=1950:00 +CPU= 29.848
  16:23:07 STEP 1951 H=1951:00 +CPU= 18.177
  16:23:25 STEP 1952 H=1952:00 +CPU= 18.427
  16:23:54 STEP 1953 H=1953:00 +CPU= 28.566
  16:24:13 STEP 1954 H=1954:00 +CPU= 18.626
  16:24:32 STEP 1955 H=1955:00 +CPU= 18.163
  16:25:04 STEP 1956 H=1956:00 +CPU= 31.432
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001956
  16:25:22 STEP 1957 H=1957:00 +CPU= 18.027
  16:25:41 STEP 1958 H=1958:00 +CPU= 18.657
  16:26:09 STEP 1959 H=1959:00 +CPU= 28.539
  16:26:28 STEP 1960 H=1960:00 +CPU= 18.654
  16:26:47 STEP 1961 H=1961:00 +CPU= 18.250
  16:27:17 STEP 1962 H=1962:00 +CPU= 29.587
  16:27:36 STEP 1963 H=1963:00 +CPU= 18.059
  16:27:54 STEP 1964 H=1964:00 +CPU= 18.583
  16:28:23 STEP 1965 H=1965:00 +CPU= 28.536
  16:28:43 STEP 1966 H=1966:00 +CPU= 18.878
  16:29:03 STEP 1967 H=1967:00 +CPU= 19.270
  16:29:37 STEP 1968 H=1968:00 +CPU= 33.736
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_81.zip
Uploading the intermediate file: upload_file_81.zip
  16:29:56 STEP 1969 H=1969:00 +CPU= 18.103
  16:30:15 STEP 1970 H=1970:00 +CPU= 18.506
  16:30:43 STEP 1971 H=1971:00 +CPU= 28.519
  16:31:02 STEP 1972 H=1972:00 +CPU= 18.443
  16:31:20 STEP 1973 H=1973:00 +CPU= 18.000
  16:31:50 STEP 1974 H=1974:00 +CPU= 29.058
  16:32:09 STEP 1975 H=1975:00 +CPU= 18.266
  16:32:28 STEP 1976 H=1976:00 +CPU= 18.635
  16:32:57 STEP 1977 H=1977:00 +CPU= 28.529
  16:33:15 STEP 1978 H=1978:00 +CPU= 18.620
  16:33:34 STEP 1979 H=1979:00 +CPU= 17.952
  16:34:06 STEP 1980 H=1980:00 +CPU= 31.775
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001980
  16:34:25 STEP 1981 H=1981:00 +CPU= 18.283
  16:34:44 STEP 1982 H=1982:00 +CPU= 18.645
  16:35:13 STEP 1983 H=1983:00 +CPU= 28.606
  16:35:32 STEP 1984 H=1984:00 +CPU= 18.481
  16:35:50 STEP 1985 H=1985:00 +CPU= 18.010
  16:36:20 STEP 1986 H=1986:00 +CPU= 29.415
  16:36:38 STEP 1987 H=1987:00 +CPU= 17.951
  16:36:57 STEP 1988 H=1988:00 +CPU= 18.570
  16:37:26 STEP 1989 H=1989:00 +CPU= 28.547
  16:37:44 STEP 1990 H=1990:00 +CPU= 18.435
  16:38:03 STEP 1991 H=1991:00 +CPU= 18.453
  16:38:37 STEP 1992 H=1992:00 +CPU= 33.763
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_82.zip
Uploading the intermediate file: upload_file_82.zip
  16:38:56 STEP 1993 H=1993:00 +CPU= 18.074
  16:39:15 STEP 1994 H=1994:00 +CPU= 18.702
  16:39:43 STEP 1995 H=1995:00 +CPU= 28.393
  16:40:02 STEP 1996 H=1996:00 +CPU= 18.450
  16:40:20 STEP 1997 H=1997:00 +CPU= 17.954
  16:40:49 STEP 1998 H=1998:00 +CPU= 29.014
  16:41:08 STEP 1999 H=1999:00 +CPU= 18.173
  16:41:27 STEP 2000 H=2000:00 +CPU= 18.833
  16:41:56 STEP 2001 H=2001:00 +CPU= 28.723
  16:42:16 STEP 2002 H=2002:00 +CPU= 18.722
  16:42:34 STEP 2003 H=2003:00 +CPU= 18.271
  16:43:07 STEP 2004 H=2004:00 +CPU= 31.917
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002004
  16:43:26 STEP 2005 H=2005:00 +CPU= 18.764
  16:43:46 STEP 2006 H=2006:00 +CPU= 19.043
  16:44:15 STEP 2007 H=2007:00 +CPU= 28.589
  16:44:34 STEP 2008 H=2008:00 +CPU= 18.522
  16:44:52 STEP 2009 H=2009:00 +CPU= 18.111
  16:45:21 STEP 2010 H=2010:00 +CPU= 29.110
  16:45:39 STEP 2011 H=2011:00 +CPU= 17.949
  16:45:58 STEP 2012 H=2012:00 +CPU= 18.642
  16:46:27 STEP 2013 H=2013:00 +CPU= 28.633
  16:46:46 STEP 2014 H=2014:00 +CPU= 18.658
  16:47:06 STEP 2015 H=2015:00 +CPU= 19.075
  16:47:41 STEP 2016 H=2016:00 +CPU= 34.103
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_83.zip
Uploading the intermediate file: upload_file_83.zip
  16:47:59 STEP 2017 H=2017:00 +CPU= 18.144
  16:48:19 STEP 2018 H=2018:00 +CPU= 18.868
  16:48:49 STEP 2019 H=2019:00 +CPU= 29.186
  16:49:09 STEP 2020 H=2020:00 +CPU= 19.115
  16:49:28 STEP 2021 H=2021:00 +CPU= 18.716
  16:49:59 STEP 2022 H=2022:00 +CPU= 29.714
  16:50:18 STEP 2023 H=2023:00 +CPU= 18.423
  16:50:38 STEP 2024 H=2024:00 +CPU= 19.077
  16:51:07 STEP 2025 H=2025:00 +CPU= 28.673
  16:51:27 STEP 2026 H=2026:00 +CPU= 19.176
  16:51:46 STEP 2027 H=2027:00 +CPU= 18.319
  16:52:19 STEP 2028 H=2028:00 +CPU= 32.317
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002028
  16:52:38 STEP 2029 H=2029:00 +CPU= 18.235
  16:52:57 STEP 2030 H=2030:00 +CPU= 18.821
  16:53:28 STEP 2031 H=2031:00 +CPU= 29.392
  16:53:48 STEP 2032 H=2032:00 +CPU= 19.116
  16:54:07 STEP 2033 H=2033:00 +CPU= 18.597
  16:54:38 STEP 2034 H=2034:00 +CPU= 30.010
  16:54:57 STEP 2035 H=2035:00 +CPU= 18.328
  16:55:16 STEP 2036 H=2036:00 +CPU= 19.000
  16:55:47 STEP 2037 H=2037:00 +CPU= 29.501
  16:56:07 STEP 2038 H=2038:00 +CPU= 18.901
  16:56:26 STEP 2039 H=2039:00 +CPU= 18.993
  16:57:02 STEP 2040 H=2040:00 +CPU= 34.281
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199145/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0501_2013050100_123_982_12199145_0_r667008025_84.zip
Uploading the intermediate file: upload_file_84.zip
  16:57:21 STEP 2041 H=2041:00 +CPU= 18.529
  16:57:40 STEP 2042 H=2042:00 +CPU= 18.940
  16:58:10 STEP 2043 H=2043:00 +CPU= 29.179
  16:58:31 STEP 2044 H=2044:00 +CPU= 19.205
  16:58:50 STEP 2045 H=2045:00 +CPU= 18.567
  16:59:21 STEP 2046 H=2046:00 +CPU= 29.754
  16:59:40 STEP 2047 H=2047:00 +CPU= 18.443
  16:59:59 STEP 2048 H=2048:00 +CPU= 19.027
  17:00:29 STEP 2049 H=2049:00 +CPU= 29.055
  17:00:48 STEP 2050 H=2050:00 +CPU= 18.906
  17:01:07 STEP 2051 H=2051:00 +CPU= 18.411
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:16:50 1509499 22285298 oifs_43r3_ps_0501_2013050100_123_982_12199145_0 6,566,400 43,338 0.0066
09 May 2023 16:15:40 1509499 22285298 oifs_43r3_ps_0501_2013050100_123_982_12199145_0 5,529,600 36,513 0.0066
09 May 2023 16:14:55 1509499 22285298 oifs_43r3_ps_0501_2013050100_123_982_12199145_0 4,406,400 28,827 0.0065
09 May 2023 16:14:01 1509499 22285298 oifs_43r3_ps_0501_2013050100_123_982_12199145_0 2,592,000 17,673 0.0068
09 May 2023 16:13:59 1509499 22285298 oifs_43r3_ps_0501_2013050100_123_982_12199145_0 1,728,000 12,133 0.0070


©2024 climateprediction.net