climateprediction.net home page
Task 22435493

Task 22435493

Name oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0
Workunit 12283914
Created 7 Jun 2024, 13:51:58 UTC
Sent 7 Jun 2024, 14:55:37 UTC
Report deadline 6 Aug 2024, 14:55:37 UTC
Received 8 Jun 2024, 14:40:42 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1532094
Run time 13 hours 33 min 20 sec
CPU time 13 hours 33 min 20 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 4.58 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,003.79 MB
Peak swap size 5,980.80 MB
Peak disk usage 2,172.73 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
      7 RECORDS TRANSFERRED IN    0.0004 SECONDS761588.7 Mbytes/s, TOTAL TIME=    0.0141( 2.8%)
 12:14:27 STEP 1140 H= 285:00 +CPU= 37.314
 NSTEP =  1141 STEPO   0AAA00AAA
 12:15:02 STEP 1141 H= 285:15 +CPU= 30.827
 NSTEP =  1142 STEPO   0AAA00AAA
 12:15:33 STEP 1142 H= 285:30 +CPU= 31.467
 NSTEP =  1143 STEPO   0AAA00AAA
 12:16:04 STEP 1143 H= 285:45 +CPU= 31.268
 NSTEP =  1144 STEPO   0AAA00AAA
 12:16:36 STEP 1144 H= 286:00 +CPU= 31.579
ISTEP=    1145 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1145 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1145 STEPO   0AAA00AAA
 12:17:07 STEP 1145 H= 286:15 +CPU= 31.130
 NSTEP =  1146 STEPO   0AAA00AAA
 12:17:38 STEP 1146 H= 286:30 +CPU= 30.654
 NSTEP =  1147 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 12:14:27 0AAA00AAA STEPO     1141    30.993   30.993   34.257     70:31     72:44 0.23143649276202E-04       2GB       0MB
 12:15:02 0AAA00AAA STEPO     1142    30.827   30.827   34.205     71:02     73:19 0.23143649276202E-04       2GB       0MB
 12:15:33 0AAA00AAA STEPO     1143    31.467   31.467   31.588     71:34     73:50 0.23143649276202E-04       2GB       0MB
 12:16:04 0AAA00AAA STEPO     1144    31.268   31.268   31.289     72:05     74:21 0.23143649276202E-04       2GB       0MB
 12:16:36 0AAA00AAA STEPO     1145    31.579   31.579   31.621     72:36     74:53 0.23143649276202E-04       2GB       0MB
 12:17:07 0AAA00AAA STEPO     1146    31.130   31.130   31.178     73:08     75:24 0.23143649276202E-04       2GB       0MB
 12:17:38 0AAA00AAA STEPO     1147    30.655   30.655   30.674     73:38     75:55 0.23143649276202E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1104",
 CTIME   = "00111200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1092, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/3/progress_file_12283914.xml

<running_values>
  <last_cpu_time>37475.850000</last_cpu_time>
  <upload_file_number>11</upload_file_number>
  <last_iter>1146</last_iter>
  <last_upload>950400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1lz
(argv4) batchid: 1017
(argv5) wuid: 12283914
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_cf6d62c72129e21b60ce5c63ab3f0e53 to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1lz_2016092300_20_1017_12283914.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1lz_2016092300_20_1017_12283914.zip
ic_ancil_file: ic_ancil_12283914
ifsdata_file: ifsdata_12283914
climate_data_file: clim_data_12283914
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/3/ic_ancil_12283914.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12283914.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_83d9cdc2ed20b5df09ee0d699c108825 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12283914.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12283914.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12283914.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12283914.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12283914.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12283914.xml
-- Model is restarting --
Adjusting last_iter, 1146, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12283914.xml
last_cpu_time: 37475
upload_file_number: 11
last_iter: 1105
last_upload: 950400
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918
The child process has been launched with process id: 13546
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:13546:13546] [20240608:123408:1717842848:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:36:02 STEP 1104 H= 276:00 +CPU= 22.788
  12:36:28 STEP 1105 H= 276:15 +CPU= 25.790
  12:36:57 STEP 1106 H= 276:30 +CPU= 29.692
  12:37:28 STEP 1107 H= 276:45 +CPU= 30.273
  12:37:59 STEP 1108 H= 277:00 +CPU= 31.040
  12:38:29 STEP 1109 H= 277:15 +CPU= 30.624
  12:39:00 STEP 1110 H= 277:30 +CPU= 30.271
  12:39:30 STEP 1111 H= 277:45 +CPU= 30.282
  12:40:01 STEP 1112 H= 278:00 +CPU= 30.771
  12:40:31 STEP 1113 H= 278:15 +CPU= 30.364
  12:41:02 STEP 1114 H= 278:30 +CPU= 31.211
  12:41:33 STEP 1115 H= 278:45 +CPU= 30.107
  12:42:10 STEP 1116 H= 279:00 +CPU= 36.776
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001116
  12:42:41 STEP 1117 H= 279:15 +CPU= 30.651
  12:43:12 STEP 1118 H= 279:30 +CPU= 31.111
  12:43:43 STEP 1119 H= 279:45 +CPU= 30.748
  12:44:14 STEP 1120 H= 280:00 +CPU= 31.173
  12:44:45 STEP 1121 H= 280:15 +CPU= 31.027
  12:45:15 STEP 1122 H= 280:30 +CPU= 30.516
  12:45:46 STEP 1123 H= 280:45 +CPU= 30.263
  12:46:17 STEP 1124 H= 281:00 +CPU= 31.126
  12:46:47 STEP 1125 H= 281:15 +CPU= 30.126
  12:47:18 STEP 1126 H= 281:30 +CPU= 30.790
  12:47:49 STEP 1127 H= 281:45 +CPU= 30.866
  12:48:26 STEP 1128 H= 282:00 +CPU= 37.176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001128
  12:48:56 STEP 1129 H= 282:15 +CPU= 30.512
  12:49:27 STEP 1130 H= 282:30 +CPU= 30.625
  12:49:58 STEP 1131 H= 282:45 +CPU= 31.218
  12:50:35 STEP 1132 H= 283:00 +CPU= 32.882
  12:51:07 STEP 1133 H= 283:15 +CPU= 31.874
  12:51:41 STEP 1134 H= 283:30 +CPU= 32.714
  12:52:15 STEP 1135 H= 283:45 +CPU= 32.817
  12:52:49 STEP 1136 H= 284:00 +CPU= 32.784
  12:53:23 STEP 1137 H= 284:15 +CPU= 31.014
  12:53:58 STEP 1138 H= 284:30 +CPU= 31.918
  12:54:30 STEP 1139 H= 284:45 +CPU= 32.521
  12:55:09 STEP 1140 H= 285:00 +CPU= 38.899
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001140
  12:55:42 STEP 1141 H= 285:15 +CPU= 32.285
  12:56:14 STEP 1142 H= 285:30 +CPU= 32.146
  12:56:46 STEP 1143 H= 285:45 +CPU= 31.913
  12:57:19 STEP 1144 H= 286:00 +CPU= 32.698
  12:57:45 STEP 1145 H= 286:15 +CPU= 26.700
  12:58:16 STEP 1146 H= 286:30 +CPU= 30.433
  12:58:47 STEP 1147 H= 286:45 +CPU= 30.910
  12:59:19 STEP 1148 H= 287:00 +CPU= 30.548
  12:59:49 STEP 1149 H= 287:15 +CPU= 30.123
  13:00:19 STEP 1150 H= 287:30 +CPU= 30.104
  13:00:48 STEP 1151 H= 287:45 +CPU= 29.364
  13:02:19 STEP 1152 H= 288:00 +CPU= 40.540
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_11.zip
Uploading the intermediate file: upload_file_11.zip
  13:02:51 STEP 1153 H= 288:15 +CPU= 30.366
  13:03:27 STEP 1154 H= 288:30 +CPU= 32.040
  13:04:03 STEP 1155 H= 288:45 +CPU= 31.878
  13:04:40 STEP 1156 H= 289:00 +CPU= 32.542
  13:05:15 STEP 1157 H= 289:15 +CPU= 31.742
  13:05:51 STEP 1158 H= 289:30 +CPU= 31.879
  13:06:25 STEP 1159 H= 289:45 +CPU= 31.876
  13:07:00 STEP 1160 H= 290:00 +CPU= 32.676
  13:07:34 STEP 1161 H= 290:15 +CPU= 32.671
  13:08:08 STEP 1162 H= 290:30 +CPU= 33.899
  13:08:41 STEP 1163 H= 290:45 +CPU= 33.800
  13:09:23 STEP 1164 H= 291:00 +CPU= 40.967
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001164
  13:09:56 STEP 1165 H= 291:15 +CPU= 33.172
  13:10:29 STEP 1166 H= 291:30 +CPU= 33.040
  13:11:02 STEP 1167 H= 291:45 +CPU= 33.443
  13:11:36 STEP 1168 H= 292:00 +CPU= 33.491
  13:12:08 STEP 1169 H= 292:15 +CPU= 31.810
  13:12:40 STEP 1170 H= 292:30 +CPU= 31.935
  13:13:11 STEP 1171 H= 292:45 +CPU= 31.591
  13:13:44 STEP 1172 H= 293:00 +CPU= 32.300
  13:14:16 STEP 1173 H= 293:15 +CPU= 32.047
  13:14:48 STEP 1174 H= 293:30 +CPU= 32.328
  13:15:20 STEP 1175 H= 293:45 +CPU= 31.889
  13:16:00 STEP 1176 H= 294:00 +CPU= 39.904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001176
  13:16:32 STEP 1177 H= 294:15 +CPU= 32.057
  13:17:05 STEP 1178 H= 294:30 +CPU= 32.035
  13:17:37 STEP 1179 H= 294:45 +CPU= 32.108
  13:18:10 STEP 1180 H= 295:00 +CPU= 33.018
  13:18:42 STEP 1181 H= 295:15 +CPU= 31.900
  13:19:14 STEP 1182 H= 295:30 +CPU= 32.210
  13:19:46 STEP 1183 H= 295:45 +CPU= 32.068
  13:20:19 STEP 1184 H= 296:00 +CPU= 32.308
  13:20:52 STEP 1185 H= 296:15 +CPU= 31.846
  13:21:25 STEP 1186 H= 296:30 +CPU= 31.486
  13:21:59 STEP 1187 H= 296:45 +CPU= 30.823
  13:22:39 STEP 1188 H= 297:00 +CPU= 37.903
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001188
  13:23:12 STEP 1189 H= 297:15 +CPU= 32.015
  13:23:43 STEP 1190 H= 297:30 +CPU= 31.588
  13:24:16 STEP 1191 H= 297:45 +CPU= 31.168
  13:24:48 STEP 1192 H= 298:00 +CPU= 31.836
  13:25:20 STEP 1193 H= 298:15 +CPU= 30.707
  13:25:51 STEP 1194 H= 298:30 +CPU= 30.650
  13:26:22 STEP 1195 H= 298:45 +CPU= 30.588
  13:26:53 STEP 1196 H= 299:00 +CPU= 31.366
  13:27:23 STEP 1197 H= 299:15 +CPU= 30.229
  13:27:54 STEP 1198 H= 299:30 +CPU= 30.443
  13:28:25 STEP 1199 H= 299:45 +CPU= 30.831
  13:29:18 STEP 1200 H= 300:00 +CPU= 40.465
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001200
  13:29:49 STEP 1201 H= 300:15 +CPU= 30.437
  13:30:19 STEP 1202 H= 300:30 +CPU= 30.566
  13:30:50 STEP 1203 H= 300:45 +CPU= 30.411
  13:31:21 STEP 1204 H= 301:00 +CPU= 31.314
  13:31:52 STEP 1205 H= 301:15 +CPU= 30.424
  13:32:22 STEP 1206 H= 301:30 +CPU= 30.263
  13:32:52 STEP 1207 H= 301:45 +CPU= 30.135
  13:33:23 STEP 1208 H= 302:00 +CPU= 30.795
  13:33:54 STEP 1209 H= 302:15 +CPU= 31.019
  13:34:25 STEP 1210 H= 302:30 +CPU= 30.986
  13:34:57 STEP 1211 H= 302:45 +CPU= 31.485
  13:35:35 STEP 1212 H= 303:00 +CPU= 38.274
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001212
  13:36:06 STEP 1213 H= 303:15 +CPU= 31.075
  13:36:37 STEP 1214 H= 303:30 +CPU= 30.823
  13:37:09 STEP 1215 H= 303:45 +CPU= 31.424
  13:37:40 STEP 1216 H= 304:00 +CPU= 31.712
  13:38:12 STEP 1217 H= 304:15 +CPU= 31.085
  13:38:42 STEP 1218 H= 304:30 +CPU= 30.887
  13:39:13 STEP 1219 H= 304:45 +CPU= 30.477
  13:39:44 STEP 1220 H= 305:00 +CPU= 31.493
  13:40:15 STEP 1221 H= 305:15 +CPU= 30.658
  13:40:46 STEP 1222 H= 305:30 +CPU= 30.654
  13:41:16 STEP 1223 H= 305:45 +CPU= 30.351
  13:41:54 STEP 1224 H= 306:00 +CPU= 37.971
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001224
  13:42:25 STEP 1225 H= 306:15 +CPU= 30.826
  13:42:56 STEP 1226 H= 306:30 +CPU= 31.039
  13:43:27 STEP 1227 H= 306:45 +CPU= 30.447
  13:43:58 STEP 1228 H= 307:00 +CPU= 31.341
  13:44:29 STEP 1229 H= 307:15 +CPU= 30.891
  13:45:00 STEP 1230 H= 307:30 +CPU= 30.971
  13:45:30 STEP 1231 H= 307:45 +CPU= 30.600
  13:46:02 STEP 1232 H= 308:00 +CPU= 31.604
  13:46:33 STEP 1233 H= 308:15 +CPU= 31.112
  13:47:04 STEP 1234 H= 308:30 +CPU= 30.953
  13:47:35 STEP 1235 H= 308:45 +CPU= 31.126
  13:48:10 STEP 1236 H= 309:00 +CPU= 34.777
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001236
  13:48:41 STEP 1237 H= 309:15 +CPU= 30.683
  13:49:11 STEP 1238 H= 309:30 +CPU= 30.050
  13:49:41 STEP 1239 H= 309:45 +CPU= 30.267
  13:50:12 STEP 1240 H= 310:00 +CPU= 30.989
  13:50:43 STEP 1241 H= 310:15 +CPU= 30.481
  13:51:16 STEP 1242 H= 310:30 +CPU= 32.680
  13:51:50 STEP 1243 H= 310:45 +CPU= 33.735
  13:52:24 STEP 1244 H= 311:00 +CPU= 34.171
  13:52:57 STEP 1245 H= 311:15 +CPU= 33.229
  13:53:31 STEP 1246 H= 311:30 +CPU= 33.673
  13:54:05 STEP 1247 H= 311:45 +CPU= 33.902
  13:54:53 STEP 1248 H= 312:00 +CPU= 42.195
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_12.zip
Uploading the intermediate file: upload_file_12.zip
  13:55:26 STEP 1249 H= 312:15 +CPU= 31.322
  13:55:59 STEP 1250 H= 312:30 +CPU= 33.087
  13:56:32 STEP 1251 H= 312:45 +CPU= 33.235
  13:57:06 STEP 1252 H= 313:00 +CPU= 33.245
  13:57:38 STEP 1253 H= 313:15 +CPU= 32.428
  13:58:10 STEP 1254 H= 313:30 +CPU= 32.363
  13:58:42 STEP 1255 H= 313:45 +CPU= 31.840
  13:59:15 STEP 1256 H= 314:00 +CPU= 33.039
  13:59:48 STEP 1257 H= 314:15 +CPU= 32.417
  14:00:20 STEP 1258 H= 314:30 +CPU= 32.430
  14:00:53 STEP 1259 H= 314:45 +CPU= 32.317
  14:01:32 STEP 1260 H= 315:00 +CPU= 39.435
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001260
  14:02:05 STEP 1261 H= 315:15 +CPU= 32.567
  14:02:39 STEP 1262 H= 315:30 +CPU= 32.164
  14:03:13 STEP 1263 H= 315:45 +CPU= 32.476
  14:03:48 STEP 1264 H= 316:00 +CPU= 33.128
  14:04:24 STEP 1265 H= 316:15 +CPU= 32.035
  14:05:00 STEP 1266 H= 316:30 +CPU= 32.288
  14:05:36 STEP 1267 H= 316:45 +CPU= 32.314
  14:06:13 STEP 1268 H= 317:00 +CPU= 32.308
  14:06:44 STEP 1269 H= 317:15 +CPU= 30.710
  14:07:20 STEP 1270 H= 317:30 +CPU= 33.468
  14:07:54 STEP 1271 H= 317:45 +CPU= 33.961
  14:08:36 STEP 1272 H= 318:00 +CPU= 42.042
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001272
  14:09:10 STEP 1273 H= 318:15 +CPU= 33.449
  14:09:42 STEP 1274 H= 318:30 +CPU= 32.317
  14:10:15 STEP 1275 H= 318:45 +CPU= 32.814
  14:10:48 STEP 1276 H= 319:00 +CPU= 33.464
  14:11:22 STEP 1277 H= 319:15 +CPU= 33.109
  14:11:54 STEP 1278 H= 319:30 +CPU= 32.852
  14:12:27 STEP 1279 H= 319:45 +CPU= 32.856
  14:12:58 STEP 1280 H= 320:00 +CPU= 30.482
  14:13:30 STEP 1281 H= 320:15 +CPU= 31.832
  14:14:02 STEP 1282 H= 320:30 +CPU= 31.922
  14:14:37 STEP 1283 H= 320:45 +CPU= 32.900
  14:15:15 STEP 1284 H= 321:00 +CPU= 38.240
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001284
  14:15:45 STEP 1285 H= 321:15 +CPU= 30.091
  14:16:16 STEP 1286 H= 321:30 +CPU= 30.665
  14:16:47 STEP 1287 H= 321:45 +CPU= 30.770
  14:17:18 STEP 1288 H= 322:00 +CPU= 31.196
  14:17:48 STEP 1289 H= 322:15 +CPU= 30.594
  14:18:19 STEP 1290 H= 322:30 +CPU= 30.892
  14:18:50 STEP 1291 H= 322:45 +CPU= 30.916
  14:19:22 STEP 1292 H= 323:00 +CPU= 31.404
  14:19:54 STEP 1293 H= 323:15 +CPU= 30.666
  14:20:28 STEP 1294 H= 323:30 +CPU= 31.200
  14:20:59 STEP 1295 H= 323:45 +CPU= 31.036
  14:21:47 STEP 1296 H= 324:00 +CPU= 42.467
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001296
  14:22:19 STEP 1297 H= 324:15 +CPU= 31.869
  14:22:51 STEP 1298 H= 324:30 +CPU= 32.486
  14:23:26 STEP 1299 H= 324:45 +CPU= 32.569
  14:23:59 STEP 1300 H= 325:00 +CPU= 32.934
  14:24:32 STEP 1301 H= 325:15 +CPU= 32.723
  14:25:04 STEP 1302 H= 325:30 +CPU= 32.017
  14:25:37 STEP 1303 H= 325:45 +CPU= 32.289
  14:26:10 STEP 1304 H= 326:00 +CPU= 32.920
  14:26:46 STEP 1305 H= 326:15 +CPU= 32.255
  14:27:22 STEP 1306 H= 326:30 +CPU= 32.234
  14:27:58 STEP 1307 H= 326:45 +CPU= 31.763
  14:28:40 STEP 1308 H= 327:00 +CPU= 39.479
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001308
  14:29:11 STEP 1309 H= 327:15 +CPU= 31.487
  14:29:43 STEP 1310 H= 327:30 +CPU= 31.521
  14:30:15 STEP 1311 H= 327:45 +CPU= 31.857
  14:30:47 STEP 1312 H= 328:00 +CPU= 32.303
  14:31:18 STEP 1313 H= 328:15 +CPU= 31.469
  14:31:50 STEP 1314 H= 328:30 +CPU= 31.206
  14:32:22 STEP 1315 H= 328:45 +CPU= 31.988
  14:32:58 STEP 1316 H= 329:00 +CPU= 32.975
  14:33:30 STEP 1317 H= 329:15 +CPU= 31.989
  14:34:01 STEP 1318 H= 329:30 +CPU= 31.409
  14:34:33 STEP 1319 H= 329:45 +CPU= 31.545
  14:35:11 STEP 1320 H= 330:00 +CPU= 38.466
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001320
  14:35:43 STEP 1321 H= 330:15 +CPU= 31.253
  14:36:14 STEP 1322 H= 330:30 +CPU= 31.441
  14:36:46 STEP 1323 H= 330:45 +CPU= 31.663
  14:37:18 STEP 1324 H= 331:00 +CPU= 32.530
  14:37:48 STEP 1325 H= 331:15 +CPU= 29.993
  14:38:20 STEP 1326 H= 331:30 +CPU= 31.677
  14:38:51 STEP 1327 H= 331:45 +CPU= 31.246
  14:39:23 STEP 1328 H= 332:00 +CPU= 32.055
  14:39:55 STEP 1329 H= 332:15 +CPU= 31.751
  14:40:27 STEP 1330 H= 332:30 +CPU= 31.702
  14:40:59 STEP 1331 H= 332:45 +CPU= 31.696
  14:41:36 STEP 1332 H= 333:00 +CPU= 37.246
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001332
  14:42:07 STEP 1333 H= 333:15 +CPU= 30.690
  14:42:36 STEP 1334 H= 333:30 +CPU= 29.235
  14:43:05 STEP 1335 H= 333:45 +CPU= 29.284
  14:43:35 STEP 1336 H= 334:00 +CPU= 29.691
  14:44:04 STEP 1337 H= 334:15 +CPU= 28.924
  14:44:33 STEP 1338 H= 334:30 +CPU= 29.085
  14:45:02 STEP 1339 H= 334:45 +CPU= 28.658
  14:45:31 STEP 1340 H= 335:00 +CPU= 29.203
  14:45:54 STEP 1341 H= 335:15 +CPU= 23.437
  14:46:23 STEP 1342 H= 335:30 +CPU= 28.658
  14:46:53 STEP 1343 H= 335:45 +CPU= 29.208
  14:47:36 STEP 1344 H= 336:00 +CPU= 38.603
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_13.zip
Uploading the intermediate file: upload_file_13.zip
  14:48:05 STEP 1345 H= 336:15 +CPU= 29.043
  14:48:34 STEP 1346 H= 336:30 +CPU= 28.832
  14:49:03 STEP 1347 H= 336:45 +CPU= 29.085
  14:49:32 STEP 1348 H= 337:00 +CPU= 29.105
  14:50:01 STEP 1349 H= 337:15 +CPU= 28.614
  14:50:30 STEP 1350 H= 337:30 +CPU= 29.264
  14:50:59 STEP 1351 H= 337:45 +CPU= 29.096
  14:51:29 STEP 1352 H= 338:00 +CPU= 29.734
  14:51:59 STEP 1353 H= 338:15 +CPU= 29.333
  14:52:28 STEP 1354 H= 338:30 +CPU= 29.065
  14:52:57 STEP 1355 H= 338:45 +CPU= 28.859
  14:53:33 STEP 1356 H= 339:00 +CPU= 36.091
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001356
  14:54:02 STEP 1357 H= 339:15 +CPU= 29.412
  14:54:32 STEP 1358 H= 339:30 +CPU= 29.430
  14:55:01 STEP 1359 H= 339:45 +CPU= 29.246
  14:55:30 STEP 1360 H= 340:00 +CPU= 29.532
  14:55:59 STEP 1361 H= 340:15 +CPU= 29.114
  14:56:29 STEP 1362 H= 340:30 +CPU= 29.212
  14:56:58 STEP 1363 H= 340:45 +CPU= 29.190
  14:57:28 STEP 1364 H= 341:00 +CPU= 29.958
  14:57:57 STEP 1365 H= 341:15 +CPU= 29.098
  14:58:26 STEP 1366 H= 341:30 +CPU= 29.087
  14:58:55 STEP 1367 H= 341:45 +CPU= 29.125
  14:59:31 STEP 1368 H= 342:00 +CPU= 35.950
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001368
  15:00:01 STEP 1369 H= 342:15 +CPU= 29.318
  15:00:31 STEP 1370 H= 342:30 +CPU= 29.275
  15:01:01 STEP 1371 H= 342:45 +CPU= 29.684
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1lz
(argv4) batchid: 1017
(argv5) wuid: 12283914
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_cf6d62c72129e21b60ce5c63ab3f0e53 to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1lz_2016092300_20_1017_12283914.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1lz_2016092300_20_1017_12283914.zip
ic_ancil_file: ic_ancil_12283914
ifsdata_file: ifsdata_12283914
climate_data_file: clim_data_12283914
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/3/ic_ancil_12283914.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12283914.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_83d9cdc2ed20b5df09ee0d699c108825 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12283914.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12283914.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12283914.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12283914.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12283914.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12283914.xml
-- Model is restarting --
Adjusting last_iter, 1371, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12283914.xml
last_cpu_time: 45994
upload_file_number: 14
last_iter: 1345
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918
The child process has been launched with process id: 14197
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKTRYOUT1:1:1:14197:14197] [20240608:151619:1717852579:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:16:48 STEP 1344 H= 336:00 +CPU= 26.186
  15:17:17 STEP 1345 H= 336:15 +CPU= 26.957
  15:17:44 STEP 1346 H= 336:30 +CPU= 27.217
  15:18:11 STEP 1347 H= 336:45 +CPU= 27.128
  15:18:39 STEP 1348 H= 337:00 +CPU= 27.849
  15:19:06 STEP 1349 H= 337:15 +CPU= 27.096
  15:19:33 STEP 1350 H= 337:30 +CPU= 27.037
  15:20:00 STEP 1351 H= 337:45 +CPU= 27.262
  15:20:28 STEP 1352 H= 338:00 +CPU= 28.001
  15:20:56 STEP 1353 H= 338:15 +CPU= 27.693
  15:21:24 STEP 1354 H= 338:30 +CPU= 27.979
  15:21:52 STEP 1355 H= 338:45 +CPU= 27.903
  15:22:26 STEP 1356 H= 339:00 +CPU= 34.380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001356
  15:22:54 STEP 1357 H= 339:15 +CPU= 27.845
  15:23:22 STEP 1358 H= 339:30 +CPU= 27.917
  15:23:50 STEP 1359 H= 339:45 +CPU= 27.902
  15:24:19 STEP 1360 H= 340:00 +CPU= 28.414
  15:24:46 STEP 1361 H= 340:15 +CPU= 27.705
  15:25:14 STEP 1362 H= 340:30 +CPU= 27.786
  15:25:42 STEP 1363 H= 340:45 +CPU= 27.651
  15:26:10 STEP 1364 H= 341:00 +CPU= 28.734
  15:26:39 STEP 1365 H= 341:15 +CPU= 28.119
  15:27:06 STEP 1366 H= 341:30 +CPU= 27.771
  15:27:34 STEP 1367 H= 341:45 +CPU= 27.771
  15:28:08 STEP 1368 H= 342:00 +CPU= 34.146
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001368
  15:28:36 STEP 1369 H= 342:15 +CPU= 27.663
  15:29:04 STEP 1370 H= 342:30 +CPU= 27.789
  15:29:32 STEP 1371 H= 342:45 +CPU= 27.737
  15:30:00 STEP 1372 H= 343:00 +CPU= 28.400
  15:30:28 STEP 1373 H= 343:15 +CPU= 27.795
  15:30:56 STEP 1374 H= 343:30 +CPU= 27.895
  15:31:24 STEP 1375 H= 343:45 +CPU= 27.725
  15:31:52 STEP 1376 H= 344:00 +CPU= 28.133
  15:32:20 STEP 1377 H= 344:15 +CPU= 27.793
  15:32:48 STEP 1378 H= 344:30 +CPU= 27.859
  15:33:19 STEP 1379 H= 344:45 +CPU= 28.335
  15:33:55 STEP 1380 H= 345:00 +CPU= 34.674
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001380
  15:34:23 STEP 1381 H= 345:15 +CPU= 28.148
  15:34:51 STEP 1382 H= 345:30 +CPU= 27.970
  15:35:19 STEP 1383 H= 345:45 +CPU= 27.872
  15:35:47 STEP 1384 H= 346:00 +CPU= 28.224
  15:36:15 STEP 1385 H= 346:15 +CPU= 27.788
  15:36:43 STEP 1386 H= 346:30 +CPU= 27.908
  15:37:11 STEP 1387 H= 346:45 +CPU= 28.294
  15:37:40 STEP 1388 H= 347:00 +CPU= 28.854
  15:38:09 STEP 1389 H= 347:15 +CPU= 28.359
  15:38:37 STEP 1390 H= 347:30 +CPU= 28.287
  15:39:05 STEP 1391 H= 347:45 +CPU= 28.118
  15:39:42 STEP 1392 H= 348:00 +CPU= 36.920
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001392
  15:40:10 STEP 1393 H= 348:15 +CPU= 28.378
  15:40:39 STEP 1394 H= 348:30 +CPU= 28.371
  15:41:07 STEP 1395 H= 348:45 +CPU= 28.156
  15:41:35 STEP 1396 H= 349:00 +CPU= 28.001
  15:42:04 STEP 1397 H= 349:15 +CPU= 29.279
  15:42:34 STEP 1398 H= 349:30 +CPU= 29.722
  15:43:04 STEP 1399 H= 349:45 +CPU= 29.737
  15:43:35 STEP 1400 H= 350:00 +CPU= 30.497
  15:44:04 STEP 1401 H= 350:15 +CPU= 29.646
  15:44:33 STEP 1402 H= 350:30 +CPU= 29.152
  15:45:03 STEP 1403 H= 350:45 +CPU= 29.221
  15:45:39 STEP 1404 H= 351:00 +CPU= 36.447
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001404
  15:46:08 STEP 1405 H= 351:15 +CPU= 29.292
  15:46:38 STEP 1406 H= 351:30 +CPU= 29.568
  15:47:08 STEP 1407 H= 351:45 +CPU= 29.852
  15:47:38 STEP 1408 H= 352:00 +CPU= 30.635
  15:48:08 STEP 1409 H= 352:15 +CPU= 29.676
  15:48:38 STEP 1410 H= 352:30 +CPU= 29.894
  15:49:08 STEP 1411 H= 352:45 +CPU= 29.794
  15:49:38 STEP 1412 H= 353:00 +CPU= 29.747
  15:50:06 STEP 1413 H= 353:15 +CPU= 28.316
  15:50:35 STEP 1414 H= 353:30 +CPU= 28.694
  15:51:04 STEP 1415 H= 353:45 +CPU= 28.821
  15:51:39 STEP 1416 H= 354:00 +CPU= 35.046
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001416
  15:52:07 STEP 1417 H= 354:15 +CPU= 28.420
  15:52:36 STEP 1418 H= 354:30 +CPU= 28.517
  15:53:04 STEP 1419 H= 354:45 +CPU= 28.417
  15:53:33 STEP 1420 H= 355:00 +CPU= 28.804
  15:54:01 STEP 1421 H= 355:15 +CPU= 28.264
  15:54:30 STEP 1422 H= 355:30 +CPU= 28.624
  15:54:58 STEP 1423 H= 355:45 +CPU= 28.371
  15:55:27 STEP 1424 H= 356:00 +CPU= 29.086
  15:55:56 STEP 1425 H= 356:15 +CPU= 28.820
  15:56:24 STEP 1426 H= 356:30 +CPU= 28.207
  15:56:53 STEP 1427 H= 356:45 +CPU= 28.313
  15:57:28 STEP 1428 H= 357:00 +CPU= 35.014
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001428
  15:57:57 STEP 1429 H= 357:15 +CPU= 28.735
  15:58:25 STEP 1430 H= 357:30 +CPU= 28.628
  15:58:54 STEP 1431 H= 357:45 +CPU= 28.553
  15:59:23 STEP 1432 H= 358:00 +CPU= 29.245
  15:59:52 STEP 1433 H= 358:15 +CPU= 28.636
  16:00:20 STEP 1434 H= 358:30 +CPU= 28.677
  16:00:49 STEP 1435 H= 358:45 +CPU= 28.531
  16:01:18 STEP 1436 H= 359:00 +CPU= 28.880
  16:01:50 STEP 1437 H= 359:15 +CPU= 29.543
  16:02:21 STEP 1438 H= 359:30 +CPU= 29.240
  16:02:50 STEP 1439 H= 359:45 +CPU= 29.421
  16:03:27 STEP 1440 H= 360:00 +CPU= 36.610
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.6660367943E-21      I      -0.4516675284E-04      I
ICloud fractionI      0.0000000000E+00      I       0.1407871734E+00      I
I  Relat. Hum. I      0.1183009669E-01      I       0.4106888803E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.2331562559E+04      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    307543652 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0122( 2.5%)
 16:03:27 STEP 1440 H= 360:00 +CPU= 36.610
 END CNT3
 NSTEP =  1440 CNT0   000000000  27.693  27.693  28.053   0.987               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         2860/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     53    0       46      1044068     2860 ************     5.0     0.9     4.1     1.5     9.5
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 2828596.9 2828596.9    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 2826235.9 2826235.9     99.92      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  105    4093.9    4093.9     15.20      0.00
   9 SPCM     - SPECTRAL COMP.                      96     790.5     790.5      2.68      0.00
  10 SCAN2M - PHYSICS                               97   14799.5   14799.5     50.75      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  8     707.6     707.6      0.20      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  5      46.6      46.6      0.01      0.00
  17 GRIDFPOS IN CNT4                                8      27.7      27.7      0.01      0.00
  27 RESTART FILES - WRITING                         2    2830.2    2830.2      0.20      0.00
  28 RESTART FILES - READING                         1    2026.1    2026.1      0.07      0.00
  29 SU4FPOS IN CNT4                                 8       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 8    5766.4    5766.4      1.63      0.00
  31 POSDDH IN STEPO                                 1       0.9       0.9      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     97    5900.1    5900.1     20.23      0.00
  39 SU0YOMB                                         1     213.4     213.4      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                     97      31.1      31.1      0.11      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               96     336.3     336.3      1.14      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               96      57.0      57.0      0.19      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  5       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      212     428.8     428.8      3.21      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       200     251.7     251.7      1.78      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        200      87.6      87.6      0.62      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       212     139.9     139.9      1.05      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      24.5      24.5      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            212      43.0      43.0      0.32      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            200      23.9      23.9      0.17      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            212      87.4      87.4      0.65      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            200      23.4      23.4      0.17      0.00
 400 GSTATS                                      12215       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 11311       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     2828.6 CPU TIME    2814.9 VECTOR TIME     2814.9


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      458.6
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/3/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/3/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12283914/ICMSHh7zg+001428
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
16:07:02 (14187): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0_r1019060918_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 14:41:12 1532094 22435493 oifs_43r3_bl_a1lz_2016092300_20_1017_12283914_0 1,295,100 48,808 0.0377


©2024 cpdn.org