climateprediction.net home page
Task 22312432

Task 22312432

Name oifs_43r3_bl_a45n_2016092300_15_991_12212174_0
Workunit 12212174
Created 13 Feb 2023, 21:55:33 UTC
Sent 13 Feb 2023, 22:00:37 UTC
Report deadline 14 Apr 2023, 22:00:37 UTC
Received 14 Feb 2023, 19:36:08 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1496882
Run time 18 hours 18 min 12 sec
CPU time 11 hours 18 min 14 sec
Validate state Valid
Credit 1,813.14
Device peak FLOPS 4.12 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 4,899.71 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,294.78 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
 +CPU= 27.602
  05:34:00 STEP 1138 H= 284:30 +CPU= 29.011
  05:34:44 STEP 1139 H= 284:45 +CPU= 27.256
  05:35:42 STEP 1140 H= 285:00 +CPU= 34.578
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001140
  05:36:34 STEP 1141 H= 285:15 +CPU= 28.041
  05:37:24 STEP 1142 H= 285:30 +CPU= 28.284
  05:38:09 STEP 1143 H= 285:45 +CPU= 27.768
  05:38:58 STEP 1144 H= 286:00 +CPU= 28.818
  05:39:53 STEP 1145 H= 286:15 +CPU= 28.756
  05:40:43 STEP 1146 H= 286:30 +CPU= 28.202
  05:41:33 STEP 1147 H= 286:45 +CPU= 28.411
  05:42:21 STEP 1148 H= 287:00 +CPU= 28.965
  05:43:18 STEP 1149 H= 287:15 +CPU= 28.300
  05:44:10 STEP 1150 H= 287:30 +CPU= 28.087
  05:45:14 STEP 1151 H= 287:45 +CPU= 28.795
  05:47:46 STEP 1152 H= 288:00 +CPU= 39.893
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a45n_2016092300_15_991_12212174_0_r1778441884_11.zip
Uploading the intermediate file: upload_file_11.zip
  05:48:50 STEP 1153 H= 288:15 +CPU= 29.702
  05:50:02 STEP 1154 H= 288:30 +CPU= 29.243
  05:51:01 STEP 1155 H= 288:45 +CPU= 28.573
  05:51:54 STEP 1156 H= 289:00 +CPU= 29.520
  05:53:16 STEP 1157 H= 289:15 +CPU= 29.853
  05:54:16 STEP 1158 H= 289:30 +CPU= 29.397
  05:55:08 STEP 1159 H= 289:45 +CPU= 29.435
  05:56:12 STEP 1160 H= 290:00 +CPU= 29.602
  05:56:55 STEP 1161 H= 290:15 +CPU= 27.378
  05:57:38 STEP 1162 H= 290:30 +CPU= 27.122
  05:58:09 STEP 1163 H= 290:45 +CPU= 26.268
  05:59:05 STEP 1164 H= 291:00 +CPU= 33.613
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001164
  05:59:55 STEP 1165 H= 291:15 +CPU= 27.532
  06:00:38 STEP 1166 H= 291:30 +CPU= 27.446
  06:01:23 STEP 1167 H= 291:45 +CPU= 27.568
  06:02:03 STEP 1168 H= 292:00 +CPU= 27.508
  06:02:36 STEP 1169 H= 292:15 +CPU= 26.865
  06:03:24 STEP 1170 H= 292:30 +CPU= 27.579
  06:04:04 STEP 1171 H= 292:45 +CPU= 27.810
  06:04:40 STEP 1172 H= 293:00 +CPU= 26.868
  06:05:15 STEP 1173 H= 293:15 +CPU= 26.709
  06:06:03 STEP 1174 H= 293:30 +CPU= 27.622
  06:06:31 STEP 1175 H= 293:45 +CPU= 25.948
  06:07:08 STEP 1176 H= 294:00 +CPU= 32.387
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001176
  06:07:58 STEP 1177 H= 294:15 +CPU= 28.444
  06:08:28 STEP 1178 H= 294:30 +CPU= 26.992
  06:08:58 STEP 1179 H= 294:45 +CPU= 26.377
  06:09:27 STEP 1180 H= 295:00 +CPU= 26.726
  06:09:57 STEP 1181 H= 295:15 +CPU= 26.414
  06:10:28 STEP 1182 H= 295:30 +CPU= 27.424
  06:11:06 STEP 1183 H= 295:45 +CPU= 27.000
  06:11:57 STEP 1184 H= 296:00 +CPU= 28.317
  06:12:43 STEP 1185 H= 296:15 +CPU= 27.579
  06:13:11 STEP 1186 H= 296:30 +CPU= 26.196
  06:13:38 STEP 1187 H= 296:45 +CPU= 26.525
  06:14:21 STEP 1188 H= 297:00 +CPU= 33.733
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001188
  06:14:50 STEP 1189 H= 297:15 +CPU= 27.196
  06:15:18 STEP 1190 H= 297:30 +CPU= 26.394
  06:15:48 STEP 1191 H= 297:45 +CPU= 27.102
  06:16:19 STEP 1192 H= 298:00 +CPU= 27.550
  06:17:05 STEP 1193 H= 298:15 +CPU= 28.485
  06:17:58 STEP 1194 H= 298:30 +CPU= 28.580
  06:18:52 STEP 1195 H= 298:45 +CPU= 28.669
  06:19:20 STEP 1196 H= 299:00 +CPU= 26.392
  06:20:08 STEP 1197 H= 299:15 +CPU= 27.934
  06:21:01 STEP 1198 H= 299:30 +CPU= 28.058
  06:21:52 STEP 1199 H= 299:45 +CPU= 29.073
  06:23:26 STEP 1200 H= 300:00 +CPU= 37.654
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001200
  06:24:32 STEP 1201 H= 300:15 +CPU= 28.465
  06:25:28 STEP 1202 H= 300:30 +CPU= 29.059
  06:26:10 STEP 1203 H= 300:45 +CPU= 27.622
  06:26:57 STEP 1204 H= 301:00 +CPU= 29.238
  06:27:37 STEP 1205 H= 301:15 +CPU= 28.086
  06:28:11 STEP 1206 H= 301:30 +CPU= 26.569
  06:28:58 STEP 1207 H= 301:45 +CPU= 27.980
  06:29:56 STEP 1208 H= 302:00 +CPU= 28.295
  06:30:52 STEP 1209 H= 302:15 +CPU= 29.096
  06:32:27 STEP 1210 H= 302:30 +CPU= 29.310
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

DDH--------------------------------------------------------------------------------------------------I
I      I Balance        II     -0.00040 I              I              I              I               I
I ADIAB- of which       II              I              I              I              I               I
I      I Hor. Flux of PEII      0.00000 I              I              I              I               I
I      I Vert.Flux of PEII      0.00000 I              I              I              I               I
I      I Work Pres.G.F  II              I      0.61043 I              I              I               I
I      I Conv. Ad.-Int. II     -0.61083 I              I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I DYN- Horizont. Flux   II      0.12001 I      0.06307 I      1.84514 I      0.01698 I     -0.03521  I
I    I Vertical Flux    II      0.00000 I      0.00000 I      0.00000 I      0.00000 I      0.00000  I
DDH--------------------------------------------------------------------------------------------------I
  
DDH----------------------------------------------------------------------------------------------------------------------------------
  I   - Mass balance    (10**(-6)  KG/M**2) 
  
  I INITIAL      I HORIZ. FLUX  I DYN. FLUX    I DYN. FLUX    I PHYS. FLUX   I PHYS. FLUX   I CURRENT      I I I RESIDUAL     I
  I MASS         I              I TOA          I SURFACE      I TOA          I SURFACE      I MASS         I I I              I
  ---------------------------------------------------------------------------------------------------------- I ----------------
  -              -              -              -              -              -              -              - I -              -
  - 10206523039. I           2. .           0. .           0. .           0. .           0. I 10206869852. - I -      346810. -
  -              -              -              -              -              -              -              - I -              -
  ---------------------------------------------------------------------------------------------------------- I ----------------
 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  300.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.5931484389E-21      I       0.9259516900E-05      I
ICloud fractionI      0.0000000000E+00      I       0.7469880285E-01      I
I  Relat. Hum. I      0.2541127238E+00      I       0.4327112455E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1258100019E+04      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1200 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001200 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001200
 MPI-TASK:   1 -    312639233 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS781598.1 Mbytes/s, TOTAL TIME=    0.1148( 0.3%)
 06:23:26 STEP 1200 H= 300:00 +CPU= 37.654
 NSTEP =  1201 STEPO   0AAA00AAA
 06:24:32 STEP 1201 H= 300:15 +CPU= 28.465
 NSTEP =  1202 STEPO   0AAA00AAA
 06:25:28 STEP 1202 H= 300:30 +CPU= 29.059
 NSTEP =  1203 STEPO   0AAA00AAA
 06:26:10 STEP 1203 H= 300:45 +CPU= 27.622
 NSTEP =  1204 STEPO   0AAA00AAA
 06:26:57 STEP 1204 H= 301:00 +CPU= 29.238
ISTEP=    1205 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1205 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1205 STEPO   0AAA00AAA
 06:27:37 STEP 1205 H= 301:15 +CPU= 28.086
 NSTEP =  1206 STEPO   0AAA00AAA
 06:28:11 STEP 1206 H= 301:30 +CPU= 26.569
 NSTEP =  1207 STEPO   0AAA00AAA
 06:28:58 STEP 1207 H= 301:45 +CPU= 27.980
 NSTEP =  1208 STEPO   0AAA00AAA
 06:29:56 STEP 1208 H= 302:00 +CPU= 28.295
 NSTEP =  1209 STEPO   0AAA00AAA
 06:30:52 STEP 1209 H= 302:15 +CPU= 29.096
ISTEP=    1210 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1210 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1210 STEPO   0AAA00AAA
 06:32:27 STEP 1210 H= 302:30 +CPU= 29.310
 NSTEP =  1211 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 06:26:57 0AAA00AAA STEPO     1205    29.239   29.239   47.094    569:39    976:04 0.84828233509213E-05       2GB       0MB
 06:27:37 0AAA00AAA STEPO     1206    28.087   28.087   40.348    570:07    976:45 0.84828233509213E-05       2GB       0MB
 06:28:11 0AAA00AAA STEPO     1207    26.571   26.571   33.620    570:34    977:18 0.84828233509213E-05       2GB       0MB
 06:28:58 0AAA00AAA STEPO     1208    27.981   27.981   46.716    571:02    978:05 0.84828233509213E-05       2GB       0MB
 06:29:56 0AAA00AAA STEPO     1209    28.297   28.297   57.966    571:30    979:03 0.84828233509213E-05       2GB       0MB
 06:30:52 0AAA00AAA STEPO     1210    29.097   29.097   56.454    571:59    979:59 0.84828233509213E-05       2GB       0MB
 06:32:27 0AAA00AAA STEPO     1211    29.311   29.311   94.457    572:29    981:34 0.84828233509213E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1200",
 CTIME   = "00121200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1188, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc/slots/1/progress_file_12212174.xml

<running_values>
  <last_cpu_time>34356.840000</last_cpu_time>
  <upload_file_number>12</upload_file_number>
  <last_iter>1211</last_iter>
  <last_upload>1036800</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a45n
(argv4) batchid: 991
(argv5) wuid: 12212174
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23e4ed9e8ec78c7e5d4ba04861b2dea7 to: /var/lib/boinc/slots/1/oifs_43r3_bl_a45n_2016092300_15_991_12212174.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_a45n_2016092300_15_991_12212174.zip
ic_ancil_file: ic_ancil_12212174
ifsdata_file: ifsdata_12212174
climate_data_file: clim_data_12212174
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/1/ic_ancil_12212174.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12212174.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_97b8815d44327c813d3ad10278b6c0d6 to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12212174.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12212174.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12212174.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12212174.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12212174.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12212174.xml
-- Model is restarting --
Adjusting last_iter, 1211, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12212174.xml
last_cpu_time: 34356
upload_file_number: 12
last_iter: 1201
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a45n_2016092300_15_991_12212174_0_r1778441884
The child process has been launched with process id: 184571
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:184571:184571] [20230214:073328:1676388808:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:34:45 STEP 1200 H= 300:00 +CPU= 29.139
  07:35:31 STEP 1201 H= 300:15 +CPU= 27.446
  07:36:07 STEP 1202 H= 300:30 +CPU= 26.344
  07:36:44 STEP 1203 H= 300:45 +CPU= 26.996
  07:37:50 STEP 1204 H= 301:00 +CPU= 28.797
  07:38:35 STEP 1205 H= 301:15 +CPU= 27.790
  07:39:27 STEP 1206 H= 301:30 +CPU= 29.109
  07:40:10 STEP 1207 H= 301:45 +CPU= 27.745
  07:41:05 STEP 1208 H= 302:00 +CPU= 30.157
  07:42:04 STEP 1209 H= 302:15 +CPU= 30.213
  07:42:57 STEP 1210 H= 302:30 +CPU= 29.163
  07:44:02 STEP 1211 H= 302:45 +CPU= 29.682
  07:45:38 STEP 1212 H= 303:00 +CPU= 36.598
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001212
  07:46:44 STEP 1213 H= 303:15 +CPU= 30.695
  07:48:01 STEP 1214 H= 303:30 +CPU= 28.638
  07:49:05 STEP 1215 H= 303:45 +CPU= 30.112
  07:49:53 STEP 1216 H= 304:00 +CPU= 28.983
  07:50:48 STEP 1217 H= 304:15 +CPU= 29.321
  07:51:33 STEP 1218 H= 304:30 +CPU= 28.268
  07:52:21 STEP 1219 H= 304:45 +CPU= 28.419
  07:53:11 STEP 1220 H= 305:00 +CPU= 29.557
  07:53:57 STEP 1221 H= 305:15 +CPU= 28.822
  07:54:44 STEP 1222 H= 305:30 +CPU= 28.471
  07:55:23 STEP 1223 H= 305:45 +CPU= 27.480
  07:56:22 STEP 1224 H= 306:00 +CPU= 34.547
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001224
  07:57:30 STEP 1225 H= 306:15 +CPU= 29.933
  07:58:20 STEP 1226 H= 306:30 +CPU= 28.288
  07:59:15 STEP 1227 H= 306:45 +CPU= 26.884
  08:00:00 STEP 1228 H= 307:00 +CPU= 27.995
  08:00:34 STEP 1229 H= 307:15 +CPU= 25.830
  08:01:10 STEP 1230 H= 307:30 +CPU= 26.909
  08:02:00 STEP 1231 H= 307:45 +CPU= 27.622
  08:02:41 STEP 1232 H= 308:00 +CPU= 27.370
  08:03:25 STEP 1233 H= 308:15 +CPU= 28.111
  08:03:58 STEP 1234 H= 308:30 +CPU= 26.303
  08:04:35 STEP 1235 H= 308:45 +CPU= 26.791
  08:05:31 STEP 1236 H= 309:00 +CPU= 34.005
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001236
  08:06:14 STEP 1237 H= 309:15 +CPU= 27.801
  08:06:55 STEP 1238 H= 309:30 +CPU= 27.632
  08:07:42 STEP 1239 H= 309:45 +CPU= 27.971
  08:08:48 STEP 1240 H= 310:00 +CPU= 29.714
  08:09:36 STEP 1241 H= 310:15 +CPU= 28.035
  08:10:15 STEP 1242 H= 310:30 +CPU= 27.219
  08:11:02 STEP 1243 H= 310:45 +CPU= 27.661
  08:11:36 STEP 1244 H= 311:00 +CPU= 26.031
  08:12:00 STEP 1245 H= 311:15 +CPU= 22.757
  08:12:23 STEP 1246 H= 311:30 +CPU= 22.944
  08:12:47 STEP 1247 H= 311:45 +CPU= 23.235
  08:13:25 STEP 1248 H= 312:00 +CPU= 30.194
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a45n_2016092300_15_991_12212174_0_r1778441884_12.zip
Uploading the intermediate file: upload_file_12.zip
  08:13:48 STEP 1249 H= 312:15 +CPU= 23.334
  08:14:12 STEP 1250 H= 312:30 +CPU= 22.812
  08:14:34 STEP 1251 H= 312:45 +CPU= 22.540
  08:15:01 STEP 1252 H= 313:00 +CPU= 26.031
  08:15:34 STEP 1253 H= 313:15 +CPU= 31.916
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a45n
(argv4) batchid: 991
(argv5) wuid: 12212174
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23e4ed9e8ec78c7e5d4ba04861b2dea7 to: /var/lib/boinc/slots/1/oifs_43r3_bl_a45n_2016092300_15_991_12212174.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_a45n_2016092300_15_991_12212174.zip
ic_ancil_file: ic_ancil_12212174
ifsdata_file: ifsdata_12212174
climate_data_file: clim_data_12212174
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/1/ic_ancil_12212174.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12212174.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_97b8815d44327c813d3ad10278b6c0d6 to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12212174.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12212174.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12212174.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12212174.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12212174.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12212174.xml
-- Model is restarting --
Adjusting last_iter, 1254, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12212174.xml
last_cpu_time: 35887
upload_file_number: 13
last_iter: 1249
last_upload: 1123200
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a45n_2016092300_15_991_12212174_0_r1778441884
The child process has been launched with process id: 185898
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:serv.home.saldino.net:1:1:185898:185898] [20230214:094705:1676396825:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:47:34 STEP 1248 H= 312:00 +CPU= 25.234
  09:47:59 STEP 1249 H= 312:15 +CPU= 24.878
  09:48:24 STEP 1250 H= 312:30 +CPU= 24.736
  09:48:49 STEP 1251 H= 312:45 +CPU= 24.700
  09:49:14 STEP 1252 H= 313:00 +CPU= 24.258
  09:49:39 STEP 1253 H= 313:15 +CPU= 24.398
  09:50:04 STEP 1254 H= 313:30 +CPU= 24.616
  09:50:28 STEP 1255 H= 313:45 +CPU= 23.558
  09:50:53 STEP 1256 H= 314:00 +CPU= 24.879
  09:51:18 STEP 1257 H= 314:15 +CPU= 24.402
  09:51:44 STEP 1258 H= 314:30 +CPU= 24.590
  09:52:08 STEP 1259 H= 314:45 +CPU= 23.757
  09:52:39 STEP 1260 H= 315:00 +CPU= 30.266
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001260
  09:53:03 STEP 1261 H= 315:15 +CPU= 24.104
  09:53:28 STEP 1262 H= 315:30 +CPU= 23.741
  09:53:52 STEP 1263 H= 315:45 +CPU= 23.763
  09:54:17 STEP 1264 H= 316:00 +CPU= 24.237
  09:54:41 STEP 1265 H= 316:15 +CPU= 23.611
  09:55:06 STEP 1266 H= 316:30 +CPU= 24.155
  09:55:31 STEP 1267 H= 316:45 +CPU= 24.736
  09:55:56 STEP 1268 H= 317:00 +CPU= 25.245
  09:56:21 STEP 1269 H= 317:15 +CPU= 24.110
  09:56:46 STEP 1270 H= 317:30 +CPU= 24.256
  09:57:11 STEP 1271 H= 317:45 +CPU= 24.565
  09:57:41 STEP 1272 H= 318:00 +CPU= 29.733
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001272
  09:58:07 STEP 1273 H= 318:15 +CPU= 25.078
  09:58:31 STEP 1274 H= 318:30 +CPU= 24.010
  09:58:57 STEP 1275 H= 318:45 +CPU= 24.924
  09:59:23 STEP 1276 H= 319:00 +CPU= 25.177
  09:59:47 STEP 1277 H= 319:15 +CPU= 24.079
  10:00:13 STEP 1278 H= 319:30 +CPU= 25.309
  10:00:37 STEP 1279 H= 319:45 +CPU= 23.843
  10:01:03 STEP 1280 H= 320:00 +CPU= 25.366
  10:01:29 STEP 1281 H= 320:15 +CPU= 24.678
  10:01:53 STEP 1282 H= 320:30 +CPU= 24.439
  10:02:19 STEP 1283 H= 320:45 +CPU= 24.626
  10:02:49 STEP 1284 H= 321:00 +CPU= 30.042
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001284
  10:03:14 STEP 1285 H= 321:15 +CPU= 24.182
  10:03:39 STEP 1286 H= 321:30 +CPU= 24.542
  10:04:04 STEP 1287 H= 321:45 +CPU= 24.443
  10:04:29 STEP 1288 H= 322:00 +CPU= 24.224
  10:04:53 STEP 1289 H= 322:15 +CPU= 24.367
  10:05:19 STEP 1290 H= 322:30 +CPU= 24.698
  10:05:43 STEP 1291 H= 322:45 +CPU= 24.223
  10:06:09 STEP 1292 H= 323:00 +CPU= 25.557
  10:06:35 STEP 1293 H= 323:15 +CPU= 24.583
  10:06:59 STEP 1294 H= 323:30 +CPU= 24.076
  10:07:24 STEP 1295 H= 323:45 +CPU= 24.087
  10:07:57 STEP 1296 H= 324:00 +CPU= 32.332
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001296
  10:08:21 STEP 1297 H= 324:15 +CPU= 24.126
  10:08:46 STEP 1298 H= 324:30 +CPU= 23.938
  10:09:10 STEP 1299 H= 324:45 +CPU= 24.127
  10:09:35 STEP 1300 H= 325:00 +CPU= 24.553
  10:10:00 STEP 1301 H= 325:15 +CPU= 24.518
  10:10:25 STEP 1302 H= 325:30 +CPU= 24.524
  10:10:50 STEP 1303 H= 325:45 +CPU= 24.158
  10:11:17 STEP 1304 H= 326:00 +CPU= 26.003
  10:11:41 STEP 1305 H= 326:15 +CPU= 23.583
  10:12:06 STEP 1306 H= 326:30 +CPU= 24.753
  10:12:32 STEP 1307 H= 326:45 +CPU= 25.639
  10:13:02 STEP 1308 H= 327:00 +CPU= 29.863
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001308
  10:13:27 STEP 1309 H= 327:15 +CPU= 24.660
  10:13:52 STEP 1310 H= 327:30 +CPU= 24.492
  10:14:18 STEP 1311 H= 327:45 +CPU= 24.697
  10:14:42 STEP 1312 H= 328:00 +CPU= 24.220
  10:15:06 STEP 1313 H= 328:15 +CPU= 23.556
  10:15:31 STEP 1314 H= 328:30 +CPU= 24.136
  10:15:57 STEP 1315 H= 328:45 +CPU= 25.070
  10:16:22 STEP 1316 H= 329:00 +CPU= 24.930
  10:16:46 STEP 1317 H= 329:15 +CPU= 23.515
  10:17:11 STEP 1318 H= 329:30 +CPU= 24.308
  10:17:37 STEP 1319 H= 329:45 +CPU= 25.291
  10:18:07 STEP 1320 H= 330:00 +CPU= 29.895
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001320
  10:18:32 STEP 1321 H= 330:15 +CPU= 24.120
  10:18:57 STEP 1322 H= 330:30 +CPU= 24.194
  10:19:21 STEP 1323 H= 330:45 +CPU= 23.733
  10:19:46 STEP 1324 H= 331:00 +CPU= 24.442
  10:20:10 STEP 1325 H= 331:15 +CPU= 23.957
  10:20:36 STEP 1326 H= 331:30 +CPU= 25.026
  10:21:01 STEP 1327 H= 331:45 +CPU= 24.287
  10:21:26 STEP 1328 H= 332:00 +CPU= 24.596
  10:21:51 STEP 1329 H= 332:15 +CPU= 24.408
  10:22:16 STEP 1330 H= 332:30 +CPU= 25.160
  10:22:41 STEP 1331 H= 332:45 +CPU= 23.619
  10:23:11 STEP 1332 H= 333:00 +CPU= 29.990
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001332
  10:23:36 STEP 1333 H= 333:15 +CPU= 24.147
  10:24:01 STEP 1334 H= 333:30 +CPU= 24.707
  10:24:26 STEP 1335 H= 333:45 +CPU= 24.166
  10:24:50 STEP 1336 H= 334:00 +CPU= 24.265
  10:25:16 STEP 1337 H= 334:15 +CPU= 24.671
  10:25:40 STEP 1338 H= 334:30 +CPU= 24.156
  10:26:06 STEP 1339 H= 334:45 +CPU= 24.796
  10:26:30 STEP 1340 H= 335:00 +CPU= 24.396
  10:26:55 STEP 1341 H= 335:15 +CPU= 24.008
  10:27:20 STEP 1342 H= 335:30 +CPU= 24.547
  10:27:44 STEP 1343 H= 335:45 +CPU= 23.764
  10:28:18 STEP 1344 H= 336:00 +CPU= 32.990
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a45n_2016092300_15_991_12212174_0_r1778441884_13.zip
Uploading the intermediate file: upload_file_13.zip
  10:28:43 STEP 1345 H= 336:15 +CPU= 24.418
  10:29:07 STEP 1346 H= 336:30 +CPU= 23.554
  10:29:32 STEP 1347 H= 336:45 +CPU= 24.161
  10:29:58 STEP 1348 H= 337:00 +CPU= 26.017
  10:30:23 STEP 1349 H= 337:15 +CPU= 24.298
  10:30:48 STEP 1350 H= 337:30 +CPU= 24.277
  10:31:13 STEP 1351 H= 337:45 +CPU= 25.005
  10:31:39 STEP 1352 H= 338:00 +CPU= 24.556
  10:32:03 STEP 1353 H= 338:15 +CPU= 24.020
  10:32:27 STEP 1354 H= 338:30 +CPU= 23.618
  10:32:53 STEP 1355 H= 338:45 +CPU= 24.748
  10:33:24 STEP 1356 H= 339:00 +CPU= 31.009
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001356
  10:33:49 STEP 1357 H= 339:15 +CPU= 24.655
  10:34:14 STEP 1358 H= 339:30 +CPU= 24.232
  10:34:39 STEP 1359 H= 339:45 +CPU= 24.434
  10:35:04 STEP 1360 H= 340:00 +CPU= 24.397
  10:35:28 STEP 1361 H= 340:15 +CPU= 23.530
  10:35:53 STEP 1362 H= 340:30 +CPU= 24.908
  10:36:19 STEP 1363 H= 340:45 +CPU= 24.783
  10:36:44 STEP 1364 H= 341:00 +CPU= 24.663
  10:37:10 STEP 1365 H= 341:15 +CPU= 25.580
  10:37:35 STEP 1366 H= 341:30 +CPU= 24.726
  10:38:00 STEP 1367 H= 341:45 +CPU= 24.309
  10:38:31 STEP 1368 H= 342:00 +CPU= 30.461
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001368
  10:38:56 STEP 1369 H= 342:15 +CPU= 24.650
  10:39:20 STEP 1370 H= 342:30 +CPU= 24.009
  10:39:44 STEP 1371 H= 342:45 +CPU= 23.692
  10:40:10 STEP 1372 H= 343:00 +CPU= 25.351
  10:40:37 STEP 1373 H= 343:15 +CPU= 26.362
  10:41:01 STEP 1374 H= 343:30 +CPU= 23.970
  10:41:26 STEP 1375 H= 343:45 +CPU= 23.800
  10:41:51 STEP 1376 H= 344:00 +CPU= 24.501
  10:42:15 STEP 1377 H= 344:15 +CPU= 23.883
  10:42:40 STEP 1378 H= 344:30 +CPU= 24.886
  10:43:04 STEP 1379 H= 344:45 +CPU= 23.505
  10:43:35 STEP 1380 H= 345:00 +CPU= 29.808
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001380
  10:43:59 STEP 1381 H= 345:15 +CPU= 24.362
  10:44:24 STEP 1382 H= 345:30 +CPU= 24.457
  10:44:50 STEP 1383 H= 345:45 +CPU= 24.899
  10:45:16 STEP 1384 H= 346:00 +CPU= 25.177
  10:45:40 STEP 1385 H= 346:15 +CPU= 23.982
  10:46:04 STEP 1386 H= 346:30 +CPU= 23.681
  10:46:29 STEP 1387 H= 346:45 +CPU= 24.145
  10:46:54 STEP 1388 H= 347:00 +CPU= 24.632
  10:47:18 STEP 1389 H= 347:15 +CPU= 23.925
  10:47:43 STEP 1390 H= 347:30 +CPU= 24.120
  10:48:08 STEP 1391 H= 347:45 +CPU= 24.060
  10:48:40 STEP 1392 H= 348:00 +CPU= 31.718
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001392
  10:49:05 STEP 1393 H= 348:15 +CPU= 24.202
  10:49:30 STEP 1394 H= 348:30 +CPU= 24.523
  10:49:54 STEP 1395 H= 348:45 +CPU= 24.215
  10:50:19 STEP 1396 H= 349:00 +CPU= 24.618
  10:50:45 STEP 1397 H= 349:15 +CPU= 25.446
  10:51:10 STEP 1398 H= 349:30 +CPU= 24.594
  10:51:35 STEP 1399 H= 349:45 +CPU= 24.368
  10:52:01 STEP 1400 H= 350:00 +CPU= 24.971
  10:52:26 STEP 1401 H= 350:15 +CPU= 24.515
  10:52:50 STEP 1402 H= 350:30 +CPU= 24.215
  10:53:15 STEP 1403 H= 350:45 +CPU= 24.230
  10:53:45 STEP 1404 H= 351:00 +CPU= 29.526
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001404
  10:54:09 STEP 1405 H= 351:15 +CPU= 23.479
  10:54:34 STEP 1406 H= 351:30 +CPU= 24.564
  10:54:59 STEP 1407 H= 351:45 +CPU= 24.001
  10:55:24 STEP 1408 H= 352:00 +CPU= 24.454
  10:55:49 STEP 1409 H= 352:15 +CPU= 24.371
  10:56:14 STEP 1410 H= 352:30 +CPU= 24.418
  10:56:38 STEP 1411 H= 352:45 +CPU= 24.167
  10:57:04 STEP 1412 H= 353:00 +CPU= 25.094
  10:57:28 STEP 1413 H= 353:15 +CPU= 23.426
  10:57:52 STEP 1414 H= 353:30 +CPU= 23.806
  10:58:17 STEP 1415 H= 353:45 +CPU= 24.390
  10:58:47 STEP 1416 H= 354:00 +CPU= 29.754
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001416
  10:59:12 STEP 1417 H= 354:15 +CPU= 23.868
  10:59:36 STEP 1418 H= 354:30 +CPU= 23.859
  11:00:01 STEP 1419 H= 354:45 +CPU= 24.552
  11:00:26 STEP 1420 H= 355:00 +CPU= 24.484
  11:00:50 STEP 1421 H= 355:15 +CPU= 23.939
  11:01:15 STEP 1422 H= 355:30 +CPU= 23.986
  11:01:40 STEP 1423 H= 355:45 +CPU= 24.171
  11:02:05 STEP 1424 H= 356:00 +CPU= 24.491
  11:02:29 STEP 1425 H= 356:15 +CPU= 24.140
  11:02:54 STEP 1426 H= 356:30 +CPU= 24.421
  11:03:20 STEP 1427 H= 356:45 +CPU= 25.089
  11:03:51 STEP 1428 H= 357:00 +CPU= 30.329
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001428
  11:04:15 STEP 1429 H= 357:15 +CPU= 23.722
  11:04:40 STEP 1430 H= 357:30 +CPU= 24.241
  11:05:04 STEP 1431 H= 357:45 +CPU= 24.300
  11:05:29 STEP 1432 H= 358:00 +CPU= 24.568
  11:05:54 STEP 1433 H= 358:15 +CPU= 24.142
  11:06:18 STEP 1434 H= 358:30 +CPU= 23.710
  11:06:42 STEP 1435 H= 358:45 +CPU= 23.604
  11:07:07 STEP 1436 H= 359:00 +CPU= 24.525
  11:07:32 STEP 1437 H= 359:15 +CPU= 23.671
  11:07:56 STEP 1438 H= 359:30 +CPU= 23.882
  11:08:20 STEP 1439 H= 359:45 +CPU= 23.594
  11:08:51 STEP 1440 H= 360:00 +CPU= 30.489
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.5931484389E-21      I       0.7340379378E-05      I
ICloud fractionI      0.0000000000E+00      I       0.8902148245E-01      I
I  Relat. Hum. I      0.2541127238E+00      I       0.4643229174E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1451055813E+04      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    320044017 BYTES IN      7 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0074( 2.7%)
 11:08:51 STEP 1440 H= 360:00 +CPU= 30.489
 END CNT3
 NSTEP =  1440 CNT0   000000000  23.783  23.783  24.228   0.982               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN         5672/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    103    0       34      1044068     5672 ************     7.4     1.2     6.2     1.8    14.1
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 4905897.8 4905897.8    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 4903132.3 4903132.3     99.94      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  209    3357.4    3357.4     14.30      0.00
   9 SPCM     - SPECTRAL COMP.                     192     686.2     686.2      2.69      0.00
  10 SCAN2M - PHYSICS                              193   13995.9   13995.9     55.06      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 16     519.6     519.6      0.17      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  9      28.7      28.7      0.01      0.00
  17 GRIDFPOS IN CNT4                               16      22.7      22.7      0.01      0.00
  27 RESTART FILES - WRITING                         4    2078.4    2078.4      0.17      0.00
  28 RESTART FILES - READING                         1    2404.3    2404.3      0.05      0.00
  29 SU4FPOS IN CNT4                                16       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                16    4824.5    4824.5      1.57      0.00
  31 POSDDH IN STEPO                                 1       5.5       5.5      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    193    4649.5    4649.5     18.29      0.00
  39 SU0YOMB                                         1     235.7     235.7      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    193      23.4      23.4      0.09      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              192     273.9     273.9      1.07      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              192      28.1      28.1      0.11      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  9       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      420     311.8     311.8      2.67      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       400     191.6     191.6      1.56      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        400      66.4      66.4      0.54      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       420     119.3     119.3      1.02      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      37.2      37.2      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            420      21.0      21.0      0.18      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            400      12.7      12.7      0.10      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            420      65.3      65.3      0.56      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            400      18.2      18.2      0.15      0.00
 400 GSTATS                                      23227       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 21563       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     4905.9 CPU TIME    4807.9 VECTOR TIME     4807.9


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      264.3
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/1/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/1/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12212174/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a45n_2016092300_15_991_12212174_0_r1778441884_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
11:12:25 (185889): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:22 1496882 22312432 oifs_43r3_bl_a45n_2016092300_15_991_12212174_0 1,295,100 40,694 0.0314
09 May 2023 16:51:09 1496882 22312432 oifs_43r3_bl_a45n_2016092300_15_991_12212174_0 864,000 27,247 0.0315


©2024 cpdn.org