climateprediction.net home page
Task 22311962

Task 22311962

Name oifs_43r3_bl_a3sw_2016092300_15_991_12211715_0
Workunit 12211715
Created 13 Feb 2023, 21:04:48 UTC
Sent 13 Feb 2023, 21:09:12 UTC
Report deadline 14 Apr 2023, 21:09:12 UTC
Received 19 Feb 2023, 10:19:43 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1537658
Run time 1 days 9 hours 52 min 58 sec
CPU time 21 hours 57 min 17 sec
Validate state Invalid
Credit 1,813.14
Device peak FLOPS 2.62 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 4,893.59 MB
Peak swap size 5,930.79 MB
Peak disk usage 2,434.06 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<stderr_txt>
ler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:1593:1593] [20230218:203436:1676752476:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:1593:1593] [20230218:203436:1676752476:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:1593:1593] [20230218:203436:1676752476:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:1593:1593] [20230218:203436:1676752476:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:1593:1593] [20230218:203436:1676752476:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:49:25 STEP 1056 H= 264:00 +CPU= 56.689
  21:23:26 STEP 1057 H= 264:15 +CPU= 68.444
  21:31:32 STEP 1058 H= 264:30 +CPU= 61.494
  21:33:31 STEP 1059 H= 264:45 +CPU= 50.809
  21:34:26 STEP 1060 H= 265:00 +CPU= 49.124
  21:35:13 STEP 1061 H= 265:15 +CPU= 45.051
  21:36:01 STEP 1062 H= 265:30 +CPU= 47.261
  21:36:57 STEP 1063 H= 265:45 +CPU= 50.629
  21:37:52 STEP 1064 H= 266:00 +CPU= 51.060
  21:38:42 STEP 1065 H= 266:15 +CPU= 49.161
  21:39:31 STEP 1066 H= 266:30 +CPU= 47.631
  21:40:18 STEP 1067 H= 266:45 +CPU= 46.766
  21:41:20 STEP 1068 H= 267:00 +CPU= 56.790
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001068
  21:42:06 STEP 1069 H= 267:15 +CPU= 45.417
  21:42:53 STEP 1070 H= 267:30 +CPU= 46.794
  21:43:39 STEP 1071 H= 267:45 +CPU= 46.461
  21:44:26 STEP 1072 H= 268:00 +CPU= 47.067
  21:45:13 STEP 1073 H= 268:15 +CPU= 46.582
  21:45:59 STEP 1074 H= 268:30 +CPU= 45.326
  21:46:44 STEP 1075 H= 268:45 +CPU= 45.138
  21:47:30 STEP 1076 H= 269:00 +CPU= 46.290
  21:48:17 STEP 1077 H= 269:15 +CPU= 45.552
  21:49:03 STEP 1078 H= 269:30 +CPU= 45.053
  21:49:48 STEP 1079 H= 269:45 +CPU= 45.581
  21:50:45 STEP 1080 H= 270:00 +CPU= 56.386
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001080
  21:51:31 STEP 1081 H= 270:15 +CPU= 45.096
  21:52:16 STEP 1082 H= 270:30 +CPU= 45.029
  21:52:59 STEP 1083 H= 270:45 +CPU= 42.785
  21:53:42 STEP 1084 H= 271:00 +CPU= 43.439
  21:54:25 STEP 1085 H= 271:15 +CPU= 42.612
  21:55:08 STEP 1086 H= 271:30 +CPU= 42.612
  21:55:50 STEP 1087 H= 271:45 +CPU= 42.675
  21:56:34 STEP 1088 H= 272:00 +CPU= 43.408
  21:57:16 STEP 1089 H= 272:15 +CPU= 42.149
  21:57:58 STEP 1090 H= 272:30 +CPU= 42.430
  21:58:41 STEP 1091 H= 272:45 +CPU= 42.328
  21:59:33 STEP 1092 H= 273:00 +CPU= 52.181
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001092
  22:00:15 STEP 1093 H= 273:15 +CPU= 42.079
  22:00:58 STEP 1094 H= 273:30 +CPU= 42.412
  22:01:40 STEP 1095 H= 273:45 +CPU= 42.122
  22:02:23 STEP 1096 H= 274:00 +CPU= 43.021
  22:03:04 STEP 1097 H= 274:15 +CPU= 41.362
  22:03:47 STEP 1098 H= 274:30 +CPU= 42.112
  22:04:29 STEP 1099 H= 274:45 +CPU= 42.408
  22:05:13 STEP 1100 H= 275:00 +CPU= 43.895
  22:05:56 STEP 1101 H= 275:15 +CPU= 42.821
  22:06:38 STEP 1102 H= 275:30 +CPU= 42.236
  22:07:21 STEP 1103 H= 275:45 +CPU= 42.430
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 NSTEP =  1094 STEPO   0AAA00AAA
 22:00:58 STEP 1094 H= 273:30 +CPU= 42.412
ISTEP=    1095 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1095 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1095 STEPO   0AAA00AAA
 22:01:40 STEP 1095 H= 273:45 +CPU= 42.122
 NSTEP =  1096 STEPO   0AAA00AAA
 22:02:23 STEP 1096 H= 274:00 +CPU= 43.021
 NSTEP =  1097 STEPO   0AAA00AAA
 22:03:04 STEP 1097 H= 274:15 +CPU= 41.362
 NSTEP =  1098 STEPO   0AAA00AAA
 22:03:47 STEP 1098 H= 274:30 +CPU= 42.112
 NSTEP =  1099 STEPO   0AAA00AAA
 22:04:29 STEP 1099 H= 274:45 +CPU= 42.408
ISTEP=    1100 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1100 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1100 STEPO   0AAA00AAA
 22:05:13 STEP 1100 H= 275:00 +CPU= 43.895
 NSTEP =  1101 STEPO   0AAA00AAA
 22:05:56 STEP 1101 H= 275:15 +CPU= 42.821
 NSTEP =  1102 STEPO   0AAA00AAA
 22:06:38 STEP 1102 H= 275:30 +CPU= 42.236
 NSTEP =  1103 STEPO   0AAA00AAA
 22:07:21 STEP 1103 H= 275:45 +CPU= 42.430
  DATE= 2016   10    4
ISTEP=    1104: Reset RSOLINC= 1361.4482 RIP0= 1361.4482
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.179466599480000E-004
  1.088035170184667E-006  4.990317592110001E-007  3.778213382954999E-009
  2.097581753513333E-009  7.940283000000001E-011
 suecaec NAERMACC =  1
 UPDT RAESHDU =  2000.0
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20160923
         810  9.999999999999998E-005
 NORMS AT NSTEP CNT4 1104
 SPECTRAL NORMS -   LOG(PREHYDS)    0.115129268475606E+02
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.418799504687064E-05 0.508010860298046E-06 0.198714206641508E+03 0.155866306694429E+02
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.902771828808329E-03 0.999999999999999E-08 0.123392437777881E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.137803074156069E-06 0.000000000000000E+00 0.540862304176490E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.382479536666403E-07 0.000000000000000E+00 0.482273986772203E-04
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.172419912373312E-07 0.000000000000000E+00 0.119865174066303E-03
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.905028101246938E-09 0.000000000000000E+00 0.339143529995915E-04
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.152361757798442E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.136291432792544E-06 0.000000000000000E+00 0.538741962352229E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.381752638933296E-07 0.000000000000000E+00 0.477603475844327E-04
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.152416159075358E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.416095873743249E-08 0.000000000000000E+00 0.156789687901793E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.208204338771482E-07 0.000000000000000E+00 0.146103344307767E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.643425321126108E-08 0.000000000000000E+00 0.206057445251764E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.299896126769682E-08 0.000000000000000E+00 0.115436341866865E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.347349802537393E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1104 STEPO   F00000000
 CREATE RESTART FILE ./srf00111200.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00111200.0001 MODE=w
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 22:03:04 0AAA00AAA STEPO     1098    41.362   41.362   41.412     33:10     88:28 0.48705369324310E-06       2GB       0MB
 22:03:47 0AAA00AAA STEPO     1099    42.112   42.112   42.183     33:52     89:11 0.48705369324310E-06       2GB       0MB
 22:04:29 0AAA00AAA STEPO     1100    42.408   42.408   42.472     34:34     89:53 0.48705369324310E-06       2GB       0MB
 22:05:13 0AAA00AAA STEPO     1101    43.896   43.896   43.919     35:18     90:37 0.48705369324310E-06       2GB       0MB
 22:05:56 0AAA00AAA STEPO     1102    42.822   42.822   42.882     36:01     91:20 0.48705369324310E-06       2GB       0MB
 22:06:38 0AAA00AAA STEPO     1103    42.236   42.236   42.295     36:43     92:02 0.48705369324310E-06       2GB       0MB
 22:07:21 F00000000 STEPO     1104    42.655   42.655   42.707     37:26     92:45 0.50801086029805E-06       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1056",
 CTIME   = "00110000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1044, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/41/progress_file_12211715.xml

<running_values>
  <last_cpu_time>63433.980000</last_cpu_time>
  <upload_file_number>10</upload_file_number>
  <last_iter>1104</last_iter>
  <last_upload>994500</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3sw
(argv4) batchid: 991
(argv5) wuid: 12211715
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/41
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/41/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/41/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3fff2b34ab37dd0282d56c92f3819210 to: /var/lib/boinc-client/slots/41/oifs_43r3_bl_a3sw_2016092300_15_991_12211715.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/41/oifs_43r3_bl_a3sw_2016092300_15_991_12211715.zip
ic_ancil_file: ic_ancil_12211715
ifsdata_file: ifsdata_12211715
climate_data_file: clim_data_12211715
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/41/ic_ancil_12211715.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/41/ic_ancil_12211715.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7b26120779d7ab5907e0d04cfa523855 to: /var/lib/boinc-client/slots/41/ifsdata/ifsdata_12211715.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/41/ifsdata/ifsdata_12211715.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/41/159l_2/clim_data_12211715.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/41/159l_2/clim_data_12211715.zip
Checking for progress XML file: /var/lib/boinc-client/slots/41/progress_file_12211715.xml
Opened progress file ok : /var/lib/boinc-client/slots/41/progress_file_12211715.xml
-- Model is restarting --
Adjusting last_iter, 1104, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/41/progress_file_12211715.xml
last_cpu_time: 63433
upload_file_number: 10
last_iter: 1105
last_upload: 994500
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a3sw_2016092300_15_991_12211715_0_r968882841
The child process has been launched with process id: 14607
Executing the command: /var/lib/boinc-client/slots/41/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:boinc5:1:1:14607:14607] [20230219:045439:1676782479:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:55:26 STEP 1056 H= 264:00 +CPU= 40.645
  04:56:05 STEP 1057 H= 264:15 +CPU= 38.448
  04:56:44 STEP 1058 H= 264:30 +CPU= 38.484
  04:57:22 STEP 1059 H= 264:45 +CPU= 38.457
  04:58:02 STEP 1060 H= 265:00 +CPU= 40.040
  04:58:42 STEP 1061 H= 265:15 +CPU= 39.390
  04:59:21 STEP 1062 H= 265:30 +CPU= 39.370
  05:00:00 STEP 1063 H= 265:45 +CPU= 39.330
  05:00:41 STEP 1064 H= 266:00 +CPU= 40.390
  05:01:20 STEP 1065 H= 266:15 +CPU= 39.247
  05:01:59 STEP 1066 H= 266:30 +CPU= 39.248
  05:02:39 STEP 1067 H= 266:45 +CPU= 39.292
  05:03:28 STEP 1068 H= 267:00 +CPU= 49.226
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001068
  05:04:07 STEP 1069 H= 267:15 +CPU= 38.951
  05:04:46 STEP 1070 H= 267:30 +CPU= 38.970
  05:05:25 STEP 1071 H= 267:45 +CPU= 38.943
  05:06:05 STEP 1072 H= 268:00 +CPU= 39.698
  05:06:43 STEP 1073 H= 268:15 +CPU= 38.530
  05:07:22 STEP 1074 H= 268:30 +CPU= 39.162
  05:08:02 STEP 1075 H= 268:45 +CPU= 39.316
  05:08:42 STEP 1076 H= 269:00 +CPU= 40.221
  05:09:21 STEP 1077 H= 269:15 +CPU= 39.243
  05:10:01 STEP 1078 H= 269:30 +CPU= 39.314
  05:10:40 STEP 1079 H= 269:45 +CPU= 39.182
  05:11:29 STEP 1080 H= 270:00 +CPU= 48.867
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001080
  05:12:07 STEP 1081 H= 270:15 +CPU= 38.589
  05:12:46 STEP 1082 H= 270:30 +CPU= 38.544
  05:13:25 STEP 1083 H= 270:45 +CPU= 39.301
  05:14:06 STEP 1084 H= 271:00 +CPU= 40.168
  05:14:45 STEP 1085 H= 271:15 +CPU= 39.156
  05:15:24 STEP 1086 H= 271:30 +CPU= 39.149
  05:16:03 STEP 1087 H= 271:45 +CPU= 39.186
  05:16:43 STEP 1088 H= 272:00 +CPU= 40.194
  05:17:22 STEP 1089 H= 272:15 +CPU= 38.970
  05:18:01 STEP 1090 H= 272:30 +CPU= 38.905
  05:18:40 STEP 1091 H= 272:45 +CPU= 38.938
  05:19:29 STEP 1092 H= 273:00 +CPU= 49.011
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001092
  05:20:09 STEP 1093 H= 273:15 +CPU= 39.536
  05:20:49 STEP 1094 H= 273:30 +CPU= 39.701
  05:21:28 STEP 1095 H= 273:45 +CPU= 39.637
  05:22:09 STEP 1096 H= 274:00 +CPU= 40.501
  05:22:49 STEP 1097 H= 274:15 +CPU= 39.568
  05:23:28 STEP 1098 H= 274:30 +CPU= 39.577
  05:24:08 STEP 1099 H= 274:45 +CPU= 39.514
  05:24:48 STEP 1100 H= 275:00 +CPU= 40.584
  05:25:28 STEP 1101 H= 275:15 +CPU= 39.417
  05:26:07 STEP 1102 H= 275:30 +CPU= 39.090
  05:26:46 STEP 1103 H= 275:45 +CPU= 38.979
  05:27:43 STEP 1104 H= 276:00 +CPU= 54.311
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001104
  05:28:21 STEP 1105 H= 276:15 +CPU= 38.790
  05:29:00 STEP 1106 H= 276:30 +CPU= 38.909
  05:29:39 STEP 1107 H= 276:45 +CPU= 38.908
  05:30:20 STEP 1108 H= 277:00 +CPU= 40.585
  05:31:00 STEP 1109 H= 277:15 +CPU= 39.565
  05:31:39 STEP 1110 H= 277:30 +CPU= 39.576
  05:32:19 STEP 1111 H= 277:45 +CPU= 39.508
  05:32:59 STEP 1112 H= 278:00 +CPU= 40.493
  05:33:39 STEP 1113 H= 278:15 +CPU= 39.373
  05:34:18 STEP 1114 H= 278:30 +CPU= 39.348
  05:34:57 STEP 1115 H= 278:45 +CPU= 39.200
  05:35:47 STEP 1116 H= 279:00 +CPU= 49.132
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001116
  05:36:26 STEP 1117 H= 279:15 +CPU= 38.981
  05:37:04 STEP 1118 H= 279:30 +CPU= 38.851
  05:37:43 STEP 1119 H= 279:45 +CPU= 38.856
  05:38:23 STEP 1120 H= 280:00 +CPU= 39.820
  05:39:02 STEP 1121 H= 280:15 +CPU= 38.819
  05:39:41 STEP 1122 H= 280:30 +CPU= 39.258
  05:40:21 STEP 1123 H= 280:45 +CPU= 39.216
  05:41:01 STEP 1124 H= 281:00 +CPU= 40.124
  05:41:40 STEP 1125 H= 281:15 +CPU= 39.279
  05:42:20 STEP 1126 H= 281:30 +CPU= 39.306
  05:42:59 STEP 1127 H= 281:45 +CPU= 39.421
  05:43:48 STEP 1128 H= 282:00 +CPU= 48.986
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001128
  05:44:27 STEP 1129 H= 282:15 +CPU= 38.786
  05:45:06 STEP 1130 H= 282:30 +CPU= 38.628
  05:45:44 STEP 1131 H= 282:45 +CPU= 38.516
  05:46:24 STEP 1132 H= 283:00 +CPU= 39.483
  05:47:02 STEP 1133 H= 283:15 +CPU= 38.529
  05:47:41 STEP 1134 H= 283:30 +CPU= 38.507
  05:48:19 STEP 1135 H= 283:45 +CPU= 38.521
  05:48:59 STEP 1136 H= 284:00 +CPU= 39.539
  05:49:37 STEP 1137 H= 284:15 +CPU= 38.526
  05:50:17 STEP 1138 H= 284:30 +CPU= 39.588
  05:50:56 STEP 1139 H= 284:45 +CPU= 39.212
  05:51:45 STEP 1140 H= 285:00 +CPU= 48.560
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001140
  05:52:24 STEP 1141 H= 285:15 +CPU= 38.560
  05:53:02 STEP 1142 H= 285:30 +CPU= 38.594
  05:53:41 STEP 1143 H= 285:45 +CPU= 38.559
  05:54:20 STEP 1144 H= 286:00 +CPU= 39.549
  05:54:59 STEP 1145 H= 286:15 +CPU= 38.565
  05:55:38 STEP 1146 H= 286:30 +CPU= 38.602
  05:56:16 STEP 1147 H= 286:45 +CPU= 38.830
  05:56:57 STEP 1148 H= 287:00 +CPU= 40.642
  05:57:37 STEP 1149 H= 287:15 +CPU= 39.792
  05:58:17 STEP 1150 H= 287:30 +CPU= 39.804
  05:58:56 STEP 1151 H= 287:45 +CPU= 39.265
  05:59:50 STEP 1152 H= 288:00 +CPU= 54.217
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001152
  06:00:29 STEP 1153 H= 288:15 +CPU= 38.749
  06:01:08 STEP 1154 H= 288:30 +CPU= 38.811
  06:01:48 STEP 1155 H= 288:45 +CPU= 39.564
  06:02:29 STEP 1156 H= 289:00 +CPU= 40.621
  06:03:09 STEP 1157 H= 289:15 +CPU= 39.519
  06:03:49 STEP 1158 H= 289:30 +CPU= 39.425
  06:04:28 STEP 1159 H= 289:45 +CPU= 39.193
  06:05:09 STEP 1160 H= 290:00 +CPU= 40.114
  06:05:49 STEP 1161 H= 290:15 +CPU= 39.522
  06:06:29 STEP 1162 H= 290:30 +CPU= 39.850
  06:07:09 STEP 1163 H= 290:45 +CPU= 39.178
  06:07:58 STEP 1164 H= 291:00 +CPU= 49.551
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001164
  06:08:38 STEP 1165 H= 291:15 +CPU= 39.269
  06:09:17 STEP 1166 H= 291:30 +CPU= 39.229
  06:09:56 STEP 1167 H= 291:45 +CPU= 39.248
  06:10:37 STEP 1168 H= 292:00 +CPU= 40.166
  06:11:16 STEP 1169 H= 292:15 +CPU= 39.006
  06:11:55 STEP 1170 H= 292:30 +CPU= 38.976
  06:12:34 STEP 1171 H= 292:45 +CPU= 38.960
  06:13:14 STEP 1172 H= 293:00 +CPU= 39.808
  06:13:54 STEP 1173 H= 293:15 +CPU= 39.058
  06:14:35 STEP 1174 H= 293:30 +CPU= 40.823
  06:15:16 STEP 1175 H= 293:45 +CPU= 41.724
  06:16:06 STEP 1176 H= 294:00 +CPU= 49.966
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001176
  06:16:46 STEP 1177 H= 294:15 +CPU= 39.339
  06:17:25 STEP 1178 H= 294:30 +CPU= 39.709
  06:18:05 STEP 1179 H= 294:45 +CPU= 39.332
  06:18:46 STEP 1180 H= 295:00 +CPU= 41.018
  06:19:25 STEP 1181 H= 295:15 +CPU= 39.193
  06:20:05 STEP 1182 H= 295:30 +CPU= 39.362
  06:20:44 STEP 1183 H= 295:45 +CPU= 39.566
  06:21:25 STEP 1184 H= 296:00 +CPU= 40.546
  06:22:04 STEP 1185 H= 296:15 +CPU= 39.336
  06:22:44 STEP 1186 H= 296:30 +CPU= 39.452
  06:23:24 STEP 1187 H= 296:45 +CPU= 39.864
  06:24:14 STEP 1188 H= 297:00 +CPU= 50.193
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001188
  06:24:54 STEP 1189 H= 297:15 +CPU= 39.968
  06:25:34 STEP 1190 H= 297:30 +CPU= 39.865
  06:26:14 STEP 1191 H= 297:45 +CPU= 40.619
  06:26:55 STEP 1192 H= 298:00 +CPU= 40.718
  06:27:35 STEP 1193 H= 298:15 +CPU= 39.679
  06:28:14 STEP 1194 H= 298:30 +CPU= 39.518
  06:28:54 STEP 1195 H= 298:45 +CPU= 39.464
  06:29:35 STEP 1196 H= 299:00 +CPU= 40.506
  06:30:14 STEP 1197 H= 299:15 +CPU= 39.857
  06:30:55 STEP 1198 H= 299:30 +CPU= 40.580
  06:31:35 STEP 1199 H= 299:45 +CPU= 39.988
  06:32:31 STEP 1200 H= 300:00 +CPU= 55.141
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001200
  06:33:10 STEP 1201 H= 300:15 +CPU= 39.127
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001200
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3sw_2016092300_15_991_12211715_0_r968882841_10.zip
Uploading the intermediate file: upload_file_10.zip
  06:33:50 STEP 1202 H= 300:30 +CPU= 39.465
  06:34:29 STEP 1203 H= 300:45 +CPU= 39.702
  06:35:10 STEP 1204 H= 301:00 +CPU= 40.900
  06:35:50 STEP 1205 H= 301:15 +CPU= 39.652
  06:36:29 STEP 1206 H= 301:30 +CPU= 39.329
  06:37:09 STEP 1207 H= 301:45 +CPU= 39.361
  06:37:49 STEP 1208 H= 302:00 +CPU= 40.574
  06:38:29 STEP 1209 H= 302:15 +CPU= 40.068
  06:39:09 STEP 1210 H= 302:30 +CPU= 39.052
  06:39:48 STEP 1211 H= 302:45 +CPU= 39.300
  06:40:38 STEP 1212 H= 303:00 +CPU= 50.089
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001212
  06:41:18 STEP 1213 H= 303:15 +CPU= 39.838
  06:41:57 STEP 1214 H= 303:30 +CPU= 39.470
  06:42:36 STEP 1215 H= 303:45 +CPU= 39.056
  06:43:16 STEP 1216 H= 304:00 +CPU= 39.907
  06:43:56 STEP 1217 H= 304:15 +CPU= 39.200
  06:44:35 STEP 1218 H= 304:30 +CPU= 39.156
  06:45:14 STEP 1219 H= 304:45 +CPU= 39.369
  06:45:55 STEP 1220 H= 305:00 +CPU= 41.318
  06:46:36 STEP 1221 H= 305:15 +CPU= 40.192
  06:47:15 STEP 1222 H= 305:30 +CPU= 39.161
  06:47:54 STEP 1223 H= 305:45 +CPU= 39.026
  06:48:43 STEP 1224 H= 306:00 +CPU= 49.390
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001224
  06:49:23 STEP 1225 H= 306:15 +CPU= 39.744
  06:50:03 STEP 1226 H= 306:30 +CPU= 39.728
  06:50:42 STEP 1227 H= 306:45 +CPU= 39.305
  06:51:22 STEP 1228 H= 307:00 +CPU= 39.792
  06:52:01 STEP 1229 H= 307:15 +CPU= 38.704
  06:52:40 STEP 1230 H= 307:30 +CPU= 38.773
  06:53:18 STEP 1231 H= 307:45 +CPU= 38.825
  06:53:58 STEP 1232 H= 308:00 +CPU= 39.923
  06:54:37 STEP 1233 H= 308:15 +CPU= 39.008
  06:55:16 STEP 1234 H= 308:30 +CPU= 38.759
  06:55:55 STEP 1235 H= 308:45 +CPU= 38.703
  06:56:43 STEP 1236 H= 309:00 +CPU= 48.590
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001236
  06:57:22 STEP 1237 H= 309:15 +CPU= 38.615
  06:58:01 STEP 1238 H= 309:30 +CPU= 38.627
  06:58:39 STEP 1239 H= 309:45 +CPU= 38.624
  06:59:19 STEP 1240 H= 310:00 +CPU= 39.605
  06:59:58 STEP 1241 H= 310:15 +CPU= 38.663
  07:00:36 STEP 1242 H= 310:30 +CPU= 38.717
  07:01:16 STEP 1243 H= 310:45 +CPU= 39.400
  07:01:56 STEP 1244 H= 311:00 +CPU= 40.172
  07:02:35 STEP 1245 H= 311:15 +CPU= 39.029
  07:03:14 STEP 1246 H= 311:30 +CPU= 38.705
  07:03:53 STEP 1247 H= 311:45 +CPU= 38.658
  07:04:47 STEP 1248 H= 312:00 +CPU= 54.213
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001248
  07:05:25 STEP 1249 H= 312:15 +CPU= 38.655
  07:06:04 STEP 1250 H= 312:30 +CPU= 38.723
  07:06:43 STEP 1251 H= 312:45 +CPU= 38.734
  07:07:23 STEP 1252 H= 313:00 +CPU= 39.671
  07:08:01 STEP 1253 H= 313:15 +CPU= 38.729
  07:08:40 STEP 1254 H= 313:30 +CPU= 38.766
  07:09:19 STEP 1255 H= 313:45 +CPU= 38.825
  07:09:59 STEP 1256 H= 314:00 +CPU= 39.878
  07:10:38 STEP 1257 H= 314:15 +CPU= 38.990
  07:11:17 STEP 1258 H= 314:30 +CPU= 38.720
  07:11:55 STEP 1259 H= 314:45 +CPU= 38.700
  07:12:44 STEP 1260 H= 315:00 +CPU= 48.838
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001260
  07:13:23 STEP 1261 H= 315:15 +CPU= 38.839
  07:14:02 STEP 1262 H= 315:30 +CPU= 38.913
  07:14:41 STEP 1263 H= 315:45 +CPU= 38.916
  07:15:21 STEP 1264 H= 316:00 +CPU= 39.855
  07:16:00 STEP 1265 H= 316:15 +CPU= 38.935
  07:16:39 STEP 1266 H= 316:30 +CPU= 38.984
  07:17:18 STEP 1267 H= 316:45 +CPU= 39.092
  07:17:58 STEP 1268 H= 317:00 +CPU= 40.181
  07:18:38 STEP 1269 H= 317:15 +CPU= 40.377
  07:19:18 STEP 1270 H= 317:30 +CPU= 39.973
  07:19:58 STEP 1271 H= 317:45 +CPU= 39.827
  07:20:48 STEP 1272 H= 318:00 +CPU= 49.762
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001272
  07:21:27 STEP 1273 H= 318:15 +CPU= 39.088
  07:22:06 STEP 1274 H= 318:30 +CPU= 39.006
  07:22:45 STEP 1275 H= 318:45 +CPU= 38.989
  07:23:25 STEP 1276 H= 319:00 +CPU= 39.924
  07:24:04 STEP 1277 H= 319:15 +CPU= 38.997
  07:24:43 STEP 1278 H= 319:30 +CPU= 38.958
  07:25:22 STEP 1279 H= 319:45 +CPU= 39.003
  07:26:02 STEP 1280 H= 320:00 +CPU= 40.174
  07:26:42 STEP 1281 H= 320:15 +CPU= 39.233
  07:27:21 STEP 1282 H= 320:30 +CPU= 39.000
  07:28:01 STEP 1283 H= 320:45 +CPU= 39.845
  07:28:51 STEP 1284 H= 321:00 +CPU= 50.238
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001284
  07:29:31 STEP 1285 H= 321:15 +CPU= 39.761
  07:30:10 STEP 1286 H= 321:30 +CPU= 39.561
  07:30:50 STEP 1287 H= 321:45 +CPU= 39.981
  07:31:31 STEP 1288 H= 322:00 +CPU= 40.376
  07:32:11 STEP 1289 H= 322:15 +CPU= 39.955
  07:32:50 STEP 1290 H= 322:30 +CPU= 39.761
  07:33:30 STEP 1291 H= 322:45 +CPU= 39.180
  07:34:10 STEP 1292 H= 323:00 +CPU= 40.258
  07:34:49 STEP 1293 H= 323:15 +CPU= 39.392
  07:35:28 STEP 1294 H= 323:30 +CPU= 39.024
  07:36:07 STEP 1295 H= 323:45 +CPU= 39.037
  07:37:02 STEP 1296 H= 324:00 +CPU= 54.505
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001296
  07:37:41 STEP 1297 H= 324:15 +CPU= 38.941
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001296
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3sw_2016092300_15_991_12211715_0_r968882841_11.zip
Uploading the intermediate file: upload_file_11.zip
  07:38:20 STEP 1298 H= 324:30 +CPU= 38.941
  07:38:59 STEP 1299 H= 324:45 +CPU= 38.846
  07:39:39 STEP 1300 H= 325:00 +CPU= 39.847
  07:40:19 STEP 1301 H= 325:15 +CPU= 39.978
  07:40:59 STEP 1302 H= 325:30 +CPU= 40.057
  07:41:40 STEP 1303 H= 325:45 +CPU= 40.150
  07:42:21 STEP 1304 H= 326:00 +CPU= 41.393
  07:43:01 STEP 1305 H= 326:15 +CPU= 40.032
  07:43:40 STEP 1306 H= 326:30 +CPU= 39.256
  07:44:19 STEP 1307 H= 326:45 +CPU= 39.070
  07:45:08 STEP 1308 H= 327:00 +CPU= 49.056
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001308
  07:45:47 STEP 1309 H= 327:15 +CPU= 38.948
  07:46:26 STEP 1310 H= 327:30 +CPU= 38.962
  07:47:05 STEP 1311 H= 327:45 +CPU= 38.986
  07:47:45 STEP 1312 H= 328:00 +CPU= 39.968
  07:48:24 STEP 1313 H= 328:15 +CPU= 38.949
  07:49:03 STEP 1314 H= 328:30 +CPU= 39.036
  07:49:42 STEP 1315 H= 328:45 +CPU= 39.094
  07:50:23 STEP 1316 H= 329:00 +CPU= 40.290
  07:51:02 STEP 1317 H= 329:15 +CPU= 39.160
  07:51:41 STEP 1318 H= 329:30 +CPU= 38.954
  07:52:20 STEP 1319 H= 329:45 +CPU= 38.964
  07:53:09 STEP 1320 H= 330:00 +CPU= 48.965
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001320
  07:53:48 STEP 1321 H= 330:15 +CPU= 38.910
  07:54:27 STEP 1322 H= 330:30 +CPU= 38.971
  07:55:06 STEP 1323 H= 330:45 +CPU= 38.960
  07:55:46 STEP 1324 H= 331:00 +CPU= 39.960
  07:56:25 STEP 1325 H= 331:15 +CPU= 39.028
  07:57:04 STEP 1326 H= 331:30 +CPU= 39.062
  07:57:43 STEP 1327 H= 331:45 +CPU= 39.462
  07:58:25 STEP 1328 H= 332:00 +CPU= 41.418
  07:59:05 STEP 1329 H= 332:15 +CPU= 40.263
  07:59:45 STEP 1330 H= 332:30 +CPU= 40.150
  08:00:25 STEP 1331 H= 332:45 +CPU= 39.881
  08:01:15 STEP 1332 H= 333:00 +CPU= 49.843
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001332
  08:01:55 STEP 1333 H= 333:15 +CPU= 39.569
  08:02:34 STEP 1334 H= 333:30 +CPU= 39.344
  08:03:14 STEP 1335 H= 333:45 +CPU= 39.449
  08:03:54 STEP 1336 H= 334:00 +CPU= 39.969
  08:04:32 STEP 1337 H= 334:15 +CPU= 38.788
  08:05:11 STEP 1338 H= 334:30 +CPU= 38.901
  08:05:51 STEP 1339 H= 334:45 +CPU= 39.254
  08:06:31 STEP 1340 H= 335:00 +CPU= 40.278
  08:07:10 STEP 1341 H= 335:15 +CPU= 38.937
  08:07:49 STEP 1342 H= 335:30 +CPU= 38.791
  08:08:28 STEP 1343 H= 335:45 +CPU= 38.928
  08:09:23 STEP 1344 H= 336:00 +CPU= 55.203
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001344
  08:10:03 STEP 1345 H= 336:15 +CPU= 39.743
  08:10:43 STEP 1346 H= 336:30 +CPU= 39.830
  08:11:23 STEP 1347 H= 336:45 +CPU= 40.044
  08:12:03 STEP 1348 H= 337:00 +CPU= 40.451
  08:12:42 STEP 1349 H= 337:15 +CPU= 38.830
  08:13:21 STEP 1350 H= 337:30 +CPU= 38.940
  08:14:00 STEP 1351 H= 337:45 +CPU= 39.086
  08:14:41 STEP 1352 H= 338:00 +CPU= 40.540
  08:15:20 STEP 1353 H= 338:15 +CPU= 39.118
  08:15:59 STEP 1354 H= 338:30 +CPU= 38.981
  08:16:39 STEP 1355 H= 338:45 +CPU= 39.789
  08:17:28 STEP 1356 H= 339:00 +CPU= 49.568
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001356
  08:18:08 STEP 1357 H= 339:15 +CPU= 39.327
  08:18:47 STEP 1358 H= 339:30 +CPU= 39.091
  08:19:26 STEP 1359 H= 339:45 +CPU= 38.856
  08:20:07 STEP 1360 H= 340:00 +CPU= 40.902
  08:20:47 STEP 1361 H= 340:15 +CPU= 39.921
  08:21:26 STEP 1362 H= 340:30 +CPU= 39.918
  08:22:07 STEP 1363 H= 340:45 +CPU= 40.150
  08:22:48 STEP 1364 H= 341:00 +CPU= 41.213
  08:23:28 STEP 1365 H= 341:15 +CPU= 39.837
  08:24:07 STEP 1366 H= 341:30 +CPU= 39.485
  08:24:47 STEP 1367 H= 341:45 +CPU= 39.474
  08:25:36 STEP 1368 H= 342:00 +CPU= 49.319
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001368
  08:26:15 STEP 1369 H= 342:15 +CPU= 39.000
  08:26:54 STEP 1370 H= 342:30 +CPU= 39.005
  08:27:33 STEP 1371 H= 342:45 +CPU= 39.067
  08:28:14 STEP 1372 H= 343:00 +CPU= 40.910
  08:28:54 STEP 1373 H= 343:15 +CPU= 39.979
  08:29:34 STEP 1374 H= 343:30 +CPU= 39.798
  08:30:14 STEP 1375 H= 343:45 +CPU= 39.661
  08:30:54 STEP 1376 H= 344:00 +CPU= 40.306
  08:31:33 STEP 1377 H= 344:15 +CPU= 38.990
  08:32:12 STEP 1378 H= 344:30 +CPU= 39.087
  08:32:51 STEP 1379 H= 344:45 +CPU= 39.266
  08:33:40 STEP 1380 H= 345:00 +CPU= 48.978
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001380
  08:34:19 STEP 1381 H= 345:15 +CPU= 38.934
  08:34:58 STEP 1382 H= 345:30 +CPU= 39.023
  08:35:38 STEP 1383 H= 345:45 +CPU= 39.484
  08:36:18 STEP 1384 H= 346:00 +CPU= 40.080
  08:36:58 STEP 1385 H= 346:15 +CPU= 39.972
  08:37:38 STEP 1386 H= 346:30 +CPU= 39.971
  08:38:18 STEP 1387 H= 346:45 +CPU= 39.991
  08:38:58 STEP 1388 H= 347:00 +CPU= 40.617
  08:39:38 STEP 1389 H= 347:15 +CPU= 39.006
  08:40:16 STEP 1390 H= 347:30 +CPU= 38.969
  08:40:56 STEP 1391 H= 347:45 +CPU= 39.301
  08:41:51 STEP 1392 H= 348:00 +CPU= 54.684
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001392
  08:42:29 STEP 1393 H= 348:15 +CPU= 38.886
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001392
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3sw_2016092300_15_991_12211715_0_r968882841_12.zip
Uploading the intermediate file: upload_file_12.zip
  08:43:10 STEP 1394 H= 348:30 +CPU= 39.686
  08:43:49 STEP 1395 H= 348:45 +CPU= 39.237
  08:44:30 STEP 1396 H= 349:00 +CPU= 40.615
  08:45:09 STEP 1397 H= 349:15 +CPU= 39.203
  08:45:48 STEP 1398 H= 349:30 +CPU= 39.176
  08:46:28 STEP 1399 H= 349:45 +CPU= 39.617
  08:47:09 STEP 1400 H= 350:00 +CPU= 41.406
  08:47:49 STEP 1401 H= 350:15 +CPU= 40.242
  08:48:29 STEP 1402 H= 350:30 +CPU= 39.839
  08:49:09 STEP 1403 H= 350:45 +CPU= 40.090
  08:49:59 STEP 1404 H= 351:00 +CPU= 49.510
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001404
  08:50:38 STEP 1405 H= 351:15 +CPU= 39.388
  08:51:19 STEP 1406 H= 351:30 +CPU= 40.505
  08:51:59 STEP 1407 H= 351:45 +CPU= 40.450
  08:52:41 STEP 1408 H= 352:00 +CPU= 41.453
  08:53:22 STEP 1409 H= 352:15 +CPU= 40.733
  08:54:02 STEP 1410 H= 352:30 +CPU= 40.511
  08:54:42 STEP 1411 H= 352:45 +CPU= 40.337
  08:55:23 STEP 1412 H= 353:00 +CPU= 40.748
  08:56:03 STEP 1413 H= 353:15 +CPU= 39.336
  08:56:42 STEP 1414 H= 353:30 +CPU= 39.480
  08:57:22 STEP 1415 H= 353:45 +CPU= 39.902
  08:58:12 STEP 1416 H= 354:00 +CPU= 49.623
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001416
  08:58:51 STEP 1417 H= 354:15 +CPU= 39.565
  08:59:31 STEP 1418 H= 354:30 +CPU= 39.623
  09:00:11 STEP 1419 H= 354:45 +CPU= 40.308
  09:00:52 STEP 1420 H= 355:00 +CPU= 41.210
  09:01:33 STEP 1421 H= 355:15 +CPU= 40.501
  09:02:13 STEP 1422 H= 355:30 +CPU= 40.176
  09:02:54 STEP 1423 H= 355:45 +CPU= 41.240
  09:03:36 STEP 1424 H= 356:00 +CPU= 41.765
  09:04:17 STEP 1425 H= 356:15 +CPU= 40.506
  09:04:57 STEP 1426 H= 356:30 +CPU= 40.255
  09:05:38 STEP 1427 H= 356:45 +CPU= 40.593
  09:06:29 STEP 1428 H= 357:00 +CPU= 51.009
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001428
  09:07:10 STEP 1429 H= 357:15 +CPU= 40.851
  09:07:51 STEP 1430 H= 357:30 +CPU= 40.993
  09:08:32 STEP 1431 H= 357:45 +CPU= 41.294
  09:09:14 STEP 1432 H= 358:00 +CPU= 41.654
  09:09:56 STEP 1433 H= 358:15 +CPU= 42.468
  09:10:38 STEP 1434 H= 358:30 +CPU= 41.300
  09:11:20 STEP 1435 H= 358:45 +CPU= 42.009
  09:12:03 STEP 1436 H= 359:00 +CPU= 43.308
  09:12:45 STEP 1437 H= 359:15 +CPU= 42.474
  09:13:28 STEP 1438 H= 359:30 +CPU= 42.695
  09:14:10 STEP 1439 H= 359:45 +CPU= 41.833
  09:15:06 STEP 1440 H= 360:00 +CPU= 55.398
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.6999737500E-21      I       0.9348847457E-06      I
ICloud fractionI      0.0000000000E+00      I       0.3114423474E-01      I
I  Relat. Hum. I      0.1071609429E+00      I       0.1949527912E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1979788627E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    304122993 BYTES IN      7 RECORDS TRANSFERRED IN    0.0014 SECONDS217230.7 Mbytes/s, TOTAL TIME=    0.0176( 8.0%)
 09:15:06 STEP 1440 H= 360:00 +CPU= 55.398
 END CNT3
 NSTEP =  1440 CNT0   000000000  40.389  40.389  40.452   0.998               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN        11344/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    203    0       34      1044068    11344 ************    34.6     3.9    30.8     5.1    64.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.96      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    5763.4    5763.4     15.38      0.00
   9 SPCM     - SPECTRAL COMP.                     384    1118.7    1118.7      2.75      0.00
  10 SCAN2M - PHYSICS                              385   20951.4   20951.4     51.62      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32    1476.2    1476.2      0.30      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      44.5      44.5      0.00      0.00
  17 GRIDFPOS IN CNT4                               32      35.3      35.3      0.01      0.00
  27 RESTART FILES - WRITING                         8    5904.7    5904.7      0.30      0.00
  28 RESTART FILES - READING                         1    6172.4    6172.4      0.04      0.00
  29 SU4FPOS IN CNT4                                32       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                32    8757.5    8757.5      1.79      0.00
  31 POSDDH IN STEPO                                 2       2.7       2.7      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    7669.8    7669.8     18.90      0.00
  39 SU0YOMB                                         1     572.9     572.9      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385      49.5      49.5      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384     269.4     269.4      0.66      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      90.7      90.7      0.22      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.5       0.5      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      836     599.3     599.3      3.21      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800     371.8     371.8      1.90      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800     134.3     134.3      0.69      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       836     221.3     221.3      1.18      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      63.0      63.0      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            836      82.4      82.4      0.44      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800      45.2      45.2      0.23      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            836      87.8      87.8      0.47      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800      56.7      56.7      0.29      0.00
 400 GSTATS                                      45376       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 42168       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    15626.8 CPU TIME   15604.8 VECTOR TIME    15604.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       83.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/41/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/41/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/41/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12211715/ICMSHh7zg+001092
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3sw_2016092300_15_991_12211715_0_r968882841_13.zip
Uploading the final file: upload_file_13.zip
Uploading trickle at timestep: 1295100
09:18:52 (14604): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a3sw_2016092300_15_991_12211715_0_r968882841_14.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:52:18 1537658 22311962 oifs_43r3_bl_a3sw_2016092300_15_991_12211715_0 1,295,100 79,037 0.0610


©2024 climateprediction.net