Skip to main content

Join thousands of industry professionals and get up-to-date information on breakthrough energy solutions

 

Persistence of Savings from Retro-Commissioning Measures

Introduction

Commonwealth Edison Company (ComEd) is a large, investor-owned electric utility covering most of northern Illinois. The utility manages over 90,000 miles of power lines across an 11,400 square-mile service territory. ComEd has over four million customers; primary sectors include residential customers (31% of total energy usage), small commercial and industrial customers (35% of energy usage), and large commercial and industrial customers (25% of energy usage).

In December 2016, the Illinois General Assembly enacted Public Act 099-0906 (PA99-0906), which amended the existing Energy Efficiency Portfolio Standard (EEPS) for electric utilities set forth in the Public Utilities Act. At the time, ComEd was in its ninth year of providing energy efficiency programs to customers to meet legislatively-mandated savings goals established in the EEPS legislation. PA99-0906 increased these goals and made many other changes, including basing energy savings goals on cumulative persisting annual savings (CPAS) instead of first-year energy savings, as was previously used. Under the new legislation, ComEd has increasing CPAS goals from 2018 to 2030.

This shift to CPAS requires more focus on the longevity of energy conservation measures; cumulative persisting annual savings are based heavily on the estimated useful life (EUL) of measures. In ComEd’s retro-commissioning (RCx) program, EUL historically was assumed to be low. This assumption was made without any specific basis (which was acceptable because measure life was not an issue; the measures simply had to last long enough to be cost-effective). But with the RCx program now measuring savings based on CPAS, the EUL of RCx measures is critically important to the program. As a result, ComEd contracted Slipstream to undertake a study of persistence in past RCx projects.

Objectives

The primary objectives of this study were to: 

  • Quantify the persistence of RCx measures based on the EUL of all installed RCx measures. 
  • Identify operational factors and energy management characteristics that influence the high persistence of savings. 

Secondary objectives included: 

  • Investigate potential for spillover savings: savings pertaining to measures that were recommended by the RCx program, but implemented at a later time. These savings are influenced by the program and count toward program savings. 
  • Understand the potential to go back to past participants and conduct RCx again, specifically past participants that would be good candidates for this follow-up FCx. 

Literature Review

Slipstream first conducted a literature review to determine the industry’s understanding of the persistence of RCx and RCx-like measures. We used the results of the literature review to inform the experimental design for the site study section of this research.

Publications Reviewed 

We reviewed thirteen publications that attempted to quantify the persistence of RCx. We also reviewed an additional five publications that described methodologies for studying persistence in utility programs. All of these documents are relatively current, published in 2003 or later, and cover research and programs across the United States.

Persistence Study Methodologies 

Persistence studies generally fall into two categories: measure life studies and retention studies. Measure life studies (Violette 2013) attempt to measure the specific Estimated Useful Life (EUL) of measures using a statistical model (known as survival analysis). This approach results in the best outcome of a persistence study because it results in a specific measure life value, and is well documented in the literature (Skumatz 2009, Viollette 2013). But it requires very large sample sizes because a significant number of both failures and successes are needed over a wide enough range of years to determine measure life with any precision. These studies are therefore generally relegated to simple, prescriptive measures where there are large sample sizes. We only found one RCx study that attempted this approach; that study still required significant extrapolation to determine a specific lifetime

All other RCx persistence studies that we reviewed were of the retention type (see Hall 2006). Retention studies choose a specific length of time and attempt to determine the rate of persistence at that time. By definition, if the persistence in that timeframe is greater than 50%, then the EUL must be longer than the measurement timeframe. For example, if the study investigates buildings that were retro-commissioned between four and five years ago, and find a persistence of 80% (with acceptable statistical uncertainty), it can be assumed that the EUL is five years or greater. These studies can be successful with a sample size that is reasonable for an RCx program.

Of the thirteen publications that quantified RCx persistence, five of them were evaluation or program reference documents that assumed an EUL or referenced it from another study or program. It appears that many RCx programs do not conduct primary studies of the persistence of RCx savings. Instead, they refer to EULs that have been established in other jurisdictions or territories, make a conservative assumption as long as it results in cost-effectiveness, or just assume an EUL based on judgment.

Persistence Study Results

Table 1 and Table 2 summarize the thirteen publications that quantify RCx savings persistence.

Eight of these publications quantify persistence based on primary research of RCx persistence, or of the persistence of new building commissioning (Cx). Only one of these eight focused solely on Cx (Potter 2002); the others all included some significant amount of retro-commissioned buildings. These studies focused on a wide variety of building types. Their methodologies were split between three studies that quantified persistence at the whole building energy level, while the other five gathered data by measure or end-use type. However, these latter five did not publish complete summary data by measure, at least not for the standard metric of persistence.

A variety of strategies were used to gather the data in these studies. Most used some type of on-site data collection, often through observation of controls or building automation interfaces. On-site observation of equipment operation was sometimes included.

Three of the studies also analyzed energy usage data to determine persistence, generally using statistical approaches to determine whether the whole-building energy (or major end-use) savings that resulted from RCx were persisting. Selch (2005) took this approach a step further and re-implemented measures that had failed or been removed in order to test the energy impact of those failures. And one study, by Mills (2011), based persistence on primary and secondary whole-building energy usage data. This study was a compilation of data collected directly by the author, as well as data from other studies.

Three of the studies used surveys or interviews to determine if a measure was persisting. Researchers would simply ask building operators questions about individual measures and control sequences to determine if measures had persisted. All three used these tools in conjunction with on-site data collection though.

Table 1. Characteristics of sites studied
  Primary Author Location Year Type of Cx No. of bldgs By measure or building?  Study type Building types
Measured persistence

Bourassa

Eardley

Potter

Mills

Peterson

Roberts

Selch

Toole

CA (SMUD)

(Anon).

W. coast

Varied

OR

CA

CO

TX

2004

2007

2003

2009

2005

2008

2005

2011

RCx

RCx

Cx

Cx, RCx

RCx

RCx

RCx

RCx

8

2

10

36

3

32

1

10

Building

Measure

Measure

Building

Measure

Measure

Measure

Building 

Survey, on-site

On-site

Survey, on-site

Energy usage

On-site

Survey (some on-site)

Usage analysis, reconducting RCx

Usage analysis, on-site, some simulation 

All, large

Large office

Lab, office, med

All

Offices (Intel)

All

Large office

University

Evaluated or published assumptions 

Cadmus-1

Cadmus-2

CPUC

SBW

Tetra Tech 

UT (Pacificorp)

ME (Eff. Maine)

CA

CA

CO (Xcel)

2010

2012

2016

2009

2011

ReCx

RCx

RCx

RCx

ReCx

   

Evaluation 

Evaluation 

Established assumptions 

Field study and evaluation 

Evaluation 

 

 

Table 2. Results from sites studied

  Primary Author Time After Cx Results: Persistence in % or EUL in years (both energy weighted, unless noted) 

Measured persistence

Bourassa

Eardley

Potter

Mills

Peterson

Roberts

Selch

Toole

1-2 years

Not stated

2-8 years

3 years

4 years

1-3 years

8 years

3 years

81%

36% (on measure count basis)

69% (on measure count basis); persistence issues on schedules, sensors, VAV boxes, boiler control

88% (on measure count basis)

Electricity: 89% (Chiller plant, economizer); Gas: 0% (VAV box control) 

Estimated (rough) average EUL of 8 years, using survival analysis

Electricity: 83%; Gas: 100%

Electricity: 100%, CHW: 83%, HW: inconclusive. Overall: 83% 

Evaluated or published assumptions

Cadmus-1

Cadmus-2

CPUC

SBW

Tetra Tech 

1 year

1 year

---

1-3 years

---

Assume 7 years for EUL based on DEER/Calmac references

Assume 5 years for EUL

EULs (years) Economizer: 5, RTU control: 5, Lightning control: 8, Supermarket: 10, Ventilation: 10, Resets: 10

EUL: varies by the utility, 3-20 years

Assume 7 years for EUL based on DEER/Calmac references 

 

As Table 2 shows, the measure by measure results across the eight quantitative studies are highly varied in type, timeframe, and approach. This variance prevents us from compiling specific persistence results by measure across a significant number of buildings, although the study by Roberts (2010) is an exception. We do have enough data to consider overall averages, however. Based on building count, much of the data falls into the two- to three-year timeframe. In that timeframe, persistence averages 80% (weighting each study by building count). If we consider all of the studies, which range from one to eight years in the timeframe, the average persistence is 76%. Little of the data in that statistic, however, comes from longer timeframe studies (six- to eight-years).

Two studies departed from this trend. Peterson (2005) investigated three buildings on Intel’s corporate campus about four years after they had been retro-commissioned. While the study found a fairly typical (83%) rate of persistence with electric measures, the few gas measures that they studied had all been disabled or changed, resulting in a rated persistence of 0% for gas. Eardley (2007) also found lower than typical persistence. That study only investigated two buildings but found that roughly two-thirds of the measures in those buildings had been disabled or had their effectiveness significantly reduced. These studies are included in the averages above but represent less than 10% of the buildings studied in the eight publications.

The Roberts (2010) study is another outlier because it provides substantially more data than the other studies. Roberts not only investigated 32 buildings in depth but also did it over a range of timeframes with the goal of completing a survival curve, with an accurate estimate of EUL. The majority of typical RCx measures (schedules, resets, setpoints, valves/dampers, etc.) were investigated, and most showed high persistence. Several measures did have some degradation, which led to 80% of the measures persisted after three years. Specific measures that degraded more than others include boiler controls (33% of measures persisted), HVAC scheduling (60%), temperature setpoints (38%), and lighting occupancy sensors (0%).

Though the measures were investigated individually, the survival analysis was completed across all measures holistically (which allowed for a reasonable sample size) and showed a survival curve of 97% persistence after one year, 87% persistence after two years, and 80% persistence after three years. Several measures were evaluated after year three but not enough to publish a persistence quantity. The researchers also created a rough estimate for EUL by extrapolating the failures in the first few years. They estimate an EUL of eight, though with “a wide band of uncertainty around this estimate.”

Five studies did not include primary data collection. These were generally program evaluations (see Table 2; lower section of the table). All five studies referenced EULs used in various utility RCx programs, with nine different program assumptions listed across the five publications. Those nine EUL values are shown in Table 3. The table also indicates whether the EUL is a uniform assumption across the entire program, or if different EULs are assumed for each major measure type. In those cases, the EUL shown in the table is the average for the program.

Table 3. Effective useful life values for nine programs stated in the publication
Utility EUL By measure?
PG&E 7 By measure
SCE 11 By measuree
SCG 14 By measure
SDG&E 10 By measure
PacifiCorp 7 Uniform
Xcel CO 7 Uniform 
Eff. Maine  5 Uniform 
Xcel MN 7 Uniform 
CA (DEER) 5-10 By measure

 

Though there are significantly more details in the measure-by-measure EULs than this table can convey, the table does indicate that most utilities consider RCx to have a lifetime in the five- to ten-year range, with seven years being particularly common. There is evidence from a few of the documents, though, that the estimate of seven years is simply proliferated from program to program by reference, as opposed to a number of independent estimates all arriving at seven years.

Literature Review Conclusions

We have not been able to develop specific measure-by-measure lifetimes based on the literature review alone, but this review did help us in understanding the persistence of RCx measures. We can draw the following conclusions based on this secondary research:

  • Most RCx programs assumed a significantly longer life for their measures than ComEd currently does.
  • When measured after two to three years, roughly 80% of RCx measures were still in operation.
  • Some programs put specific persistence strategies in place during implementation. These strategies included a bonus for persistence (Manitoba Hydro), incentives for commissioning, or incentives for training (Manitoba Hydro).
  • Most RCx program evaluations did not include any savings benefits for spillover, which could occur if a building operations team had gone through an RCx program and used the lessons learned in that program to implement other measures in the months and years followed.

ComEd Persistence Study Methodology 

Slipstream designed a field study to gather information on the persistence of savings from RCx measures. The field study involved a walk-through audit with direct observation of measures (either visually or via building automation interfaces). These audits followed an established protocol for a sample of buildings that had participated in ComEd’s Retro-commissioning program.

Sample Site Characteristics 

Our building sample frame consisted of 80 buildings that had participated in the Retro-commissioning program in Program Year PY2, PY3, and PY6. Within the 80-building dataset, 12 buildings were completed in PY2, 26 in PY3 and 42 in PY6. We divided this sample population into two groups: projects that were roughly six years old (PY2 and PY3) and projects that were roughly three years old (PY6). Since the numbers of participating buildings were relatively small in the early years of the RCx program, we combined PY2 and PY3 to have a reasonable number of building samples. For the remainder of this report, this combined category is referred to as PY3, and it is a conservative viewpoint for persistence. We looked at the two separate timeframes in order to analyze persistence in savings at two different points in time, potentially allowing for evaluation of changes in persistence over time.

Our goal was to complete our RCx persistence protocol at 38 buildings (from the original set of 80 buildings,) with the number of sample buildings selected for study equally distributed between PY3 and PY6. We established this target to gather data on a total 209 installed measure instances, based on the average of 5.5 installed measure instances per building according to ComEd’s program evaluation data.

We identified fifteen different measure categories that were most commonly implemented and accounted for at least 75% of total RCx energy savings. The measure categories are listed below: 

  • Airflow adjustment
  • Airflow adjustment (zone)
  • DCV
  • Duct sealing 
  • Economizer and OA control 
  • Filters
  • Lighting control 
  • Optimum start for AHU's
  • Plant control optimization 
  • Plant sequencing and isolation 
  • SAT and reset
  • Static pressure control 
  • System schedule 
  • Thermostat setting 
  • Other (includes measure instances with a negligible sample set, such as individual unit heater controls, fan staging modifications, etc.) 

We initially planned to randomize our building sample frame and contact buildings until we achieved our sample targets. However, recruiting building sites for the study proved to be challenging. Some common challenges we encountered were: 

  • Change in building ownership and/ or management: many commercial buildings have since been sold to a new company who was not aware of the RCx program and were not interested in participating in a follow-up study.
  • High turnover in facilities staff presented a similar challenge as the change in building ownership for conducting an RCx follow-up. We often had difficulty finding the right point of contact to talk to about the study
  • Some buildings had since had another implementation of RCx or MBCx. Although we planned to omit all of these sites originally, we included a couple of these sites where we were able to confirm that there was no overlap between the measures we were evaluating and the new implementation effort.
  • Some sites were currently going through major additions and renovations. The facilities teams could not spare time to participate in the study.
  • Some facilities personnel could not get approval from upper management to participate in the study due to data confidentiality reasons. Although we were open to signing non-disclosure agreements in a format that the management required, they were resistant to the idea.
  • The extremely cold winter weather in January 2018 that lasted about two weeks caused a few participants to cancel scheduled site visits. The facilities teams were in a pinch to maintain normal building operating conditions under the extreme weather situation and did not have the bandwidth to spare resources for our study.

To overcome this hurdle and enroll enough number of buildings for our study, we often made cold calls and asked to be connected with the facilities department, relied on contacts from the RCx service providers, leveraged ComEd outreach staff and account managers, and generally used any possible method at our disposal to get to the appropriate contact person at each site. These challenges led us to abandon the randomization method and simply attempt to contact every building in the building sample frame.

In the end, we completed the RCx persistence protocol at 28 building sites.

Field Work 

Field visits consisted of four main components: pre-site visit measure review, measure persistence review, staff interviews, and persistence evaluation.

Prior to each site visit, we thoroughly reviewed the original RCx report that was prepared by the service provider and submitted for post RCx verification. This allowed us to review all the measures that were implemented during RCx, document savings associated with each measure, and identify the system parameters that were modified to achieve the savings. We documented key data on the site visit instrument.

On average, we spent 2.5 hours per site to evaluate measure persistence. The majority of our time at the site was spent reviewing the measures installed via the BAS and assessing their persistence. We had help from both the service provider and the facilities staff responsible for energy/ BAS management at the site. Their knowledge of the facilities operation and RCx measures allowed us to document the measures accurately while minimizing the amount of time spent searching the BAS system for relevant data. The facilities staff were also comfortable navigating the intricacies of site-specific BAS configurations, which was extremely useful in collecting trended BAS data.

We documented measure persistence through one of four methods.

Measure trends: We reviewed specific data points for each measure that help us determine whether the measure persisted and to what extent. We evaluated data points consistent with trends analyzed during RCx. The length of trends varied from site to site, depending on BAS data trending availability and seasonal variations impacting the measure. For example, since most site visits were conducted in the fall 2017 – early winter 2018 timeframe, we reviewed data logs up to three months retroactively to evaluate cooling plant measures.

Control logic: At times we also resorted to reviewing the control programming in place to evaluate measure persistence. We encountered this scenario particularly with very old BAS systems without significant trending capability. The BAS program logic clearly indicated how the measure currently operates, and we verified this control sequence against the RCx verification reports. We also reviewed if any measure logics or limits were overwritten by site engineers, which would compromise persistence in savings.

Functional testing: In very rare instances, where both BAS trending data was not available and the BAS sequences were not readily accessible (or too ambiguous to verify concretely), we requested permission to conduct some live functional testing of measures. We made sure to test zones that weren’t heavily occupied at the time of testing and avoid zones with critical functions, such as labs and patient spaces. We changed measure set points via the BAS system to observe if the system responds as intended by the RCx measure. For example, to test static pressure reset, we changed the temperature of a zone to observe if static pressure modulates to meet the demands of the space. Although this method of testing gave us very good results to analyze persistence, we didn’t use it broadly due to practical constraints in the buildings.

Setting up new trends: In some sites, historical data trends were not available for one or all measures, we were able to set up new data trends during the site visit, and followed up to observe the trends once enough data was collected.

Our observations of persistence were guided by a set of a priori assumptions that we agreed to with project stakeholders, including ComEd, the RCx program implementers, and independent evaluators.

The other significant task completed at each site visit was an interview with facility management staff to document various factors impacting energy and measure performance.The interview questions covered four main categories:

  • General building information, including vintage, retrofits since RCx, floor area impacted by RCx, repeat participation in RCx or MBCx
  • Operations and maintenance, including energy management and BAS operations manager (internal vs. external), current personnel involvement and knowledge of RCx measures, any major changes in the building (occupancy, scheduling, functions, etc.,) the frequency of BAS adjustments, personnel continuing education (building operator certification, Local 399, etc.)
  • Capital investments, including system replacements or upgrades, the new control sequence for the upgraded equipment, BAS system age, BAS hardware components and software upgrades
  • Spillover, to check if measures that were recommended but not installed during RCx had been installed at a later point in time.

We completed 28 site visits during the study period and investigated 231 different measure instances. Thirteen sites were retro-commissioned in PY 3, and the remaining 15 sites were retro-commissioned in PY6. Of these total measure instances, we were able to evaluate the persistences of 167 measure instances across 15 different measure categories. We found 63 measure instances that were recommended during RCx but were not installed at that time, so the savings were not verified as part of this study. We reviewed these 63 instances for spillover savings but did not include them in our persistence quantification.

The 167 measure instances that we evaluated for persistence brought us close, but fell short, of our original target of evaluating 209 measure instances across 38 buildings. This was due to challenges with recruiting projects for our study as discussed above.

Data Analysis 

Following the field work, we compiled the data from the 28 sites and analyzed the data set. The data analysis had two major components:

1. Persistence calculations 

2. Correlation analysis 

For the purposes of analyzing confidence intervals for persistence, the study was treated as a two-stage sample, with facilities as the first sampling stage, and individual measure instances within a given facility as the second stage. This approach accounts for the possibility that observed persistence among measure instances installed in the same facility may be correlated.

We used data gathered from the interviews with facility managers for the correlation analysis. We analyzed the impacts of various factors potentially influencing building performance and their impacts on persistence via statistical analysis including simple linear regression.

Statistical computing software R was used in processing the data and generating the statistical results.

Results 

Results from our analysis include quantitative findings on persistence across programs and by measure category. Additionally, we investigated site characteristics and management approaches for correlations with high persistence of savings.

Persistence 

The analysis of persistence involved three steps: evaluating measure instance persistence at all sites and program years, weighting overall persistence by total verified energy savings for each measure, and evaluating persistence across various measure categories.

Persistence by measure instance 

The first step in our analysis was to evaluate all 167 observed measure instances across all sites and program years to identify overall persistence in savings from participating in the RCx program. From our analysis, we found that overall measure persistence based on measure count alone (not weighted by energy savings) was 58.1% (± 8.6%) across all program years. A persistence higher than 50% reflects that a measure is still within its useful life (see Persistence study methodologies). Results suggest that RCx persists at least this well.

Table 4. Persistence by measure instance
Period Persistence (# of instances weighted) Standard Error 90% Confidence Low 90% Confidence High
All  0.58 0.09 0.44 0.73
PY3 0.63 0.15 0.37 0.89
PY6 0.53 0.09 0.38 0.68

 

Further, we isolated study participants based on program years, and evaluated measure persistence in each program year group. As Table 5 shows, measures across PY3 yielded an average of 63% (± 15%) persistence, while PY6 yielded 52.8% (± 8.9%) persistence.

Persistence by energy savings 

More importantly, we also calculated persistence weighted by electric energy savings for each measure. We first calculated persistence for each measure instance and weighted these persistence values by the total verified energy savings for each measure based on savings values from the RCx verification reports (as written by service providers). By this metric, we found a 60.8% (± 8.9%) persistence in electric energy savings across the program, as shown in Table 5. 

Table 5. Overall persistence weighted by electric energy savings per measure
Period Persistence (electric energy savings weighted) Standard Error 90% Confidence Low 90% Confidence High 
All  0.61 0.09 0.46 0.76
PY3 0.76 0.09 0.61 0.92
PY6 0.49 0.11 0.30 0.68

 

By program year, PY3 and PY6 showed a 76.4% (± 9.1%) and 49% (± 11.2%) persistence, respectively. It was surprising to see persistence decrease for the more recent program year; this result is discussed in much more detail later.

Note that results for persistence presented in the remainder of this report reflect persistence weighted by energy savings in each measure instance.

Persistence by measure category 

We grouped measure instances into the fifteen different measure categories that we established prior to the study based on program data. The number of observed instances varied significantly across measure categories, and it should be noted that some measures were only observed a few times and so reported results have a high degree of uncertainty. The highest number of instances were observed in the Economizer and OA control and Plant control optimization categories, with 32 instances each.

Table 6 shows calculated energy savings persistence in each measure category. We observed the highest persistence in savings for filters and airflow adjustment measures. Lighting controls, optimum start for AHU’s and thermostat settings had the poorest persistence; these showed zero persistence in the data points we observed (though sample sizes for all three were very small). No observation was made in the duct sealing category, hence this category was excluded from the study. Due to the small number of sample instances for some measures, we only calculated the 90% confidence intervals for the measures having more than 14 sample instances in the following table.

Table 6. Persistence by measure category, with confidence intervals and number of instances
Measure Category Overall Persist. 90% CI PY3 # of Inst.  PY6 Perist. # of Inst.
Airflow adjustment 0.81   0.75 1 0.88 2
Airflow adjustment (zone) 0.17   2.00 1 0.000 1
DCV 0.29   0.66 2 0.00 1
Economizer and OA control 0.65 0.37-0.93 0.91 15 0.50 17
Filters 0.81   0.46 3 1.00 2
Lighting control  0.00   0.00 1 N/A 0
Optimum start for AHU's 0.00   N/A 0 0.00 3
Plant control optimization  0.61 0.35-0.87 0.61 20 0.61 12
Plant sequencing and isolation  0.72   0.93 5 0.08 3
SAT and reset  0.63   0.87 5 0.23 3
Static pressure control  0.63 0.35-0.89 0.76 8 0.58 15
System schedule  0.76 0.51-1.00 0.76 19 0.74 11
Thermostat setting  0.00   N/A 0 N/A 1
Other 0.63 0.28-0.97 1.00 5 0.58 9

 

We considered persistence in measure categories between PY3 and PY6. Measure persistence varied heavily in each measure between program years, but that is likely not an indication of changes to that measure over time; sampling error due to the different service providers and sites in the two program years would overwhelm any significant change in measure performance from one program year to the next.

Since such detailed measure categorization gave us a limited number of measure instances for statistical analysis, we subsequently re-categorized the measures into six broader measure categories:

  • Air distribution (includes pressure, temp resets, and thermostat setting)
  • Heating and cooling plant optimization and sequencing (including Pumps)
  • Ventilation (includes economizer, OA adjustment, DCV)
  • Scheduling 
  • Filters
  • General (all else) 

Table 7 shows calculated energy savings persistence in each new measure category. The results for Ventilation, Scheduling, Filters, and General categories remained similar to the previous analysis. The Air Distribution category, which now included airflow adjustment, AHU static pressure and temperature reset, and thermostat settings, had an overall 36% persistence rate, by far the lowest of any category. 

The Plant Optimization and Sequencing category, which combined the original Plant Control Optimization and Plant Sequencing and Isolation, yielded 62% persistence overall. 

Table 7. Persistence by broader measure category
Measure Category Overall Persist. 90% CI PY3 Persist. # of Inst.  PY6 Persist. # of Inst.
Air distribution  0.36 0.15-0.57 0.69 17 0.23 25
Plant optimization and sequencing  0.62 0.40-0.85 0.69 25 0.57 15
Ventilation  0.65 0.38-0.92 0.90 17 0.49 18
Scheduling 0.76 0.51-1.00 0.76 19 0.74 11
Filters 0.81 0.47-1.15 0.46 3 1.00 2
General 0.62 0.28-0.96 0.95 6 0.58 9

 

The overall persistence by measure category are shown graphically in Figure 1 and Figure 2, in order from measures with the highest energy savings (that we observed) to the lowest energy savings, based on detailed and then broader measure categorization.

Figure 1Figure 2

 

More details including the total electric savings (savings vs. persisted), persistence percentages for the overall program, PY3 and PY6 can be seen in Appendix 2 - Number of measure instances and savings in each measure category.

Factors Affecting Persistence 

We tested our dataset for several correlation factors that could potentially impact savings persistence. From the qualitative information that we gathered from site personnel interviews on building operation and maintenance, we identified key metrics against which to test correlations with energy savings persistence. Some factors showed a clear correlation with persistence; these are highlighted in green in Table 9.

Table 8. Correlation coefficients and p values
Factors Affecting Persistence Correlation Coefficient  P value
Year of retro-commissioning  -0.25 0.20
Building type  0.15 0.44
Building area -0.17 0.39
Major retrofit  -0.33 0.09
Energy management (internal/external) -0.05 0.80
BAS management (internal/external) 0.29 0.13
RCx training  0.34 0.08
Staff turnover 0.16 0.44
Age of the building  0.14 0.49
BAS upgrades 0.01 0.96
BAS hardware age  0.04 0.88
Major control repairs or upgrades 0.00 0.99
Operator's job title  -0.05 0.82
Service provider's continued involvement  0.10 0.60
Onsite evaluation  -0.02 0.92
Scale of project impact (in total cost savings) -0.36 0.06

 

In the remainder of this section, we explore those factors in bold in more depth. 

ComEd’s independent program evaluator had expressed interest in whether their evaluation of projects actually influenced persistence. As Table 9 shows, we found that impact of this factor is likely to be very low to non-existent, based on correlating the instances of on-site evaluation to our persistence observations.

Year of retro-commissioning 

We performed a correlation analysis between the year a building was retro-commissioned versus the persistence at the time of our site visit. The goal of this analysis was to check for degradation in savings over time. Program years 2-3 spanned 2010 and 2011, while program year 6 spanned 2013-2014 projects.

Figure 3

 

As shown in Figure 3, we found that 2013 participants had the highest savings persistence, while 2014 had the lowest persistence. This result was somewhat unexpected; we expected 2013 and 2014 participants to have the higher persistence compared to 2010 and 2011 data sample because sites retrocommissioned more recently would have less time for performance degradation to occur. To further understand the cause of the unexpected drop in 2014 persistence, we investigated this year further.

Further investigation of the Year 2014

We reviewed data from participating 2014 buildings at the site and measure levels and found that it is largely driven by two sites with very high savings but with lower than average persistence, 0.34 and 0.09. Low persistence in these sites impacted the overall 2014 persistence significantly, as seen in Figure 4 (the two aforementioned sites circled in red). These two poor performers account for 45% of total 2014 verified kWh savings.

Figure 4

It is worth noting that the Optimum start for AHUs measure was only implemented in 2014 buildings and yielded zero persistence across observed measure instances (with relatively high savings). The verified savings associated with this measure accounts for 896,007 kWh or 59% of total verified savings in one of the two poor performing sites. In one of these two sites, the site interview with the facilities personnel revealed that this measure was overridden very early after RCx, since the building could not maintain comfortable indoor thermal conditions with the recommended AHU start schedules. Mass wall construction and single-pane windows in this large commercial facility were not conducive to later AHU start times. The other program years do not include this measure.

The other site has a brand-new property management team, and the chief engineer had been at his position for only a few months at the time of our site visit. The team was not aware of the previous RCx effort or the measures that were installed. However, overriding of Optimum start for AHUs may be more of an outlier than a new property management team; we had a number of sites that showed shades of the latter situation (though not as severe).

So we can’t state definitively that these two cases are outliers, but for the sake of comparison if we consider them as such and exclude them from the data analysis, the average 2014 persistence becomes 61.8% (vs. 44.2%.) The updated PY6 persistence then would be 66.7% (± 11.1%.) Comparing to the PY3 persistence number of 76.4% (± 9.1%,) the two means are within one standard error. One possible explanation for the somewhat lower PY6 persistence number might be that organizations signed up for ComEd’s retro-commissining program in earlier years (PY3) were more proactive towards energy savings and had better energy management teams. Table 10 shows the updated overall persistence results.

Table 9. Overall persistence excluding two worst-performing buildings in 2014
Period Persistence (electric energy savings weighted) Standard Error 90% Confidence Low 90% Confidence High
All 0.72 0.07 0.60 0.84
PY3 0.76 0.09 0.61 0.92
PY6 0.67 0.11 0.48 0.85

 

In any case, our overall sample of data across all years suggests that performance may not slowly degrade over time in a more linear fashion, but may instead degrade mostly in the first few years after RCx.

Building type

The correlation between building type and energy savings persistence is shown as a box plot in Figure 5. We found that offices and hospitals may have somewhat higher persistence than other building types; both persisted at a rate above 75%. However, for some building types (industrial, lodging, other, office and industrial) we only have 1 sample building so those results may not be statistically significant.

Figure 5

Building area

Smaller buildings with less than 500,000 ft2 tended to have modestly higher persistence, as demonstrated in Figure 6. It is possible that facilities teams with less ground to cover are better informed about the control parameters and their buildings in general, than for larger facilities. Our interviews with facilities teams also suggested that larger buildings had more frequent hot/cold calls to deal with from occupants, which in turn more often led operators to overwrite BAS setpoints or sequences.

Some of the larger buildings that we observed also had many different types and units of HVAC equipment, as a result of different use areas in the large facility, or additions that had been put on to the facility over time. All of this different equipment necessitated some more individualized changes to control sequences as needed, as opposed to uniform control sequences across the building.

Figure 6

Major retrofit 

Our analysis for correlation between buildings with major retrofits (since it was constructed) and persistence reveals that buildings that have never had a major retrofit on average had a higher persistence than buildings that have had one or several retrofits, which were often completed in phases (Figure 7).

Figure 7

BAS and energy management 

We investigated the impact of a facility having internal energy or building automation management staff, versus facilities that outsourced all energy and controls (BAS) management through a regular service contract. It was found that persistence was more likely to be higher in facilities with a dedicated contractor who managed the BAS system (Figure 8).

Figure 8

A building automation service contract ensures that the system is constantly monitored and adjusted for optimum performance. In situations where comfort conditions are not met, the controls contractor may have more ability to address the underlying cause of system malfunction, rather than adopt a quick fix for the symptom (such as disabling sequences) as compared to the average internal operator. In our field studies, we did observe some highly skilled internal operators but also observed a number of operators that were quick to simply overwrite or disable more complex control sequences (i.e., the RCx measures), and not restore them back to optimal operating conditions, causing lower persistence.

We also tested whether internal vs. external energy management teams made an impact on savings persistence. This analysis yielded no significant correlation (Figure 9).

Figure 9

While most facilities personnel are very proficient in their understanding of building energy systems and its operations, the same is not true for BAS management. BAS software platforms are mostly proprietary and require familiarity with the system, as well as programming, for optimal operation. Given these challenges, we found that persistence is significantly impacted by BAS management method, but not significantly by energy management protocol.

RCx training 

In considering the impact of training, we found a trend toward higher persistence in sites where the facilities staff had received some RCx-related training immediately after RCx participation (Figure 10). This demonstrates that having facilities staff both participate in the RCx process and also gain knowledge on how to operate and maintain installed measures can potentially yield longer-term energy savings benefits.

Figure 10

Staff turnover 

We found staff turnover was one of the biggest challenges in our study, both in recruiting participants for our site visits and in understanding how the measures have been managed since RCx occurred.

We often found facilities staff who were part of the original RCx process to be more knowledgeable about the installed measures and were aware of the measure’s intent and outcome. Newer facilities staff were often not aware of the RCx measures and were not particularly invested in persistent savings. The majority of buildings we visited were managed by a person who had held the operator position for less than three years. In every one of those cases, they would not have been part of the RCx program.

Our analysis shows personnel who have been in their positions for more than six years tended to be able to limit the downside of performance degradation (Figure 11). However, the analysis did stop short of showing a very clear and strong correlation between staff turnover and savings persistence (Figure 12).

Figure 11Figure 12

Scale of the project 

Our analysis showed some correlation between the scale of the project (measured by total energy cost savings) and persistence. Smaller-scale projects had somewhat higher persistence. Because the scale of the RCx project is directly proportional to the building, this is likely primarily a confounding of building area (shown above in Figure 6) and scale of the RCx project.

Anecdotally, facilities with higher operating costs are also typically buildings that house critical functions that operate 24 x 7, such as hospitals, data centers, and labs. When operational adjustments are made after RCx, to meet comfort or humidity requirements, the facilities staff often are not able to adjust the system back to RCx recommended settings in these spaces. Several sites in our study included critical activities, and these sites typically were higher energy consumers.

Analysis of factors that did not demonstrate a clear correlation can be found in Appendix 4 - Factors that did not show a significant correlation to persistence.

Other Observations 

In addition to our focus on persistence in the program and the impacts on it, we made a number of other observations that are worth sharing with program staff and stakeholders.

  • We received overwhelmingly positive feedback about participation in the RCx program. When asked to rate the program’s influence on energy savings on a scale of 1-10 (1 being no influence to 10 being a very high positive influence), the participants gave the program an average rating of 7.7.
  • We found that about 64 % of participants have made energy efficiency improvements to the buildings that were not incentivized by a utility program. Common upgrades often quoted include adding VFD’s to pumps, replacing equipment such as chillers or AHU’s, and sometimes envelope improvements such as window replacements. About 71% of participants made upgrades to the building that were incentivized by a utility program—the most commonly implemented upgrade in this category was LED lighting replacement and additions/major renovations to existing facilities.
  • 71% of sites had facilities managers that are currently building operator certified. This certification is currently mandated by the RCx program.
  • The RCx program primarily focused on reducing electric energy savings, but some measures had a gas savings compenent associated with them as well. Only three measures out of all 167 evaluted measures were solely gas saving measures. The mean persistence for measures with any gas savings was 42% across all sites studied. This is lower than the 58% persistence we assesed for the electric saving, but the standard error is relatively larger due to a small sample size available. No significant conclusions on gas savings persistence can be made with this study.

Spillover 

In our study, we made a concerted effort to evaluate spillover in our observations of BAS and interviews with building operators. We attempted to look for spillover in two ways:

1. Investigate measures that were recommended during the RCx program but not implemented at the time the program was involved in the building. We looked to see if the measures had been implemented since but not tracked by the program.

2. We also tried to investigate any way the RCx program may have influenced operators to make additional changes, even though they were not recommended by the program, because they had learned or been influenced to by the program.

The second item turned out to be nearly impossible to evaluate due to the staff turnover challenge that we encountered, coupled with operator’s inability to recall the specifics and reasons behind any changes they made in the time shortly following the RCx program implementation. We unfortunately were not able to capture enough meaningful feedback to say anything in general about the program’s influence in this way. The first item was slightly easier to evaluate, though we caution that the results still may be missing a few responses. The results of that evaluation are in Table 11. Out of 39 such measures, we found 4 that had been implemented since RCx implementation; roughly a 10% success rate. This is out of a population of buildings where 167 total measures were initially executed during program implementation.

Table 10. Instances of spillover that we observed
Measure Category  Total Possible Instances Spillover Observed
Economizer and OA control  1 0
Plant control optimization 10 0
Plant sequencing and isolation 3 0
Static pressure control  4 3
System schedule  21 1

 

Additionally, our interview results showed that about 64 % of participants had made additional energy efficiency improvements to their buildings that were not incentivized (and therefore not tracked) by a utility program. Common upgrades often quoted include adding VFD’s to pumps, replacing equipment such as chillers or AHU’s, and sometimes envelope improvements such as window replacements. Again, the response was too limited to determine any significant correlation between these additional improvements and the RCx program.

Conclusions

As demonstrated in the literature review, various RCx program across the country assume highly varied measure life assumptions, from 2 – 20 years. Our study reviewed and evaluated persistence at a measure level, and program year level, in 28 participating sites. In addition to evaluating persistence, we also tested a number of building characteristics and management practices to understand contributing factors to high energy persistence.

The RCx program has been successful in identifying improvements in building operations and making functional adjustments to achieve those improvements, resulting in significant savings in electricity and natural gas. From our study of 28 sites that were retro-commissioned over a period of six years, we found the average persistence in that savings at the end of 2017 to be 61%, with a higher persistence from PY2- 3 projects (76%) than PY6 projects (49%). Results are inconclusive as to whether persistence between program years is significant; there was significant variability in persistence from site to site.

We also identified a number of persistence rates for specific types of measures; measure-specific results were detailed in Table 6.

With a mix of persistence rates from site to site, we also explored site-level factors that could potentially impact those persistence rates. We found a number of potential driving factors, but those with the strongest correlation to persistence were:

  • Building automation system management. Sites with an external contractor providing most of the changes and service for the building automation system were more likely to have higher persistence.
  • Staff turnover. The site where building management personnel had been in their jobs longer were more likely to have higher persistence.
  • Major retrofits. Whenever a site had undergone some type of significant retrofit in the time since retro-commissioning, persistence was more likely to have decreased.

We also evaluated spillover, but found evaluation to be very difficult. The only resulting conclusion we were able to reach was that 10% of measures that were recommended at the time of RCx, but not implemented at that time, have since been implemented.

Future work 

Perhaps the most useful future scope for this type of research would be an evaluation of an additional few years of measure persistence, now that we have observed high persistence over a six-year period. The program could also benefit from conducting a pilot of sites that implement recommendations from our findings, to assess if there is potential for energy performance follow-up in the years following RCx.

Program Recommendations

Consideration and inclusion of persistence factors in future program design efforts could potentially capture lost savings and make the program even more robust. We make recommendations for integrating these lessons into existing program practice, and also suggest some additions to the program, postimplementation. Finally, we suggest some ‘triggers’ to identify which projects to most strongly consider for these improvements.

Recommendations for implementation

  • We found that building operator training of any kind had a positive impact on persistence. Many operators received training through the local union, or as part of the Building Operator Certification program. The most conscientious operators seemed to make a direct connection between their training and the RCx program. Persistence may improve if the RCx program adds training elements.
  • In the Literature Review section, we documented how some other programs put persistence strategies in place during implementation—bringing persistence to the forefront of program implementation and conversations with the customer. One such program (Manitoba Hydro) even offered a bonus incentive for persistence, based on follow-up data collected.

Recommendations for follow-up with sites

It is clear that significant events that occur after RCx implementation, such as personnel turnover and retrofits, can have a very negative effect on persistence. It seems that persistence could be increased significantly with even minor follow-up activities, which could be applied to all sites or just to those that exhibit certain triggers (see next section for triggers). Those follow-up activities include:

  • Consider brief re-visits to these sites, in which the service providers essentially do quick checks of all the implemented measures, similar to the site visits we made in this study. The goal of these visits would not be to look for new measures: there wouldn’t need to be any calculations, detailed reports, or even functional testing. It’s not a full verification; just an inexpensive check-up. Our visits, for example, took only about 4 hours of staff time, generally split between 90 minutes of preparation and 150 minutes on-site. If many measures are found to have not persisted, they could potentially be reinstated wherever there was not a strong reason for them being undone in the first place (which was often true in our experience). When follow-up changes are needed, a bit more effort (and associated expense) can then be put forth by the service provider and controls contractor to re-implement measures. But even that additional effort should be much less costly than the original program, because all the needed data points and controllers will have been put in place during RCx; they just need to be reconnected or reinstated. In some instances, the original sequences are even still in place; they have just been disabled with a simple toggle.
  • Some type of “Monitoring-based Cx (MBCx) Lite” approach seems appropriate to attempt. The current MBCx program is likely to have much better persistence than RCx because measures are tracked over time. However, many buildings are not able to follow MBCx due to BAS compatibility, IT or data security concerns in integration, or simply cost. Those sites that cannot be connected via MBCx could still be tracked in a simpler fashion. Many projects have significant enough savings to track via utility bills or main-meter readings (where chilled or hot water, or steam are metered, for example). This type of MBCx Lite would require some highlevel analysis of future energy-efficiency measures that get installed at the building.

This brings us back to one of the secondary objectives of the study, to understand the potential to go back to past participants with low persistence and conduct RCx again. Premise specific findings from this study can be shared with individual buildings and its service provider, to motivate them to re-evaluate original RCx measures. This is particularly beneficial to buildings with lower overall persistence since they can re-gain savings that have since been lost. Data from individual buildings will not be shared either in this report or through any other means to protect data anonymity.

But generally speaking, in the three- to six-year timeframe following RCx, there is considerable variation between sites with high persistence across most measures, and sites with at least some major degradation. The results of the study, therefore, would not suggest that broad re-implementation of RCx should be conducted in the three- to six-year timeframe. They would suggest that there is value in conducting RCx again at sites with significant degradation in measure performance, where those can be identified. Those sites could be identified through analytical follow-up activities across all sites, such as the two suggested above. If a more targeted approach is needed, sites could potentially be identified for follow-up through outreach conducted with the knowledge of certain triggers, described in the next section.

Triggers for elevated support

Through all of the program recommendations above, certain triggers could be used to focus efforts, improving program efficiency. The following triggers could be used to identify sites that are more likely to yield lower persistence, and therefore may benefit from follow-up activities.

Personnel turnover. Building turnover was one of the factors that correlated most clearly to persistence. Outreach activities should identify when key personnel who were present for RCx are no longer operating RCx sites; when that occurs one of the follow-up activities above may become more worthwhile.

Major retrofits. When significant retrofit activity followed RCx, persistence suffered. Outreach discussions could identify when building retrofit, significant HVAC equipment retrofit, or BAS hardware or software retrofit are occurring.

Internal BAS management. We found that persistence was more likely to decrease when the BAS was managed primarily internally, without substantial intervention from an outside contractor. Though not true everywhere, sites with this approach to managing their BAS could be watched more closely.

Large projects. Large projects (those with the most energy savings) showed some slight correlation to lower persistence in the data. More importantly, such projects simply represent a larger share of the program’s impact, and therefore are worth more effort to ensure persistence.

References

Methodology

Friedman 2011. Annex 47 Report 3: Commissioning Cost-Benefit and Persistence of Savings. National Institute of Standards and Technology, U.S. Department of Commerce, Washington, DC.

Hall 2006. California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements for Evaluation Professionals. State of California Public Utilities Commission, San Francisco, CA.

Nexus Market Research 2008. Residential Lighting Measure Life Study. New England Residential Lighting Program Sponsors.

Skumatz 2009. Lessons Learned and Next Steps in Energy Efficiency Measurement and Attribution: Energy Savings, Net to Gross, Non-energy Benefits, and Persistence of Energy Efficiency Behavior. California Institute for Energy and Environment, Berkeley, CA.

Violette 2013. The Uniform Methods Project: Methods for Determining Energy Efficiency Savings for Specific Measures. Chapter 13: Assessing Persistence and Other Evaluation Issues Cross-cutting Protocols. National Renewable Energy Laboratory, Golden, CO.

Measured Persistence

Bourassa 2004. An Evaluation of Savings and Measure Persistence from Retrocommissioning of Large Commercial Buildings. Conference Proceedings from the 2004 ACEEE Summer Study on Energy Efficiency in Buildings, Pacific Grove, CA.

Eardley 2007. Persistence Tracking in a Retro-commissioning Program. Conference Proceedings from the National Conference on Building Commissioning.

Mills 2011. Building commissioning: a golden opportunity for reducing energy costs and greenhouse gas emissions in the United States. Energy Efficiency, 4:2, p. 145-173.

Peterson 2005. Evaluation of Retrocommissioning Results After Four Years: A Case Study. Conference Proceedings from the National Conference on Building Commissioning.

Potter 2002. Investigation of the Persistence of New Building Commissioning. California Energy Commission Public Interest Energy Research Program, Lawrence Berkeley National Laboratory, Berkeley, CA.

Roberts 2010. Do Savings from Retrocommissioning Last? Results from an Effective Useful Life Study. Conference Proceedings from the 2010 ACEEE Summer Study on Energy Efficiency in Buildings, Pacific Grove, CA.

Selch 2005. Recommissioning Energy Savings Persistence. Conference Proceedings from the National Conference on Building Commissioning.

Toole 2011. The Persistence of Retro-commissioning Savings in Ten University Buildings. Texas A&M University.

Evaluations and EUL Assumptions

Cadmus 2010. PacifiCorp Recommissioning 2007-2008 Utah Program Evaluation. PacifiCorp, Portland, OR.

Cadmus 2012. Impact and Process Evaluation of Efficiency Maine Trust’s Retro-commissioning Pilot Program. Efficiency Maine Trust, Augusta, ME.

DEER 2016. Database for Energy Efficiency Resources. California Public Utilities Commission. http://www.deeresources.com/

SBW 2010. Final Report 2006-08 Retro-commissioning Impact Evaluation. California Public Utilities Commission, San Francisco, CA.

Tetra Tech 2010. Xcel Energy Comprehensive Process and Impact Evaluation of the Recommissioning Program – Colorado. Tetra Tech, Madison, WI.