Forecasting is like driving a car blindfolded while following directions given by someone who is looking out of the back window. — Anonymous
Utility regulators beware: Not all forecasts are objective. Some are normative or biased, while others are based on science. When making important decisions, regulators must frequently choose between competing forecasts submitted by parties with varying agendas.
With potentially billions of dollars at stake, regulators need to reconcile the “forecast” discrepancies. Just as important but often overlooked, regulators also need to know the range of plausible forecasts and the risks associated with accepting one forecast over others. The risks triggered by uncertainty can play an important role in regulatory decisions.
Much of the push for a particular decision, whether for long-term planning purposes, merger proposals, determining future utility rates or other matters, comes from interest groups.
Regulators should receive their forecasts, which are critical for decision-making, with a grain of salt. They should ask if the forecasts are self-serving or are they legitimate and reflect objective analysis? Gaming by different stakeholders can present regulators with biased forecasts, which would require special regulatory-staff expertise to uncover.
Hedging Under Uncertainty
Often ignored, regulators should hedge their decisions to account for the inherent uncertainty associated with forecasting the future. A rational decision-maker would tend to respond to future unknowns by exercising caution in committing to a major action today.
Regulators therefore should require utilities and other parties to submit a reasonable range of forecasts to justify their positions. Basing a large investment or other major decision solely on the “best guess” forecast, or the future deemed most likely to occur, can result in substantially higher costs relative to the best action determined ex post facto with actual outcomes. In other words, an avoidable risky decision is more likely when based only on information provided by a “best guess” forecast without considering other possible futures and their implications for the right decision.
A range of forecasts or scenarios can help regulators quantify and evaluate the risks associated with individual decisions, related to electric-generation planning, energy efficiency initiatives or other actions, then judge whether the risks are intolerable. Uncertainty requires regulators and utilities to ask if the possible maximum losses from a particular decision are large enough to disqualify that decision from further consideration?
I use the term “forecast” to encompass both 1.) the future outcome that is most likely to occur (i.e., the “best guess” or single-point forecast) and 2.) a future outcome that is less likely to occur based on an alternative set of assumptions like economic conditions, the price of electricity, the price of substitutes for utility electricity, and the economics of renewable energy.
Some analysts refer to “best guess” forecasts as reference forecasts when they reflect the future with the highest probability of occurrence. The forecast is based on a set of events the forecaster expects will occur or considers more likely to occur than other events. If one has to choose a single forecast with a bet of $100 on the line, what would it be? It would presumably be the “best guess” forecast since the payoff would go to the person whose forecast lies closest to the actual outcome.
The regulator makes choices by using forecasts provided by utility stakeholders. First, it could approve the utility action based on the single-point price forecast; for example, the “best guess” demand growth of electricity 4% per annum, so the decision is contingent only on this forecast. This is a valid decision, however, only when 1.) the regulator places a high degree of confidence in single-point forecasts, and 2.) the consequences of incorrectly forecasting demand within a large range are minimal. For example, the preferred decision does not depend on whether demand growth is 2% or 4%. Otherwise, the regulator lacks access to valuable information to decide.
This situation is analogous to a person choosing a financial asset with the highest expected return, say, stock in a high-tech company, without considering its risk relative to other assets.
Most people would decide not to allocate all their investments to this high-return, high-risk asset. They would tend to diversify their investment portfolios to balance the tradeoff between return and risk. For financial assets, diversification implies an objective other than maximizing expected return or minimizing risk. Diversification reflects managing risk at a cost acceptable to the decision-maker given the degree and nature of their risk adversity.
Modern portfolio theory considers the inherent risk in various financial and physical assets and develops methods for aggregating investments to maximize the tradeoff between risk and return. In a different context, selecting a specific generation technology, or group of technologies, may stem from its lower risks relative to other technologies, even if the other technologies have lower expected levelized costs.
Using Different Forecasts
In our above example, as an alternative, the regulator could approve the utility action based on a range of demand-growth forecasts. It could, for example, review several forecasts from credible sources to select high, medium and low forecasts that represent reasonable demand-growth possibilities.
The evidence might show that demand-growth forecasts within a certain range result in the same preferred decision (e.g., expand generating capacity by a certain level by the year 2035). This sensitivity analysis makes the regulator more confident that the action taken will carry little risk unless it assigns a non-trivial probability to demand growth beyond the selected range. (The risk would be the opportunity cost of making a particular decision when another decision would have produced a better outcome after the fact.) Analysts consider such actions to be robust under a wide range of conditions. Robustness means that regulators would require less precision from a “best guess” forecast.
The regulator could approve the utility action after considering the cost of making the wrong decision based on erroneous demand forecasts (i.e., the loss function). The building of a generating facility based on demand growth of 5%, for example, could cost the utility an additional $100 million a year, compared with building the facility when the actual demand growth turns out to be 3%.
The regulator might want the utility to “hedge” its plan to moderate the cost (i.e., loss) from mis-forecasting demand growth. One idea is for the regulator to instruct the utility to take a wait-and-see approach as it accumulates more information to improve its forecasting accuracy before committing to a decision. To the extent that waiting reduces demand-growth uncertainty, the utility may reap an “option value” from an investment delay stemming from this uncertainty.
Loss Function
Rational risk-averse decision makers, implicitly if not explicitly, apply what is called a “loss function.” This function calculates the cost of a decision conditioned on a single forecast or range of forecasts that turn out to be wrong. Assume the decision to build a new gas-fired generating plant is contingent on the natural gas price being in the range of $3 to $5.
If the actual price is $7, the utility’s revenue requirements would be $500 million lower if it chose to build a solar facility instead. The $500 million represents a loss from relying on the wrong forecasts, which is inevitable when dealing with something as dynamic and unpredictable as demand growth, natural gas prices and other factors affecting the optimal decision.
The above example has a parallel to the current climate-change debate. Studies have shown that catastrophic consequences can follow if we do not take actions today to reduce greenhouse gases, but these consequences are highly uncertain, so much so that scientists cannot assign probabilities to their likelihood.
We may, therefore, spend money today to avoid an outcome that may never occur. The question is: What should we do today? The same question applies when an event is unlikely to occur but will cause a catastrophic outcome if it does. A society, group or individual that is risk-averse would tend to spend something today, for example buying insurance, to mitigate possible financial consequences in the future.
Distorted Incentives?
Although less guilty than in the past, utilities, in my observation, place excessive reliance on “best guess” forecasts to justify major decisions and fail to include a loss function in their forecasting exercise. One question still lingers: Does this problem reflect flawed decision-making, or do utilities and regulators deliberately produce and approve forecasts with overblown sureness and absent information on the negative consequences of erroneous forecasts?
The latter reason could be to buttress a particular, politically palatable action or, in some other way, advantageous to a utility or the regulator. One has to wonder.
Kenneth W. Costello is a regulatory economist and independent consultant who resides in Santa Fe, N.M.
