North American electric utilities are generally prepared for minor problems with their systems for conducting real-time assessments, but new tools may be needed to deal with larger failures, according to a new joint report from FERC, NERC and the regional entities issued on Thursday.

The report is based on a review conducted in 2019 by a team consisting of representatives from FERC and the ERO Enterprise. The team conducted on-site discussions with nine participating reliability coordinators and transmission operators seeking information on their utilities’ performance during “events where the participants or its [RC/TOP] experienced a loss or degradation of real-time data or of primary tools used to perform real-time assessments.”

Real-time assessments are required by NERC reliability standards TOP-001-5 (Transmission operations) requirement R13 and IRO-008-2 (Reliability coordinator operational analyses and real-time assessments) requirement R4. These requirements, which the report refers to collectively as the “real-time analysis requirements,” mandate that TOPs and RCs perform an assessment at least once every half hour “to ensure that instability, uncontrolled separation or cascading outages that could adversely impact the reliability of the interconnection will not occur.”

NERC developed the requirements in response to its joint report with FERC on the outages in Arizona and Southern California in September 2011. That document recommended that TOPs “ensure that their real-time tools are adequate, operational and run frequently enough to provide … the situational awareness necessary” to plan for potential issues and smoothly operate the system. NERC was also inspired by the 2003 Northeast Blackout, which was primarily caused by “failure to assess and understand the real-time risks to the grid,” according to investigators.

Focus on Improvements, not Blame

The joint review team emphasized that their goal “was not a compliance review of prior activities” but to understand the strategies and techniques that utilities use to meet the real-time assessment requirements. The report covers seven key areas:

• real-time assessment tools under normal operating conditions;
• real-time data and data quality;
• managing the loss of real-time data;
• alternative real-time assessment and study tools;
• model management;
• control center hardware configuration; and
• major system upgrades and vendor changes.

For each topic, the authors provided an overview of participating entities’ activities and recommended improvements, along with any beneficial practices they observed that other utilities might consider emulating.

The tools used for real-time assessment under normal conditions were of interest to the team because these assessments “are complex processes that involve a variety of computer tools and control room displays,” and hence a wide range of processes might be available to meet the real-time assessment requirements. The authors found that all participating utilities consider a state estimator and real-time contingency analysis (RTCA) “the most essential tools” for performing assessments, but they did not clearly define what other tools and data points might be needed.

This could represent an important oversight, the report said, because entities may not be prepared for out-of-the-ordinary situations such as an outage of the energy management system (EMS) or changes to the grid’s performance from the introduction of new generation resources. Authors recommended that utilities periodically review their tool suite to ensure they can complete real-time assessments within the required 30-minute span, while reviewing trends that might require new tools or updates to existing ones.

Utilities’ systems for capturing and processing real-time data are also essential to successfully performing assessments, and the team found that “all participants have processes for identifying problems with … individual real-time data points and … correcting the errors.” However, several did not have tools for aggregating errors, which could help utilities identify fundamental issues with data collection that might lead to flaws in the real-time assessment. The report recommended that entities “develop procedures and metrics that provide … the means to measure the degree to which data accuracy affects real-time assessments.”

Entities must also be able to cope with the loss of real-time data, which represents a leading cause of EMS issues and failure to conduct on-time assessments: According to the report, 172 of 521 EMS failures between October 2013 and April 2019 were caused by communication issues, including equipment failures.

The authors found that none of the participants had “feasible internal solutions for the complete loss of real-time data over an extended period,” despite acknowledging “the benefit of redundant communications to minimize the loss of real-time data.” In response, the report recommends that utilities develop criteria to identify when real-time data quality has degraded to the point that it is no longer useful while collecting and using redundant data sources that can be used to continue the real-time assessment process when the primary data source fails.

The team also suggested that utilities invest in alternate real-time assessment processes that could be done without the use of primary tools. This could include using an offline alternative to RTCA and pursuing backup communication processes to obtain the necessary data. Models and modeling processes also need to be updated and enhanced, with most participants identifying this as the biggest challenge with ensuring quality assessments.

Design and construction of control centers can also contribute to real-time assessment issues in unexpected ways. The report found that “single points of failure in the architecture of control center hardware or software represented a common cause of real-time assessment-related events,” whether the single point of failure was human, mechanical or software-related. Recommendations in the report include ensuring redundancy of all information paths and actively tracking down sources of personnel errors.

Finally, system upgrades can lead to difficulties with real-time assessments through disrupting information-gathering processes, either at the time of installation or even on an ongoing basis through unforeseen incompatibilities. The authors recommended testing EMS upgrades on an incremental basis on offline systems before moving to production systems in order to work out any kinks, while coordinating these changes with operations to take place during planned outages of generation or transmission facilities.