Last year’s GridEx VII security exercise demonstrated the importance of communications in maintaining grid reliability along with the need for continued discussions between industry and government on prioritization of loads during system restoration, NERC said in its report on the exercise released this week. 

The Electricity Information Sharing and Analysis Center (E-ISAC) conducted GridEx VII from Nov. 14 to 16, 2023. The exercise comprised a distributed play portion, held during the first two days and involving more than 15,000 individuals from 252 participating organizations. Additionally, an executive tabletop was held on the third day with about 230 attendees from 75 organizations, including electric utilities; U.S. and Canadian government agencies and law enforcement; and representatives from the oil and natural gas, telecommunications, finance and nuclear industries. 

GridEx VII marked the second exercise in a row to see a decline in organizations participating in the distributed play from the previous event: 293 organizations took part in the distributed play in 2021, and 526 in 2019. (See NERC ‘Very Happy’ With GridEx VII Participation.) 

Of the organizations in the distributed play, 174 represented electricity asset owners or operators; 55 were government or “other”; 17 were reliability coordinators; and six represented the regional entities. All categories of participants were up or steady from last year except for government/other; 105 groups from this category took part in GridEx VI in 2021 

NERC acknowledged the change in participation in its report while observing that the number of individuals taking part seemed to have increased significantly from the 3,000 estimated for GridEx VI. As in previous years, the number of individuals taking part was estimated based on responses in the after-action report. 

The ERO attributed the participant decrease to the continuing impacts of the COVID-19 pandemic, as well as the requirement — implemented for GridEx VI — that participating entities must be E-ISAC members. NERC also noted that participating organizations may have coordinated their exercise play with unregistered entities, whose participation the E-ISAC could not track. 

Organizations participating in the executive tabletop also fell from 88 participants in GridEx VI; however, the 230 individuals attending represented an increase from nearly 200 in the last exercise. 

Cyber, Physical Attacks Hit Hard

The distributed play scenario was developed by the E-ISAC and customized by participants, so details of the exercise varied between entities. However, the outlines were shared by all. 

The game consisted of five “moves.” It actually began over the week prior to the exercise, with Move 0 consisting of threats injected according to the “organizational objectives” of participants. Moves 1 to 4 comprised the core exercise over Nov. 14-15: 

• • 1: Cyberattacks and ransomware hit utilities’ communication software, internal information technology networks and third-party systems that operate the electricity markets. Additionally, disruptions to natural gas supply reduce generation capacity. 
  • 2: Attackers launch a coordinated physical assault against multiple substations, with gunfire targeting critical transformer components. A social media misinformation campaign and further cyberattacks hamper utilities’ responses. 
  • 3: As recovery gets underway, further attacks occur at telecommunications facilities. Protesters, frustrated by the ongoing power outages, begin to harass utility personnel. Attackers detonate explosives at equipment storage and staging areas, damaging equipment needed to restore service. 
  • 4: The game jumps forward a week after the attacks, and players consider long-term recovery challenges. Issues such as fuel and equipment shortages were highlighted, with entities having “to rely on their current inventories.” 

NERC developed a set of recommendations from after-action surveys, feedback during exercise design and other engagement data. 

First, NERC suggested that electric utilities continue to engage proactively with nonfederal government partners on emergency response plans. The report authors mentioned feedback from one planner who normally coordinates with county emergency managers but “realized [during GridEx VII that] it was not feasible to communicate individually with each county … in an incident that spanned many counties.” 

NERC also noted the trend of declining participation by government entities. Observing that incident response “will likely require involvement from government partners at all levels,” the ERO urged municipal and state governments to step up participation. 

The report also called on utilities to improve their communication and response measures in light of changes to work habits caused by the COVID-19 pandemic. Because gathering all responders together into a single room is not as feasible as it once was, it is important that utilities update their plans to account for a more distributed workforce. 

NERC highlighted a comment from one planner that the simulated public unrest rendered the location intended for an in-person response inaccessible. The planners’ organization decided that a secondary location must be identified and added to emergency response plans in the future. 

Additional recommendations related to communication of technical information across critical stakeholders, along with the E-ISAC’s support for organizations of varying sizes and levels of experience. Participants provided positive feedback on the inclusion of Move 4 and its focus on long-term consequences of the previous days’ events. 

Communications Struggle in Executive Tabletop

The executive tabletop also comprised four “acts,” with facilitators leading participants “through discussions designed to simulate the communication and coordination during a real event.” 

• • 1: A cyberattack compromises utilities’ inter-control center communications protocol (ICCP) software, through which grid operators receive data from transmission and generation facilities. Operators had to use alternate and manual methods and “suspend the electricity market systems that automatically dispatch and price generation.” Voice and data communications networks fail across a large swath of the country as well. 
  • 2: Coordinated cyber and physical attacks damage transformers and other equipment at substations in Louisiana and Texas. This leads to power outages at natural gas hubs. 
  • 3: Cyberattackers compromise and deface MISO’s website, demanding ransom. Backup systems are corrupted, and critical IT staff members cannot be reached. 
  • 4: One month later, ICCP telemetry is mostly restored, but MISO’s electricity market systems still are suspended, damaged substation equipment is not yet fixed and power has not been restored to natural gas facilities. 

Recommendations from the tabletop included evaluating technology and processes to increase ICCP communication resilience. NERC emphasized that ICCP systems already are “highly reliable, supported by layers of redundant infrastructure and cybersecurity protections.” But the ERO said the exercise prompted participants to ask if the systems are adequately protected against certain vulnerabilities and if alternative measures would help secure the system. 

NERC also suggested the industry study communication alternatives between grid operators, which could be needed if automated telemetry becomes unavailable or compromised. In addition, the ERO said industry and government should discuss whether utilities’ established restoration procedures conflict with government priorities during sustained, complex outages. Finally, NERC urged the industry to evaluate how to manage the reliability impacts of extended market system or data unavailability. 

“Today’s threat landscape is dynamic, presenting challenges that are increasingly difficult to detect and protect against,” Manny Cancel, senior vice president of NERC and CEO of the E-ISAC, said in a statement. “The scenario created for GridEx VII reflected this by testing the collective ability of industry, government and cross-sector partners to restore the grid under the most extreme circumstances. … I am encouraged that several participants have already begun to implement some of the recommendations in their organizations.”