By Rich Heidorn Jr.

WASHINGTON — The limits of grid exercises and simulation tools and the need to prepare for a successful cyberattack were recurrent themes at the National Academies’ Committee on the Future of Electric Power in the U.S. daylong conference on computing, communications and cyber resilience.

Observations from the Nov. 1 conference — which featured officials from FERC, NERC, the Department of Homeland Security and Department of Energy — will be in a report by the committee to Congress and DOE, scheduled for release in Fall 2020.

The project was ordered by Congress as part of the 2018 DOE appropriations bill. It directed the National Academies of Science, Engineering, and Medicine to appoint an ad hoc committee of experts to “conduct an evaluation of the expected medium- and long-term evolution of the grid [with a] focus on developments that include the emergence of new technologies, planning and operating techniques, grid architecture, and business models.”

Here are some of the highlights of the day.

Embracing Redundancy

Committee Chair Granger Morgan, professor of engineering at Carnegie Mellon University, asked panelists what recommendations they would like to see in the upcoming report.

“If no one else is jumping on the grenade, I will,” said Scott Aaronson, executive director of security and business continuity for the Edison Electric Institute. “I will continue to beat the drum of resilience.”

Aaronson, part of a panel on moving from a culture of compliance to one of security, decried what he called the “Whack-a-Mole” approach to grid threats, saying the industry should set a goal of “consequence management” that takes advantage of the grid’s inherent redundancy and resilience. Whether it’s “EMP or GMD or cyber or physical or storms or zombies, there’s always going to be a new threat,” he said.

He cited the 2013 sniper attack on Pacific Gas & Electric’s Metcalf substation, in which 17 transformers were damaged at a cost of $15 million. “You know what was cool about that? The lights didn’t blink in San Francisco or Silicon Valley. Why? Because of redundancy.”

“NAS could provide some leadership about how we engineer — on top of this extraordinary machine — more resilient capabilities,” Aaronson said.

Morgan noted the Academies did a report on the resilience of the transmission and distribution system in 2017. What’s new to say? he asked.

Joe McClelland, director of FERC’s Office of Energy Infrastructure Security, suggested the academies could “narrow the focus” to identify what capabilities are required to ensure the continuity of mission critical functions.

“Is it skeletal service, to say, large urban areas? Is it off-site power to a nuclear power plant? There are not very many facilities, but what is the model for a sustainable power source for these facilities — self-sufficient and sustainable — that could dissuade a potential attack by a sophisticated adversary?”

Sobering Reading

McClelland gave his panelists a homework assignment: the February 2017 report of the Defense Science Board Task Force on Cyber Deterrence.

The report concluded Russia and China “have a significant and growing ability to hold U.S. critical infrastructure at risk via cyber attack, and an increasing potential to also use cyber to thwart U.S. military responses to any such attacks.

“This emerging situation threatens to place the United States in an untenable strategic position,” the report continued. “Although progress is being made to reduce the pervasive cyber vulnerabilities of U.S. critical infrastructure, the unfortunate reality is that for at least the next decade, the offensive cyber capabilities of our most capable adversaries are likely to far exceed the United States’ ability to defend key critical infrastructures. The U.S. military itself has a deep and extensive dependence on information technology as well, creating a massive attack surface.”

“That’s sobering,” said McClelland.

The report also called for “additional cost recovery mechanisms” so critical infrastructure owners can invest in resilience that supports U.S. military capabilities.

Vendors’ Roles, Responsibilities

Brian Harrell, assistant director of DHS’s Cybersecurity and Infrastructure Security Agency (CISA), said technology vendors are “part of the solution” and should not be shunned from industry cyber discussions for fear “they just want to sell us a bunch of stuff.”

“I think this industry … is a little apprehensive to bring vendors into the conversation,” he said. “I will say in your time of need, when things go bump in the night, you will be reaching out to your vendor. And so, let’s ensure the vendors are part of the conversation. … We need to build security in from the beginning and not bolt it onto the rear because that is expanding the threat exposure for us.”

Electric Power Research Institute (EPRI) CEO Michael Howard questioned whether software vendors should be held liable for security vulnerabilities in their products.

“In the rush to market with many products, designers will use software languages like C++ with many known vulnerabilities. They will copy sub-routines that also have many known vulnerabilities,” he said. “Should there be regulation so that if these vulnerabilities are then sold and [this] results in a breach — because all the bad actors know what these vulnerabilities are, and they can be prevented with some of the latest software languages — should there be regulation that says if you do this and you rush to market that you will be liable for that?”

In 2016, Taiwan-based Asus agreed to independent audits for 20 years to settle a Federal Trade Commission complaint over a security flaw that allowed hackers to take control of almost 13,000 home routers. Although Asus claimed the routers would “protect computers from any unauthorized access, hacking and virus attacks,” the FTC said it found a “pervasive security bug” in the router would allow an attacker to disable security settings remotely.

Johns Hopkins University computer science professor Yair Amir responded with a cautionary note. “In the cloud domain, there’s a lot of very good use of open source [software] … It’s very effective. If you regulate against it, maybe we lose something.”

Morgan said although Howard was referring to software sold to utilities, “I think the problem exists in spades in the IoT [Internet of Things] space.”

“I don’t even know who would play the role [of regulator],” he said, dismissing the Consumer Product Safety Commission. “They’re totally ineffectual in a lot of other places. They’re not going to be out front, cutting edge, on this,” he said.

“There are several layers of equipment we’re talking about and not all of them are covered by the same regulations,” noted Washington State University Professor in Power Anjan Bose. While relays on the bulk power system are covered by critical infrastructure protection (CIP) standards, “once you start going down the chain into the distribution system … I’m not sure the CIP compliance covers anything, especially if it’s on the other side of the meter.”

“It’s the grid edge things that are now having to send a lot of data into the control center,” he added. “So … the threat surface is increasing.”

The National Institute of Standards and Technology’s (NIST) Information Technology Laboratory recently took comments on a draft discussion paper seeking feedback to identify core cybersecurity capabilities important for IoT devices.

Kevin Stine, chief of the lab’s Applied Cybersecurity division, said feedback was “overall very positive. We hope to move forward with baseline recommendations in the next quarter or so.”

Eliminate Financial Penalties?

Marc Child, chair of the NERC Critical Infrastructure Protection Committee, said he’d like to see an end to the constant churn of standards development.

“I want my [computer science] engineers back,” said Child, information security program manager for Great River Energy. “They have been distracted by spreadsheet land for a decade. I need them working on cutting-edge technology. I want them to go out and buy effective technology, not compliant technology — there is a big difference.
I want them looking at software-defined networks. I want them looking at decoy networks.”

Child said the CIP standards are “a good baseline that covers 75% of the problem. It will raise all of our boats. But I’d like to challenge them to cap the efforts. Any new threats are going to be incremental and could be addressed outside of mandatory standards. I’m going to say something controversial here … I would like to propose we reduce or remove the financial penalties associated with noncompliance. We need a culture of cooperation, and in so doing, we can change the auditor and utility dynamic to one of a shared mission. I want the auditor … on my side of the table.”

Pondering Manual Operations

EEI’s Aaronson said the industry must be prepared for “the inevitability of impact” because “standards simply can’t keep up” with new threats.

He noted grid operators in Ukraine resorted to manual operations to restore power after suspected Russian hackers took remote control of utilities’ SCADA system and cut off service to about 220,000 customers for a few hours in 2016. (See How a ‘Phantom Mouse’ and Weaponized Excel Files Brought Down Ukraine’s Grid.)

“Do we have that capability here in North America? Sort of,” he said. “And that’s not a good answer for chief executives. So, we are beginning to develop the capacity for supplemental operating strategies. I like to call it the MacGyver project. How do we hold the grid together with bubble gum and duct tape?”

“I think the audit regime, and I think [FERC] and state commissions … are starting to realize [the limitations of] check-the-box exercise[s]. ‘Alright, I’ll do x, y and z — I’m secure.’ No, you telegraphed your defenses and you’re complacent,” he said. “… We’re not going to get there overnight, but I think the tide has shifted just a bit to acknowledge the limitations of explicit … binary standards we see today.”

He called for efforts like the Spare Transformer Equipment Program (STEP) and the nuclear industry’s Pooled Inventory Management system (PIM). “What can the electric industry do to mimic that [database] of assets we might need when a bad day comes?” he asked.

David Batz, EEI’s senior director of cyber and infrastructure security, also cited STEP as an example of the efforts industry should pursue. “Let’s broaden the aperture and think about where else within our critical infrastructure we can invest toward resilience and not in all cases drive toward the lowest cost,” he said.

Morgan said guaranteed cost recovery may be needed to fund utilities’ defenses. “Some of the other things that are going to be required if we’re going to address this nation state threat are going to be harder to do and not that cheap,” Morgan said. “The flip side is if I start, as the federal government, providing various cash incentives or other ways to finance stuff, there’s going to be a temptation to gold plate.”

Government Duplicating Private Sector Efforts?

Robert M. Lee, founder of Dragos, said the partnership between government and the private sector is not as effective as it could be.

“I think that we often times publicly spend a lot of time on complimenting each other versus saying, ‘Well, actually this doesn’t work and here are the things that are a waste of time.’ When I look at DHS and DOE, as an example, I see a lot of opportunity. I see a lot of really wonderful people and I see the ability for them to have a significant role in things like amplification, prioritization, helping with … government resources during a time of crisis,” he said.

“But then I see other efforts like, ‘Oh, yeah, let’s go build an incident response team.’ Why? We actually have all of that in the private sector. Why are we spending time and taxpayer money on that? My recommendation is cut out the stuff that we have helped the private sector get really good at and let’s be proud of that momentum and let’s focus on the things like supply chain that actually the private sector shouldn’t take on and that there’s a very significant government role in.”

When corporate boards ask him how to know if they are underspending or overspending on security, Lee said he tells them to meet their regulatory requirements and prepare for known scenarios, such as Ukraine 2015, Ukraine 2016 and ransomware.

“If you prepare for those and then Russia gets crafty and [does] something extra, it happens,” he said. “Your response strategy — that’s your design basis. And everything else above that: invest if you’d like. It’s risk reduction, but there’s no right answer. … If you didn’t prepare for those and you get attacked with the 2015 Ukraine [strategy], you should be in jail. Because it’s an absolute travesty that your community didn’t prepare.”

Unintended Consequences?

Jeff Dagle, chief electrical engineer for electricity infrastructure resilience at the Pacific Northwest National Laboratory (PNNL), said CIP standards have dissuaded some utilities from deploying synchrophasors that can provide situational awareness.

“If an operator can use that data and make a decision within 15 minutes, it is required to be compliant to the NERC cybersecurity requirements,” he said. “There are utilities that are choosing not to deploy technology that’s readily available … because of the … regulatory risk. … If your auditor doesn’t like the way you’ve set it up – bam!”

“And reliability coordinators are having trouble getting this data from the transmission operators because this handoff” is subject to CIP rules, he added. “These … aren’t critical things that somebody could hack in and shut down the grid. This is supplemental information to the operators for better situational awareness to make better decisions. We don’t [require] CIP compliance on some of the other things in the control room. There’s a weather map they can look at and see the thunderstorms coming across their service territory. We don’t require the Weather Channel to be CIP compliant. I suspect this same comment applies to other nascent technologies [and is] slowing innovation,” he added.

FERC’s McClelland noted standards are open to comment at any time. “So, if a standard [or] a requirement is in the way, of security or … reliability, then my expectation is that industry will petition [to change] that requirement.”

McClelland also suggested synchrophasors could be of interest to hackers.

“If you’re saying that … the synchrophasor technology makes [it possible to] react in 15 minutes and that that would be a needed function on the grid, as an adversary, I’m now targeting synchrophasors. … Adversaries are intelligent.”

“If we know adversaries are mapping the power system, you can doggone well bet they’re using electrical engineers to identify critical locations and they’re looking at specific equipment that’s become … absolutely necessary to operate these networks and systems.”

Boundaries Blurring

“The boundaries between utilities and national security are blurring,” said Caitlin Durkovich, director at Toffler Associates, the strategic advisory firm founded by “Future Shock” author Alvin Toffler. “I believe the security and resilience of our country is becoming more intertwined with critical infrastructure than ever before.”

Durkovich, former DHS assistant secretary for infrastructure protection, called for a strategy for an “integrated and resilient modern infrastructure.”

“I think you need a central coordinating body that is different than the post-WWII structure we have today, that is responsible for advancing a modern infrastructure.”

“We have to increasingly focus on this concept of foreign interference and the ability of our adversaries to meddle just enough and not get a kinetic response. We have to rethink what that means given how far they’ll go and what their capabilities are.”

Paul Stockton, managing director of Sonecon, said he expects any attack by the likes of China to be more than just an annoyance. He cited the Worldwide Threat Assessment finding that China could disrupt gas pipelines for days or weeks.

“China is not going to attack a single pipeline. If they’re going to roll the dice and do something that exposes them to such extraordinary risks of U.S. response, they’re going to go whole hog. They’re going to take down as much gas flow as they can to totally disrupt the generation of power to achieve their national security and political goals,” he said. “So, we need to think about this … indirect way of jeopardizing grid reliability in the context of a modernizing grid. Because gas is going to be with us for at least the near- to mid-term and maybe longer.”

Stockton suggested development of a design basis threat for the oil and natural gas (ONG) sector like NERC has for electric substations. (See Design Basis Threat: ‘Best Security Training Ever.’)

“Let’s get going on that because right now owners and operators are left to figure it out for themselves, as are RTOs and ISOs. So, let’s agree on what the threat [is] … . It exists in the classified level. Let’s get something unclassified.”

Stockton said generators in the cranking path for black start plans are also likely to be targeted. “We never really think to test black start in a realistic way because you’d have to have a blackout,” he noted.

In the past, the assumption has been grid operators can import power from outside the blackout footprint to start the cranking path. “Not anymore,” Stockton said. “It is likely — in fact we should expect — Russia and China would like to achieve interconnection-wide blackout or maybe even nationwide. And black start is going to be absolutely vital under those circumstances in a way that just wasn’t true when you think of a New Madrid scenario, as horrible as it would be,” a reference to a worst-case earthquake originating in southeast Missouri.

“The bad guys know that. … They will intentionally target black start assets, cranking paths, generation units, communications — everything they possibly can.”

Limits to Exercises

“I think exercises are getting better,” said Stockton, who is a GridEx facilitator. “But I think they need to focus on this holistic challenge of interdependent infrastructure. That brings the different tribes together. … the tribe of the transmission operators, substation operators, together with cybersecurity personnel. Because they don’t usually kiss on the lips, do they?”

“We don’t have the tools to adequately understand the interactions of these multi systems like gas with electric,” said EPRI’s Howard. “We talk about it. At a high level, we understand it. But it’s the interactions — we don’t have the simulation tools to be able to do a good job with that.”

DOE is attempting to build such a tool, the North American Energy Resiliency Model (NAERM). (See “Grid Resilience Model as a ‘Platform’” in DOE’s Walker Sees Big Cuts in Storage Costs.)

In addition to participating in national exercises such as GridEx, Harrell said utilities should conduct their own exercises with regularity “to ingrain it into the culture” and ensure familiarity with their response plans.

“I don’t know we do that enough outside of, ‘We have to do this once a year because CIP compliance says we must,’” he said.

Need for Simulation Tools

NERC’s chief engineer Mark Lauby said he would like simulation tools “that allow us — just like we do for an N-1 [scenario] — [to] build to a certain level of risk, understand what the mitigations are that we’re building into the system, and then after that [consider] recovery strategies.”

Lauby said grid operators need to “get in front of” the technology changes, such as the increase in inverters and asynchronous generation on the system, to “be sure we’re not building in more [attack] surface but rather de-risking and taking advantage of the technologies.”

William H. Sanders, interim director of the University of Illinois’ Discovery Partners Institute, said “The trick is to find the models with the right level of detail and abstraction that you can discover things … surprising things emerge, not just you fill everything in, and the model tells you what you knew it would tell you. I think we are making great progress. We have test beds. We have examples of models that can help us understand … I think we need to scale those up in a big way.”

Communication Breakdown?

“There’s no simulation that can fully appreciate the consequences of how things are going to cascade,” Durkovich said. “It all depends on the circumstances and the factors of the day.”

“I know GridEx is continuing to try and [address] this but we do all these exercises and I think live in a fantasy world where somehow communication is not degraded and is fully there.”

In large crowds, cellular service can be difficult because the local network is congested. “What makes us think that’s not going to happen on a really bad day? I was here on 9/11. You couldn’t get anything out.”

She said there aren’t enough exercises at the state and local level. “That’s really where we need to build capacity. Yes, you have DHS. But really, at the end of the day, … they’re not going to be there to respond to critically important state-level assets. … I don’t think states and localities have a full appreciation of how much of the burden they’re going to share on a bad day.”

Morgan said the previous National Academies study “talked precisely to that point and argued there was an urgent need to do something. As best as I can tell, [the report is] sitting on a bunch of shelves around town. We did brief quite a large number of people. But as several of you have said, there needs to be a wider recognition of the urgent [need for] moving towards greater resiliency.”