Regulations governing the safety of battery energy storage systems (BESS) “have developed slower than the technology” and may not be sufficient to prevent damage on their own, according to a Lessons Learned report published by NERC on Monday.

The report concerns a fire and explosion that occurred in April 2019, at the McMicken BESS in Surprise, Ariz., managed by Arizona Public Service. Normally a NERC Lessons Learned would not include such identifying information, in the interest of privacy; however, in this case APS “wished to be known in order to expedite the dissemination of information” on the incident. The company has also made its full report on the event public on its website.

At the time of the fire, McMicken had been in service for more than two years. APS sourced the system from AES in June 2016, with AES agreeing to “design, engineer, procure, construct and commission” the facility. Later that year AES signed a long-term maintenance and services contract for the battery; it assigned the contract to Fluence Energy, a joint venture between AES and Siemens, in 2018.

All components of McMicken — except for interconnection equipment such as transformers, breakers and switch gear — were located inside a structure about the size of a standard shipping container. The facility contained 27 vertical racks, each holding 14 lithium-ion battery modules with 28 cells each, an 80-kW inverter, a node controller for data collection and communication, and a battery protection unit.

An additional rack held communications equipment for the system, and eight more racks were available for expanding or upgrading the facility, although at the time they were being used for storage.

Battery Fault Leads to Runaway Heating

On the day of the fire, McMicken was performing a solar-smoothing function. This means absorbing excess energy produced by rooftop solar panels on the circuit during the day and discharging during evening peak load. The subsequent timeline was reconstructed by investigators using electronic logs from the BESS and dispatch data from entities including APS, Fluence and first responders.

Shortly before 5 p.m. PST, while undergoing a charging cycle, an internal fault in a battery cell led to a sudden drop in voltage of about 0.24 V. Within minutes the cell went into thermal runaway, which cascaded into other cells in the module and then into other modules in the rack.

The thermal runaway generated off-gassing and smoke from the batteries that triggered the facility’s fire suppression system, which discharged a Novec 1230 “clean agent” chemical manufactured by 3M. However, while this system worked as intended, the agent was designed to extinguish incipient fires and ineffective at stopping thermal runaway once underway.

As a result, the heat continued to build, as did the flammable gases in the container — which did not ignite, despite the high temperature, because the Novec 1230 displaced oxygen in the air. While the agent dissipated within 30 minutes, by that time there was not enough oxygen in the environment to start a fire. The gas buildup continued, with no ventilation because the shutdown of the facility’s heating, ventilation and air conditioning (HVAC) system (also part of normal fire procedure).

Blast Followed Firefighters’ Entry

About a minute after the Novec 1230 discharged, APS contacted Fluence to verify that the fire-suppression system was engaged. Fluence then sent a field service engineer, with APS sending its own personnel about five minutes later. By 5:45 p.m. both APS and Fluence had called 911. Firefighters arrived shortly afterward.

At 8:02, emergency responders opened the front door of the container. APS’ report does not specify why they decided to enter the BESS at this time or what they had done between arriving and opening the door.

The explosion occurred almost immediately afterward, most likely because of the mixing of oxygen from outside with the flammable gases inside and then contacting a spark or residual heat from the battery module. Several firefighters were injured, and the BESS and its containing structure were “essentially destroyed”; photos by APS show buckling walls, doors blown off their hinges and damage to the HVAC systems.

Thermal Runaway not Addressed in Standards

APS’ investigators identified five main contributing factors to the incident.

• The thermal runaway was triggered by an internal failure in a battery cell.
• The thermal runaway was able to cascade because of a lack of thermal barriers between cells.
• The fire-suppression system was incapable of stopping the thermal runaway condition.
• The structure had no ability to ventilate flammable off-gases once the HVAC switched off.
• APS’ emergency response plan (ERP) did not have an extinguishing, ventilation and entry procedure.

Both the entity and NERC acknowledged that McMicken was “constructed according to the standards at the time”; the shortcomings that led to the explosion were because “the contributing factors … were not well known at the time this BESS was commissioned.” While standards organizations such as the National Fire Protection Association have taken notice of the incident, mandatory regulations take time to develop. In the meantime, many utilities operate facilities similar to McMicken that may face similar risks.

In addition to improved cell quality, a manufacturing issue, the APS report identified five stages in the buildup to the explosion where prior action by the utility could have prevented disaster. First, barriers inside the battery modules could have prevented cell-to-cell cascading. Spacing battery modules farther apart could have hindered module-to-module cascading as well.

Designers also should have provided some means for the flammable gases to be ventilated before they reached dangerous levels, along with alternative ways of reducing the temperature of the battery modules, the report said. The reliance on a clean agent like Novec 1230 — acknowledged by its own manufacturer to be insufficient to address thermal runaway — speaks to the designers’ lack of consideration for thermal issues other than fire.

Management “of thermal runaway differs from that of a conventional fire, specifically because starving a thermal runaway event of oxygen may have little effect,” APS’ report observed. “Instead, heat must be removed (by cooling with water), or the fuel must [be] dispersed in order to stop a thermal runaway event.”

Finally, the ERP developed by AES lacked instructions for entering the system after the fire-suppression system had been discharged, focusing instead on electrical shutdown procedures, roles and responsibilities in an emergency, and when to notify the fire department. APS noted that “none of the BESS suppliers [at the time] had conveyed that a large flammable gas hazard or … cascading thermal runaway was possible.”

This communication breakdown had major consequences when first responders were on the scene. Firefighters with no knowledge of BESS construction had no choice but to improvise an entry procedure on their own, with catastrophic results. APS recommended that detailed procedures be worked out ahead of time, communicated to all appropriate responders, and posted outside the BESS structure.

In its Lessons Learned report, NERC reiterated these recommendations, along with additional suggestions from an “industry substation fire expert” such as fire alarm control panels in facilities, including flammable gas monitoring capabilities, so that first responders can monitor conditions from a safe distance. Most importantly, all of these systems, and their risks, should be familiar to anyone who may find themselves in harm’s way.

“The fire services should not be seeing a BESS for the first time when 911 is called. … Conduct training, familiarization tours and exercises with your local fire department,” NERC recommended, adding that regular exercises with the first response community can help to “validate the plan or identify gaps.” The organization suggested that such exercises could follow the approach laid out in a previous Lessons Learned report that focused on utilities’ relationships with first responders in emergencies.