Members of the Pennsylvania National Guard cyber team joined 18 Aqua employees and representatives from the FBI, Homeland Security, PA DEP, Chester County Emergency Management, and Schuylkill Township Police to participate in Aqua Pennsylvania’s most recent emergency tabletop exercise that involved a cyberattack and Black Sky event that targeted water, gas and electric utilities and eventually disrupted utility service for millions of customers.
Aqua Pennsylvania provides water to 1.4 million people in 32 of Pennsylvania’s counties.
Alarms indicating pump failures began sounding at Aqua Pennsylvania’s Pickering water treatment plant—the corporation’s largest surface water plant. An immediate check of the pumps showed that everything was working properly. Plant operators, after communicating with their supervisors, began inspecting the pumps, chemical systems, filter operations, analyzers—particularly those that measure turbidity. They also checked the intakes and the Upper Merion reservoir, which is remotely operated from the Pickering facility.
Three hours later, SCADA alarms indicated chemical feed system problems and dosage fluctuations. The plant manager informed Production Director Jeff Bickel. “The previous pump failures now combined with chemical-feed systems problems, requires us to shut the plant down and operate it manually, removing it from SCADA. Because the incident took place on Veterans’ Day, multiple personnel were called in to support manual operation of the plant,” said Bickel.
Once the plant is shut down. The first restorative measure is to turn on the sendout pumps, to ensure water service is continued using stored capacity. Afterward, raw water pumps and pre-treatment and filtrations systems are restored. All unnecessary processes are shut down, including sludge handling.
It turned out not to be just another day in the neighborhood for production employees at Pickering. By the end of the day, malware had affected the operations of critical infrastructure in the entire mid-Atlantic region and a ransom request was received from a location outside the US. Power and communications in the entire area were down and everything was completely dark.
Fortunately, this was only a training exercise that we at Aqua periodically conduct to ensure our systems and operations personnel are prepared in how to respond.
Aqua Pennsylvania’s Vice President, Production Curt Steffy and Director, Environmental Compliance Deb Watkins planned and coordinated the exercise, using outside consultants, over the previous six months. “An exercise of this magnitude takes a great deal of time and effort to coordinate,” said Steffy. “We had a great team that worked to make this scenario as realistic as possible. Bringing everyone from Pennsylvania together to participate is consistent with our One-PA initiative.”
The exercise was presented by MHS Consulting Services of Marysville. PA. Dr. Stanley States of Texas A&M led the emergency exercise along with MHS owner Mike Snyder. The same team conducted a tabletop exercise that focused on harmful algal blooms at Aqua’s Shenango water treatment facility in Western Pennsylvania in December 2018.
Bickel, who worked with plant managers from all of Aqua Pennsylvania’s surface water plants to address the emergency said, “The exercise was also a great opportunity to share knowledge and skills with outside agencies, which all participants appreciated. I also enjoyed having the chance to work again with the plant managers from other parts of the states. I’ve worked closely with them on emergencies in the past and continue to be impressed by their knowledge.”
Everyone agreed that Aqua responded well under the circumstances and far better than other companies with whom they worked in similar exercises.
“Based on what I heard today, these guys have their act together. A lot was thrown at Aqua and they have a lot of resources in place,” said Rich Turzanski of the Department of Homeland Security. “Aqua had answers and measures in place to address every situation. That’s huge.”
“I applaud Aqua for investing the time and energy to put this together and for getting great participation,” said National Guard Colonel Frank Montgomery. “You’re probably ahead of your peers out there, so I applaud you.”
FBI Weapons of Mass Destruction Coordinator Mike Schlottke agreed. “Having gone through the exercise, you are well ahead of the power curve. Curt Steffy is already in the working group and in contact with everyone.”
Aqua’s Director, IT Security John Childers described the exercise as “… awesome. We are only better for having completed this exercise. Hearing the processes other departments go through when faced with these issues, really helps us as we determine how best to address these problems.”
“Fortunately, we don’t face these situations with any regularity, so practice is key,” said Aqua Pennsylvania President Marc Lucca. “It’s important that we have these exercises and that they take place with a frequency that keeps these procedures top-of-mind and addresses any turnover that takes place in the interim,” said Lucca. “The feedback we’ve gotten internally and from our emergency partners has been great and I appreciate their participation as well.. It affirms that we are doing the right thing.”