15 minute read
Focus on Water & Energy: Protecting your water
Protecting your water utility from cyber threats
By DENISE FEDOROW | The Municipal
Oldsmar, Fla., averted a potential catastrophe in February when someone breached the computer system at the Bruce T. Haddock Water Treatment Plant and changed the levels of sodium hydroxide from 100 parts per minute to 11,100 ppm.
An operator monitoring the system happened to see the cursor moving across the screen and corrected the levels.
“That’s called getting lucky,” Kevin Morley, manager of federal relations for American Water Works Association, said.
During a press conference organized by the sheriff and city officials following the event, Pinellas County Sheriff Bob Gualtieri said it was an “unlawful intrusion” to “part of the nation’s critical infrastructure.” The perpetrator actually made two attempts. The first was at 8 a.m. Feb. 5, but it was very brief, and the operator thought maybe supervisors were accessing the system through remote access. At 1:30 p.m., the system was again breached, but this time the perpetrator changed the amount of sodium hydroxide—a Kevin Morley, caustic ingredient in drain cleaners—to “sigmanager of federal nificant and dangerous levels,” according to relations for Ameri- the sheriff. can Water Works The sheriff and city officials — Mayor Association Eric Seidel and City Manager Al ABOVE: Vigilance is key to preventing cyberattacks on water utilities. Water utilities should focus on implementing best security practices, such as avoiding exposure of critical assets to the internet, establishing redundancy mechanisms for critical assets, employing strict access control policies and raising security awareness among employees. (Shutterstock.com)
Braithwaite—stressed residents were never in danger because even if the cyber attack had not been detected, there are other controls in place that would’ve set off alarms before the increased ingredient could have entered the drinking water system. They also noted it takes 24-36 hours to hit the water system. At the time of the press conference on Feb. 8, Sheriff Gualtieri said they didn’t know whether the threat came from inside or outside the country.
The FBI is still investigating the incident, so when called, Oldsmar’s assistant city manager said, “We’re not engaging in any conversations on that topic at this time.”
During the press conference, Braithwaite responded to a reporter’s question by stating, “We anticipated this day coming—we talked about it and studied it.”
However, Morley said the Oldsmar system had no firewall and a weak password, so “it didn’t require a lot of sophistication to hack it.”
In a press report days after the incident, an FBI investigator was cited as stating the cyber actors likely accessed the system by exploiting cybersecurity weaknesses, including poor password security and an outdated Windows 7 operating system, and they likely used a shared software, Team Vision, to gain unauthorized access to the system.
Morley said this is not an isolated incident and shared a man was just indicted in Kansas for breaching a water utility. In that case, it was a terminated, disgruntled employee who shut down the water system.
A press release dated March 31 stated that Wyatt A. Travnichek, 22, of Ellsworth, Kan., was indicted with one count of tampering with a public water system. On or about March 27, 2019, Travnichek knowingly accessed the Ellsworth County Rural District’s protected computer system without authorization. It is alleged Travnichek performed activities that shut down the processes at the facility, which affected the facility’s cleaning and disinfection procedures, with the intention of harming the Ellsworth Rural Water District #1, known as Post Rock Rural Water District.
A special agent in charge of the U.S. Environmental Protection Agency’s Criminal Investigation Division in Kansas said the indictment sends a clear message that individuals who intentionally violate these laws will be vigorously prosecuted. Upon conviction, the alleged crimes carry the following penalties: tampering with a public water system—up to 20 years in federal prison and a fine up to $250,000—and reckless damage to a protected computer during unauthorized access—up to five years in federal prison and up to a $250,000 fine.
When asked how common these instances are, Morley replied, “It’s the number one threat to critical infrastructure and not exclusive to water.”
Prioritizing and protecting cybersecurity
Daniel Kapellman Zafra, manager of analysis, Mandiant Threat Intelligence at FireEye, a cybersecurity firm, spoke to the threats utilities face.
He said all organizations are vulnerable to cyberattacks. “However, the water and wastewater sector is generally less mature in cybersecurity than other sectors. This is partly caused or exacerbated by the local nature of most operations: mostly small, municipality-owned utilities that rely on limited resources to support complex cybersecurity programs. Additionally, there is currently a lack of regulation and resources developed specifically to guide and enforce implementation of security controls in this sector.”
Zafra added, “Some of the challenges water utilities face—similar to other critical infrastructure industries—include slowly maturing cybersecurity infrastructure, outdated hardware/software, unauthenticated protocols and lack of security resources and knowledge. This is especially important in water utilities, given their relevance for other industries and social well-being.
“More than large-scale cyber physical attacks, water utilities face an immediate threat resulting from financially motivated criminals and low-sophistication actors that leverage commodity-type attacks or target internet-exposed assets (such as the case of the recent hack on a Florida water utility).”
To protect against these types of attacks, Zafra said, “Water utilities should focus on implementing best security practices such as avoiding exposure of critical assets to internet, establishing redundancy mechanisms for critical assets, employing strict access control policies and raising security awareness among employees. They should also place emphasis on understanding the safety processes in place and how these could be (or not) bypassed by attackers via cyber means.”
He added, “Further investment in regulation and guidance for supporting water utilities to mature their security programs would also be beneficial.” The Bruce T. Haddock Water Treatment Plant in Oldsmar, Fla., was the victim of an attempted cyberattack on Feb. 5, 2021. The cyberattack was caught by a vigilant employee, and city officials stated in a press conference there were other redundacies in place that would have alerted them there was a problem before the elevated levels of sodium hydroxide hit the drinking water system, but an outdated operating system and weak passwords put the water utility in that vulnerable position, according to the FBI and cyber security officials. (Photos provided)
Even if a utility is confident it has a working security system in place, Zafra said, “Ideally any organization should invest at least in regular monitoring of traffic from non-trusted networks. In the case of organizations such as utilities, this traffic volume could be limited by implementing a robust segmentation that limits access to and from production networks. Organizations that reach a high maturity level and have access to the resources may also be able to hunt for malicious behaviors based on threat intelligence.”
Morley pointed out in an article, “Priority on Cybersecurity,” which he wrote for DC Beat in March 2019: “Drinking water and wastewater systems not only manage sensitive personal data, they also operate process control systems that are essential to day-to-day operations. A cybersecurity breach in the water sector could result in serious harm to public health and safety, as well as other damages from service interruptions, lost data, compromised systems, litigation and repair costs and reputational harm. In fact, government intelligence has confirmed that drinking water and wastewater systems have been directly targeted by nation states, as part of multi-stage intrusion campaigns, and by individual criminal actors and other groups seeking to harm the United States or obtain illicit proceeds.”
Pinellas County, Fla., Sheriff Bob Gualtieri — at the podium — held a press conference regarding the attempted hack at Oldsmar, Fla.’s, water treatment plant. Also with the sheriff are Oldsmar City Manager Al Braithwaite, far left, and Mayor Eric Seidel. (Screenshot via https://youtu.be/MkXDSOgLQ6M) This table shows the deadlines community water systems had to comply with the America’s Water Infrastructure Act of 2018’s requirements. (Table provided)
TABLE 1 Who must comply and when?
Community Water System
(population served)
Risk and Resilience Assessment
>100,000 Mar. 31, 2020
Emergency Response Plan
Sept. 30, 2020
50,000-100,000 3,300-50,000 Dec. 31, 2020 June 30, 2021 June 30, 2020 Dec. 30, 2021
He wrote those attack campaigns used various tactics, including spear-phishing emails from a compromised legitimate email, watering hole domains, credential gathering, open-source and network reconnaissance, host-based exploitations and industrial control system infrastructure targeting.
But AWWA has developed tools and training to help. “Since 2014, we developed a set of guidance and assessment tools for utilities to examine vulnerability in line with the National Institute of Standards and Technologies (NIST) cybersecurity framework,” Morley said, adding these were developed along with the Obama administration under executive order 13636.
He said the impetus for the tools goes back to 2008 when officials developed a roadmap for security processes and discovered there was “a whole lot of different standards but no consistent template to apply those standards.”
So in 2013, at the same time as the Obama executive order, AWWA put in guidance for water utilities. Morley said the organization collaborated with the Department of Homeland Security, EPA and NIST “to ensure the end product aligned with the federal family.”
“We had the first sector-specific approach to applying controls for cybersecurity framework,” he said.
According to Morley, AWWA recognized it couldn’t expect someone in small-town Mid-America to become a cyber expert, so the association developed the tools in a way that its 52,000 diverse community water systems would find relevant.
“We said let’s take this from the perspective of the utilities—how do these controls apply to the guidance they use?” he said.
The guidance tool asks questions about how utilities use technology, such as whether they allow employees to “bring your own device” or not? He said there’s a set of 22 questions that utility managers either answer that they do or they don’t, and if they don’t know, they’ll need to find out.
“So there’s some self-discovery there, and that’s really important,” Morley said.
The tool also allows them to prioritize controls, with priority one being a must-have. “They found it to be very useful in the decision-making process,” Morley said.
As the utility answers questions, one feature is a status check of 100 controls, and the utility gets a scorecard of its level of progress. The scorecard is something that can be taken to management to spur action. For instance, if there are 20 priority one controls but the utility is only implementing 10, managers can see the other 10 controls need working on ASAP.
Morley stressed all these safeguards don’t mean a utility won’t be breached, “but it makes it harder to do so. This is risk management, not risk elimination.”
AWWA also has a set of resources for small systems and developed training as well through a USDA grant. “It’s been very successful in helping smaller utilities move along in getting things done.”
Risk and resilience assessment
Morley shared a reminder that communities are under a statutory obligation under the America’s Water Infrastructure Act of 2018 to provide a risk and resilience assessment, including cybersecurity, and also to have an emergency response plan, which requires an action plan. Larger systems serving over 100,000 had their deadline last year. Systems serving between 50,000 to 100,000 had their first deadline in December 2020, and the next one to certify the emergency response plan with the EPA is due June 30. Smaller systems, which compose the largest group—utilities serving 3,300-50,000—have a deadline of June 30 for its risk and resilience assessment and Dec. 30 for the emergency response plan.
“Getting cybersecurity right is not an easy issue—threats are persistent and mutable,” Morley said, adding, “The scary thing is, from a cybersecurity perspective, the parameter isn’t at the gate—the criminal element can be anywhere in the world.”
But he said, “The priority one controls are foundational speed bumps mitigating a lot of the large number of threat attempts we see.
“If the Kansas utility had disabled the employee’s credentials when he was fired and if Oldsmar had a firewall and a strong password—in these cases, there were no real speed bumps to accessing the systems.”
Utilities “should have a basic working knowledge of how their system is moving information through the system. Once they know that, it’s easier to assess the risk.”
“Cyber risk is a serious threat, and it is critical that organizations in the water sector make cybersecurity a top priority,” Morley said, adding, “Use the tools.”
An ‘array’ of benefits:
Pictured is an overhead shot of Healdsburg, Calif.’s, south pond solar array. (Photo provided)
Taken by the Healdsburg Police Department’s drone, this photo shows the solar project — near pond and far pond — during a dedication ceremony. Included in the picture from left are Felicia Smith; Rhea Borja; Michael Kremer; council member David Hagele; Mayor Evelyn Mitchell; Vice Mayor Ozzy Jiménez; council member Ariel Kelly; Jeff Kay; Terry Crowley; and Jon Wank. (Photo provided)
By JULIE YOUNG | The Municipal
The city of Healdsburg, Calif.’s, new floating solar array didn’t start out to be a triple threat of environmental responsibility, but since its completion in January, it’s been generating quite a bit of buzz. Not only does the 4.78-megawatt photovoltaic array produce 8% of the community’s annual electrical needs, but it also shades the ponds at the Healdsburg wastewater treatment facility, reducing algae build-up so the recycled water can be sold for agricultural purposes. As if that weren’t enough, it is the largest floating solar array in the U.S.
Made for shade
According to Terry Crowley, P.E., utility director for the city of Healdsburg, having the largest floating array in the country was not the intent of the project, nor was producing nearly 5 megawatts of power for the city. The genesis of the project was the desire to mitigate algae growth within the city’s recycled water ponds.
“The city explored various options to shade the ponds,” he said. “The two top tier options were shade balls (small balls that float on the water) or floating solar. In looking at the economics, float solar was found to be more economical due to the energy created.”
Healdsburg’s wastewater treatment facility is a state-of-theart tertiary treatment system, which processes raw sewage into clean and disinfected recycled water that can be used in orchards, vineyards and for other agricultural purposes. The water is stored in large thermoplastic-lined ponds and funneled through pipelines to those who need it, which reduces the demand for groundwater.
The floating solar array is a concept that has been around for about 10 years, but the pricing was never within a range that was competitive with wholesale energy markets. After drafting the project with the support of the Northern California Power Agency and releasing a request for proposals, the contract was awarded to Dissingo as a power purchase agreement in June 2020. Construction began four months later and was completed in March.
In addition to the novelty of being located on water and — at the present time — being the largest floating solar array in the nation, the floatation devices can be moved so city wastewater staff
can inspect and repair the pond liners if necessary. Crowley said the project has a design life of 25 years but could operate a lot longer thanks to its ability to incorporate a future utility-scale battery storage system. The array is located on a 25 million-gallon tertiary pond and contains 11,600 panels.
“Construction of the project went surprisingly smoothly, largely due to the efforts of White Pine Renewables and Collins Electrical,” Crowley said. “These two entities were able to bring real-world experience and knowledge to the project, completing it within an incredibly short period of time.”
A power play
In a PPA, the developer pays for the entire project and retains ownership of the array, while Healdsburg Electric pays a fair market price for the electricity that is delivered to its publicly owned utility. In addition to allowing the solar developer to apply for tax incentives that the governmental entities are not eligible for and cutting Healdsburg’s costs for energy, the new array helps Healdsburg meet the state of California’s environmental sustainability requirements and climate goals to add renewable and carbon-free energy.
“The city of Healdsburg owns and operates its own electric utility,” Crowley said. “This helps the city keep energy costs low, but also allows the city to commit to building new green energy sources for our customers. Not only will the floating solar project provide 8% of citywide annual energy needs, it fits within the city’s goal of providing 50% renewable energy by 2025 and 60% renewable energy by 2030.”
Crowley said community response to the project has been very supportive, and the knowledge of the project has helped raise awareness about how the city of Healdsburg meets the community’s energy needs. He suggests communities considering a similar project take the time to vet potential contractors regarding their experience in constructing large solar arrays and spend time developing a good power purchase agreement that contemplates future problems and end-of-life issues.
“It’s easy to get excited about these projects on the front end, but thought should be given to what might happen over the duration of the agreement,” he said. This photo provides a closeup look of the floats with panels attached; the flat portions are for walking on when the floats are in the water. The total project required 11,600 panels, each capable of producing 410 watts. (Photo provided)
Displayed is the construction method to assemble the floats and panels. Like a large knitting machine, the floats are laid out in a row before panels are installed and wired atop them. Once complete, each row is pushed into the water to make room to construct the next row. (Photo provided)
From the ground level, this picture shows the north pond’s floating solar array. The array is anchored into the berms on all four sides. (Photo provided)
