Back to the Nov-Dec 2019 issue

SCADA Upgrade: Protect City Utility Systems From Hackers

By Mary Jane Smetanka

Around Minnesota, a potentially big problem lurks in computer systems linked to a basic but critical city function: providing sewer service, clean drinking water, and other utilities to residents.

On Jan. 14, Microsoft will stop supporting Windows 7, the computer platform that many cities use to run software that monitors and controls public works functions such as operation of lift stations, well pumping, water storage levels, electricity, and other utility services. Unless cities upgrade their technology, they could be open to malicious activity like hacking.

“As cities have automated, they’ve started to use their ability to get on the internet and connect everything,” says League of Minnesota Cities Chief Information Officer Melissa Reeder. “Public works manages these systems day and night, and it’s great to monitor them wherever you are. But you do have exposure to the outside world, and the possibility of hacking.”

A very real threat

While the chances of being hacked may seem unlikely, cities nationwide have been targeted by people who look for vulnerabilities in computer systems to demand ransoms or to see if they can penetrate a seemingly secure network. Once Windows 7 support goes away in January, hackers will be looking for those security holes, Reeder says.

Most of the software used to control city water systems operates on a Windows platform. Those supervisory control and data acquisition systems, or SCADA, are usually installed on a PC in a public works building or connected to devices like laptops and phones, allowing remote monitoring and control, and alerting users to problems like a loss of water pressure or a broken pump.

Years ago, Microsoft announced that it would stop supporting Windows 7 in 2020, but Reeder worries that many small cities with old computers may not be aware of how vulnerable they could be. She advises cities to address this now.

“I think this situation is widespread,” she says. “Hackers will program for these vulnerabilities.”

Upgrading in New Prague

New Prague built a new wastewater plant in 2010 and has been operating the plant and nine lift stations on software that uses Windows XP. Microsoft ended support for XP in 2014.

Public Works Director Glen Sticha says that though the city’s system has operated safely so far, city officials hired a technology consultant last summer to help upgrade the system. In addition to buying new Windows 10 computers and other assorted hardware, the city is spending an estimated $38,000 to buy updated SCADA software.

]New Prague wants to hold taxes down, Sticha says, but city officials agreed that it wasn’t worth the risk to run critical utilities on very old platforms that could fail or be compromised.

“We have to do it. We’ve lived on borrowed time with XP for a long time,” he says. “You have to watch where you’re spending your money so you’re not raising taxes all the time, but what if it fails and someone says, ‘You’re using XP? Are you kidding me?’ These are the things that keep me up at night.”

To help keep its SCADA system safe, it is backed up daily and is not directly connected to the internet. The public works staff has remote access to the SCADA system via their smartphones and touch-screen monitors in the treatment plant.

But on occasion, hardware has failed, Sticha says, and the monitors work only with XP. Now, parts to fix them are almost impossible to find.

Getting new computers that use Windows 10 means that even if something in the system fails, there will be redundant computers to back everything up. And regular software updates will make the system more secure.

Budgeting for Windows 10

The City of Victoria has a single PC with a Windows 7 SCADA program running its water treatment plant and lift stations. Brady Lee, a public works employee who is dedicated to the water treatment plant, says the city began planning for a Windows update in 2018. He has been working on it with the city’s technology consultant.

“We’re trying to prepare for the overall cost so it can be fit into next year’s budget,” Lee says. “We will need to buy a new computer with Windows 10 for the water plant. We already purchased the SCADA programs, so they can be upgraded.”

Like New Prague, Victoria’s public works computer system was removed from the internet to improve security, and Lee is confident the city can update the system soon. “I’m just trying to stay ahead of the curve and keep things updated,” he says.

Disconnect old systems from internet

Only a handful of firms make SCADA software that cities use for public works and utilities. One of the biggest is Wonderware. Joe Finn, director of sales for Wonderware Midwest, says about 700 cities in the six upper midwest states use Wonderware software with interfaces that allow users to monitor and control water systems.

Finn says the firm doesn’t continue support for obsolete operating systems, and issues updated software within three to six months after a new version of Windows is released. For cities that continue to rely on old Wonderware software that runs on pre-Windows 10 computers, he recommends they disconnect that system from the internet.

“Hackers will be out looking for copies of Windows 7, and if you’re not connected, they can’t get to you,” Finn says.

Upgrading is best option

Microsoft is offering Windows 7 security updates for a fee for some clients until 2023, but those updates will be expensive and don’t protect cities if older computers using obsolete software die. That could mean real trouble for cities, Finn says. In the long run, it’s easier and safer to buy a new Windows 10 computer and get new or updated software.

Cities that already have a SCADA program like those sold by Wonderware can update their software, something that Finn says generally costs about half the price of buying a new program. For a small city with a single computer, he estimates the cost of basic Wonderware SCADA software would range from about $1,500 to $3,000, depending on what features a city chooses.

Reeder urges cities not to take any chances. “Your network is only as secure as your weakest link,” she says. “Cities tend to leave old technology in public works because it’s working great. But that’s where the mistakes can happen.”

Sidebar: Security Risks Beyond Public Works

Microsoft’s ending of support for Windows 7 on Jan. 14 has implications for city operations beyond just the public works department. Older versions of Windows are more susceptible to security issues, from hacking to phishing schemes that trick users into clicking on links with viruses or ransomware.

Though cities can download Windows 10 if they already have Windows 7 or 8.1, the free update period is over, and it will cost about $100 to download Windows 10. But older computers may not have the capability to handle Windows 10, so even if the new version is successfully downloaded, it may not work on older machines, says League CIO Melissa Reeder.

If computers are three years old or older, Reeder recommends buying a new computer that has Windows 10 already installed. While that is no small cost for a small city — new PCs cost $500 to $800 or more per computer — it is the safest and easiest route to go, she says.

“The only safe way to continue using an old operating system like Windows 7 is to remove the computer completely from your network and the internet,” Reeder says. “In this day and age, it’s pretty impractical to think you are going to operate any business without an internet connection.”

But, she warns, if you connect a Windows 7 computer to any portion of the city’s network, you are putting all computers on that network at risk. “In other words, just deciding to not surf the net with a Windows 7 computer will not be enough,” Reeder says.

There could be additional issues with moving to Windows 10. Cities need to look at the software packages they use. Utility billing, general ledger, payroll, and other programs may need to be updated to work with Windows 10, and some cities may need help from consultants to do that.

With Windows 7 support ending in January, IT folks could be busy, Reeder points out. Cities need to start working on their technology now.

Mary Jane Smetanka is a freelance writer.