A wee-hours alert of unusual traffic on Midland’s email server last Labour Day weekend launched a cyber nightmare.

A town IT staffer confirmed a hacker was holding Midland’s municipal computer system hostage, demanding ransom in exchange for an electronic key to unlock the data.

“We shut down all of our systems to ensure there was no further spread of it but, by that point, the hackers had pretty much encrypted everything,” Randy Fee, Midland’s communications and marketing co-ordinator, recalled in an interview from the Georgian Bay town, adding that even backup files were encrypted.

City staff had to dig out paper to process permit payments. Original blueprints and maps had to be pulled from storage. It took weeks to recover computer systems.

The crime known as “ransomware” has become an “epidemic,” the U.S. Federal Bureau of Investigation told the Star, with cities and other local governments increasingly among victims forced to choose — reward criminals or rebuild networks from scratch at great time and expense.

Hackers have penetrated almost two dozen municipalities in the U.S. this year alone, along with school boards, libraries, even police departments. Canadian ransomware victims include individuals and businesses, plus Stratford, Wasaga Beach and, city officials recently learned, Toronto.

Beverly Romeo-Beehler, Toronto’s auditor general, revealed last month that two city “entities” were “reportedly attacked by ransomware and their systems compromised. In both situations, the incidents were not communicated to the Chief Information Officer because protocols do not exist.”

That revelation, three years after testing by Romeo-Beehler’s department exposed vulnerabilities to hacker attack in Toronto’s system, and triggered recommendations for stronger safeguards, has set off alarm bells at city hall.

The relatively minor attacks did not compromise the city’s main digital backbone. But concerned audit committee members urged city IT officials at a June meeting to accelerate development of notification protocols and other steps to bolster existing safeguards ranked by one senior official, when pressed, as “between a 7 and an 8” out of 10.

Asked if Toronto’s system is still vulnerable Romeo-Beehler, whose unit will launch renewed “penetration” testing and report back in October, told committee members: “Until (city) management provides the answer that they are ready, the answer is yes.

“This was the number one issue for me when the new city manager came in (last August), it’s been outstanding for a while.”

The city is creating, staff said, a new chief information security officer position to oversee Toronto’s defences to attacks including ransomware, plus measures including bolstering in-house security infrastructure and hiring a “managed security provider” — contracted private-sector experts.

The city is also getting a new chief information officer after Rob Meikle, who came to Toronto from Brampton six years ago, left city hall in late June. The city refused to give details on his departure.

Meikle did not respond to Star messages sent through social media and left at the Brampton church where he is a longtime pastor. Meikle recently told IT World Canada he was looking forward to focusing more on “several other projects in communities across the city.”

Romeo-Beehler told audit committee: “Management has a plan they’re executing. My concern is I think it’s important to do it as quickly as possible.”

The threat of ransomware got attention in 2017 when malware dubbed “WannaCry” wormed through Windows operating systems worldwide, seizing systems including Britain’s National Health Service.

Variations have proliferated. Many sneak in via “phishing,” where an employee is suckered into clicking an attachment or link. Sometimes hackers use automated password generators to break through ill-protected digital doorways used by IT staff to remotely control computers.

Hackers demanded payment in untraceable cryptocurrency, often bitcoin.

Toronto police Det. Shawn Marshall said cities and companies that pay ransoms get encryption keys that sometimes fail to unlock all their data, leaving electronic payment systems and other compromised networks unusable.

That’s because the first generation of ransomware experts, who were adept at undoing their mischief, now sell software packages used by less skilfull hackers. They can seize networks but not always troubleshoot when a victim has trouble decrypting the data.

Police, despite all their expertise and resources, can’t smash the encryption, Marshall said.

“If (hackers) are using some of the strongest methods available,” he said, “then every computer ever made in the history of the world, using all of the time remaining before the heat death of the universe, would not crack that encryption.”

The best defence, he said, is “out-of-band backup” — copies of all data, routinely updated but stored unconnected to the main network. Identifying, much less capturing, ransomware attackers has proven extremely difficult, Marshall said.

“Attribution is definitely the hardest part of these investigations. It could be anybody. It could be a 16-year-old in your neighbour’s basement.”

Loading... Loading... Loading... Loading... Loading... Loading...

In an interview, FBI Section Chief Herbert Stapleton said ransomware has become an evolving “epidemic.” Authorities know that attacks reported to them are a small fraction of occurrences, he added, with many victims embarrassed or afraid of bad publicity if they report.

Municipalities hit by ransomware generally contact an insurer who helps engage security consultants to deal with the hackers, said Stapleton, adding they should also always contact authorities.

“The FBI position is the organization should not pay the ransom because it encourages further criminal activity,” he said. “The more ransoms that are paid, the more the bad guys will be encouraged to to continue this type of activity.”

But, asked if he’s aware of police ever catching hackers during a ransomware attack, Stapleton said: “I’m not aware of any cases where that’s happened.”

This month more than 225 U.S. mayors signed a resolution not to pay ransoms to hackers. But potential costs are steep. Baltimore refused a hacker’s demand for $75,000 (U.S.) worth of bitcoin and now faces total costs of $18 million to repair the damage.

After seeing Wasaga Beach hit by ransomware in April 2018, Midland got special insurance. The insurer steered the town to a consultant who negotiated a ransom of six bitcoins. Decryption keys failed to unlock all of the data, so Midland had to pay two more bitcoins for more keys.

According to bitcoin trading prices at the time, the ransom would have been about $67,000 (Canadian), though the town won’t confirm the cost.

Most of the cost was covered by insurance. Staff and other related expenses boosted the toll.

Computer services were back online within about three weeks, although one server was deemed unrecoverable.

Lawrence Eta, Toronto’s acting chief information officer, says the city is trying hard to keep ahead of attackers in what he calls “an ongoing battle. It just doesn’t stop.”

A big part of the defence is training staff not to fall for phishing emails. “We are sending staff emails that essentially act like those of intruders and, if staff click on those (emails), they will go through some education” to ensure they don’t fall for a ploy again, he said.

But while Toronto and other municipalities are trying to batten down their digital hatches, sealing every electronic nook and cranny that hackers can exploit, residents are demanding more and more electronic interaction with local governments.

“They want to engage with us electronically, at their convenience, anytime, anywhere,” to pay bills, book recreation classes and more, Eta says. “That means infrastructure, technology applications, solutions, are transitioning at a rapid rate to a digital platform.”

Lake City, Fla., with about 12,000 residents, was hit by hackers June 10. They asked for $718,000 (U.S.) but the ransom was negotiated down to $462,000, city manager Joe Helfenberger told the Star. All but $10,000 was covered by insurance.

Lake City staff are now allowed to use thumb drives and other memory devices only if examined and approved by management. Vendors wanting to do business with the city cannot bid on contracts electronically — they must use old-fashioned mail or drop off paperwork in person.

“We can’t take any risks at all,” Helfenberger says. “There’s always a way to attack, but if we make it a lot harder, maybe they’ll pick easier targets.”