Four bitcoin for your data: How a roll of the dice by the City of Valdez paid off after a cyber attack

It was Friday, July 27 when City of Valdez employees first realized something was terribly wrong with the city's computer network.

10 days before that, on July 17, Mat-Su Borough IT staff discovered activity from malicious code — a Trojan virus — in its network after attempting to install the latest update of its anti-virus software.

That discovery prompted a multi-agency collaboration with state, federal, and private security analysts and engineers, many of whom volunteered their time and expertise to help contain the fallout and restore the borough network.

According to an

given to the Alaska Partnership for Infrastructure Protection in September, the Mat-Su Borough's anti-virus software appeared to be working — detecting and fighting off the Trojan horse over the next week. But what IT staff didn't see at the time was the virus leaving behind other components that spread throughout the network.

Matanuska-Susitna Borough IT Director Eric Wyatt described the threat at the time as a previously unseen combination of malware designed to bypass multiple layers of security.

"This is an Advanced Persistent Threat." Wyatt wrote in a status update to the Mat-Su Assembly on July 30.

[Related:

]

After six days of cat and mouse, on July 23, the malware unleashed its next trick: ransomware called Crypto Locker, which encrypted all 500 Mat-Su Borough desktop workstations, and 120 of 150 borough servers.

"They grab the data, they lock it up, and then they get a nice little message to send a whole lot of bitcoin to some company or somebody in Sweden, and they'll unlock your stuff for just a mere $400,000" said Mat-Su Borough Manager John Moosey during a panel discussion on cybersecurity at the Alaska Municipal League on Wednesday. "So immediately when we got that, we called the FBI."

Just three months prior, the Mat-Su Borough had acquired cybersecurity insurance, which would have covered the approximately 52 bitcoin ransom, worth $400,000 at the time.

[Related:

]

Moosey says he ultimately decided that using taxpayer dollars spent on insurance premiums to pay a ransom to criminals was untenable.

"They do this because they can," Moosey said of cyber criminals. "I did not want to be, as the Mat-Su Borough, an encourager of this activity." Moosey says the decision was made with the knowledge that even if the borough paid the ransom, its data may not be returned anyway.

Borough office phones and computer systems were quickly taken offline to contain the spread of the malware, forcing borough employees to resort to typewriters, hand-written receipts, and runners to deliver messages throughout the borough.

"We became very good at walking to people's office and talking and working through some things as opposed to just sending emails, so that was a good thing," Moosey said.

Within a few days, the Mat-Su Borough's misfortune

, some 250 miles south to the City of Valdez, but not without a silver lining, according to Valdez City Manager Elke Doom, who served as the incident commander for the cyber incident response.

"It didn't take us but a moment to realize that there was a good chance we were being attacked with the same ransomware," Doom said. "We knew from speaking with our neighbors that the first thing you typically do is you go ahead and do an update, you run your virus software, and...that's when it will really infiltrate and seize your system."

In a strange twist though, the FBI said in its September briefing that no evidence was found to indicate that the two attacks were related.

"While the (sic) attacks occurred in a very close timeframe to each other, the ransomware was not the same," said FBI Anchorage Division's Scott Sandback. "The specific infection vectors and malware strains were not the same across the incidents."

The City of Valdez was hit with a variant of ransomware called Hermes, also delivered by a Trojan virus, which encrypted 27 servers and 170 workstations.

The system-wide lockdown prompted Valdez Mayor Pro Tempore Dennis Flemming to declare a local government disaster on July 27.

Despite the different strains of malware, both attacks are believed to have started with phishing emails containing attachments like Microsoft Word or Excel documents with malicious macros.

While Mat-Su Borough officials ultimately decided against paying the exorbitant $400,000 ransom, the City of Valdez considered an alternative avenue.

Valdez Police Chief Bart Hinkle says that after notifying the FBI and reaching out to the Mat-Su Borough and Alaska-based specialists, the police department engaged with a Virgina-based cyber incident response firm specializing in digital forensics.

"We reached out to them (the third-party security company,) and through the dark web, anonymously, they reached out to the cyber attackers, so these people had no idea who we were," Doom said on Wednesday. "And they said 'We understand you've seized my client's system. What is it you're looking for? And they (the attackers) came back with 'Are you a hospital? Are you a bank? What are you?’ Trying to decide what they wanted to charge us for ransom."

The attackers, assuming they had ransomwared a small company, decided on a relatively modest ransom.

"And so that particular day they requested — if you want your information back, four bitcoin," Doom said, the equivalent of $26,623.97 at the time according to the Deputy City Clerk.

With the understanding that the attackers may not actually decrypt the city's data even if paid, city officials carefully weighed the risks of paying.

"Our police officers and our police chief were very concerned — they had lost 15 years worth of data,” Doom said, “and we have a lot of information — that court cases are coming up — put our police department in jeopardy of not being able to supply the information."

Ultimately, the city decided to roll the dice, but not without conditions.

"We didn't just say 'yes,' first they had to pass some tests," Doom said. "We sent — anonymously — documents that they had to unlock that we knew we already had...We said 'Okay, unlock this. Show us that you can really do this.'"

According to a Nov. 13 statement from the City of Valdez, the terms of payment also required verification that the decryption key would not reinfect the city's network.

After demonstrating their ability to decrypt the sample data and receiving the four bitcoin, the attackers provided a decryption key to the City of Valdez, enabling IT staff to begin unlocking its servers and work stations. But the work didn't stop there.

"I'd like to say it's as easy as saying 'We got the code, we unlocked the system, we're good to go.' We were not good to go," Doom said. "We can't trust that data. We have to run it through virus protections, we have to scrub it, we have to put it on a different server, we have to test it because there could be a virus lurking in there and we could be in the same situation again."

After going through what was described as a "deliberately slow and methodical process" of cleaning and verifying the integrity of its data over the course of several weeks, including a full quarantine of all files and databases, the city began implementing changes to its data management regime to help protect against future loss.

"Using lessons learned from this incident, the new system will meet or exceed current industry standards, with more robust security protections and additional efficiencies to better serve our citizens,” Valdez IT Director Matt Osburn said.

The City of Valdez says that to date, no evidence has been found that any city information was leaked or stolen by the attackers. Meanwhile, city officials say that as of Nov. 13, city staff continue to work with limited access to electronic data, documents, and software, which is expected to be fully restored in early 2019.

The cost of the ransomware attack, including the ransom itself, negotiation fees, forensic work, privacy council, and the replacement of equipment is covered by cyber-crimes insurance.

"But we're still vulnerable," Doom said. "We all are. Every single one of us."

There are no guarantees when it comes to cybersecurity, as a constant and neverending arms race between attackers and defenders has yet to deliver an unhackable system.

For average individuals and organizations, even just the thought of getting hacked can induce debilitating fear, analysis paralysis, or a complete disregard for securing their digital lives in the face of innumerable threats.

But cybersecurity consultant Peter House, founder & CEO of Deeptree Inc., who worked on the Mat-Su Borough incident response, says there are simple things you can do to help protect yourself and your organization from cyber intrusions.

"I think the first part is to just understand that we're all up to the task. In a sense, this is evolutionary pressure that we see with any implementation of anything new in the world," House said. "At one point in time, gas was new, cars were new, aviation was new. If you look at the history of the 50s and 60s of flight, we had a high level of incidence of high-publicity crashes, and yet over time what we did was we built out the processes and the instrumentation, and now, flying is as safe as ever."

One simple thing that House says can go a long way is changing the way you create and manage passwords.

"Grabbing four or five different words that you know that nobody else would put together, putting them together as a password, that's a really good password, and it's really not that hard," House said.

Reusing passwords across accounts is also not advised, as well as plugging in random USB drives which may have been left behind to be picked up by an unsuspecting victim.

"So there's different things that we can definitely do, but just understand that our forefathers, our ancestors have gone through this similar kind of change and we can handle this as well too, I think is the beginning of that," House said. "We recognize that taking preventive measures is just part of the ticket we punch to use the technology in and of itself. Gotta have insurance for your car, for a good reason. The same sort of thing applies to technology as well."

To learn more about protecting yourself and your business from ransomware, the Department of Justice offers a guide to

.

For a comprehensive guide about how you can secure your digital life, including how to create and manage passwords, two-factor authentication, and the Dos and Don'ts of desktop and mobile security, visit

.