It’s been two months since the massive attack through the SolarWinds Orion came to light. And while the full impact of the attack still is unknown, the Austin, Texas, company is going on the offensive.
Beyond naming Sudhakar Ramakrishna as its new CEO in December, the company said it’s taking a multitude of other steps to recreate trust with federal customers.
“We know a lot more than we did a couple of weeks ago. There has been a lot going on from an investigation perspective, including the analysis of tools and from outside companies. The last thing we wanted to do is to put out information that we were not confident about, and I think every day our confidence increases that we are getting a better handle on what happened, and how to prevent it from happening in the future and to help customers prevent it in the future,” said a SolarWinds official, who requested anonymity in order to speak about the ongoing investigation. “When we began our remediation efforts and looked inside our operations, the third parties we brought in discovered the attack had nothing to do with SolarWinds. Our customers understand that this could’ve happened to them as easily as it happened to us. This was a unique and unprecedented incident.”
Experts say recreating that trust with their federal customers means having to go above and beyond with internal changes and taking responsibility for the attack.
“A lot of what they are doing is probably overkill. They are showing they are not just the basics, but changing everything and taking security into overdrive to re-establish that trust,” said Bryson Bort, a senior fellow for cybersecurity and emerging threats at R Street Institute, a think tank, in an interview.
The Cybersecurity and Infrastructure Security (CISA) Agency at the Department of Homeland Security said the attack affected 18,000 public and private sector customers of SolarWinds’ Orion product, including 10 federal agencies.
The SolarWinds official said they have been meeting with defense and civilian customers over the last few months, including the National Security Agency, Army Cyber Command, CISA and many others.
“There is a significant initiative across the federal government to identify areas of concern. We have implemented testing across agencies and branches to make sure they have the latest and greatest version as they ramp back up to using SolarWinds in a careful and methodical way and using our partner community to help them deploy it. We want to make sure our federal customers are secure because they were a primary vector for an attack,” the official said. “We know now this was a targeted attack. It was not meant for ransomware or anything like that, but federal agencies were one of the targets.”
A second SolarWinds official said they also know now that the attack was much broader than SolarWinds Orion. The official said 30% of the incidents researched so far do not have their technology deployed.
SolarWinds is providing agencies with free remediation services through trained and cleared third parties.
“We have taken on or shared the responsibility for securing the Orion deployments or rebuild and redeploying the technology within any given agency environment,” the second official said. “We are making those resources available for agency customers. We actively have partners in place and are working with federal agencies and third parties.”
Experts say it’s a good sign that SolarWinds is helping agencies with remediation efforts as many still are trying to figure out the impact of the attack and whether they will “rip and replace” the company’s software or just update it.
“Any time a company that goes beyond statements and actually puts resources, dollars and/or time into addressing a problem definitely shows a level of commitment and accepting more than formal legalities or contract but more of customer focus,” said Michael Daniel, the former White House cybersecurity coordinator and now president and CEO of the Cyber Threat Alliance. “The damage assessment for agencies will take a long time. Every damage assessment that I’ve ever seen in government took months or years to fully produce the facts about what happened. This is a tough call on the government side of what to actually do. Do you have the confidence that SolarWinds and you have eradicated the adversary? This is a much more complex problem set than whether to just rip and replace. It’s about how much of your network needs to be burned to the ground, discarded and rebuilt from scratch? That is a monumental undertaking.”
The first SolarWinds official said some agencies are considering ripping and replacing, while others are figuring out how to remediate the vulnerabilities.
Larry Clinton, the president and CEO of the Internet Security Alliance (ISA), said part of what the company is doing is trying to maintain their business by being good business partners.
But he and other experts say the after-the-hack efforts, not just providing free third-party assistance, but all the changes they are making to their internal processes calls into question why did it take a massive attack for them — or for any company for that matter — to harden their cybersecurity efforts.
“How successful will they be is hard to say, but the question that is more pressing is did they do the appropriate things on the front end so that they were practicing due diligence to ensure this kind of thing would not have happened?” Clinton said. “The big question for me is did they do enough realizing how critical an element they were in government and industry infrastructure? Were they doing enough on the front end to ensure their own security? SolarWinds should’ve known enough that that were a critical element and should’ve been doing a really good front end security. That’s what I’m more interested than what they were doing on back end.”
The question of what SolarWinds did pre-breach is one experts continue to ask.
Want to stay up to date with the latest federal news and information from all your devices? Download the revamped Federal News Network app
One cyber expert, who requested anonymity, said there has been a lot of “swirl” around SolarWinds pre-breach cyber practices.
“They need to address their reputation problem. While I personally don’t have any knowledge of any of those issues, their reputation was they didn’t take cyber as seriously as they should have, and that they chose to under invest in cyber to prioritize growth in other areas,” the expert said. “Certainly if I were a customer of SolarWinds, I would want them to demonstrate what their cyber practices are, and if I’ve never asked before, I will now. And this applies to many contractors, not just SolarWinds.”
The first SolarWinds official said among the steps the company took are expanding its multi-factor authentication environment, forced password resets across all of its domains ranging from production to lab to staging, and created new software build environments with stricter controls that include zero trust architectures and controls and creating reproducible software builds across multiple pipelines.
“We do not believe our digital code signing certificate was compromised but we did ask for it to be revoked. That was the best way to effectively kill those impacted builds,” the official said. “We want to make sure no one can install that build so as of March 8, the previous code signing certificate will be revoked. Our new software builds are coming and adhere to common criteria and are signed with the new certificate. Those are builds for our federal customers who are upgrading and using testing labs to push through their internal processes.”
R Street’s Bort said while these changes are important, the real question for SolarWinds and really all organizations is how fast can they detect, respond and mitigate future intrusions.
“At the end of the day a determined adversary will always win. If SolarWinds implemented all of these defensive measures two years ago, this still would’ve happened because a nation state that wants to get into a network will,” he said. “Your risk is an embodiment of every vendor in your environment and agencies have to look hard at detect and response. What is your ability to see what happened afterwards? That is the big question.”
Daniel, of the Cyber Threat Alliance, warned against over rotating on the supply chain security issue. He said agencies still need to do the basics of cybersecurity before they spend too much time or energy on supply chain risks.
At the same time, Daniel said SolarWinds will have both a similar and different impact than the hacks suffered by the Office of Personnel Management.
He said SolarWinds, like OPM did, will act a “wake-up call” to non-IT executives who either didn’t understand the supply chain risks or didn’t think it was a big deal for their agency and only a DoD problem.
“You have to make sure you have all the basics in place first before you move on to supply chain risks. Your everyday ransomware and phishing are still your main vector of cyber threats and vulnerabilities,” he said. “Incidents like this highlight the fact this isn’t about a fancy piece of technology. You can’t just buy something to put on the network to address the problem. This is about process, organization, contracts and agreements, which in some cases can be harder. You have to have both technology and organization practices.”