SolarWinds’ transparency trying to ensure others are safer from cyber attacks

Sudhakar Ramakrishna, the president and CEO at SolarWinds, said the high-profile attack his company experienced, which came to light in December but likely star...

Jason Miller

June 7, 2021 11:42 am

6 min read

The Difference in the SolarWinds Attack

In every experience, whether it's a bug or a security breach, there is something to be learned that will fortify what we can do going forward to make it that much more difficult for a threat actor to perform their duties, so to speak. We are approaching this in exactly the same way. I also have another attitude which is one dissatisfied customer or one impacted customer is one too many. We are keeping the customer in focus and keeping the constant learning of these experiences in focus and continue to improve your processes, your tools, your training, your behaviors, to help to build a more safer set of environments.

Sudhakar Ramakrishna

President and Chief Executive Officer, SolarWinds

Working with Customers and Lessons Learned

The federal government customers are very important to us and I personally have spoken to many of them at this point. And I continue to do so. We're doing this for multiple reasons. One is touching the customers making sure that they are happy and satisfied with our performance and support. Another is articulating to them what we have learned and what we are doing because many of our federal government customers are also having complex supply chains from a software standpoint, and we are trying to drive our learnings into their environments. The third is a two-way open dialog where we can understand their requirements and their concerns better and take action. So I'm again very grateful, I would say is probably the right word to use in this context, to the government customers who have engaged with us who have helped us and who have been patient with us. And many of them actually have now turned our systems back on and are experiencing the benefits of the solutions that we deliver.

Sudhakar Ramakrishna

President and Chief Executive Officer, SolarWinds

Without a doubt, it’s been a busy 2021 for federal and private sector chief information security officers.

While the number of cyber attacks may be the same, or near the same, the severity and the impacts on every day society are not.

From the supply chain attack on SolarWinds to the Microsoft Exchange vulnerability to the PulseSecure VPN, all organizations have been reminded that their dependence on technology can both a blessing and a curse.

What these and so many attacks have taught agencies is the need to be resilient.

The most recent Federal Information Security Management Act (FISMA) report to Congress found agencies are doing a better job managing their cyber risks. In fact, their scores across the NIST Cyber frameworks around identify, protect, detect, respond and recover are among the highest in the last four years.

This means agencies are also doing a better job of communicating to their stakeholders about their planning and performance metrics around their recovery activities based on risk tolerance.

Still, one thing is clear from the last several months, no amount of planning, people or tools will stop a determined adversary.

This is why agencies, and really all organizations, must have confidence in their suppliers and ability to react and recover to threats and attacks.

Sudhakar Ramakrishna, the president and CEO at SolarWinds, said the high-profile attack his company experienced, which came to light in December but likely started a year before, is both a learning experience and an opportunity to double-down on software development approaches.

“In every experience, whether it’s a bug or a security breach, there is something to be learned that will fortify what we can do going forward to make it that much more difficult for a threat actor to perform their duties, so to speak,” Ramakrishna said on the Innovation in Government: Cyber Resiliency show sponsored by Carahsoft. “We are approaching this in exactly the same way. I also have another attitude which is one dissatisfied customer or one impacted customer is one too many. We are keeping the customer in focus and keeping the constant learning of these experiences in focus and continue to improve your processes, your tools, your training, your behaviors, to help to build a more safer set of environments.”

One of the ways SolarWinds is attempting to do just that is through an internal approach it launched after the breach came to light called “secure by design.”

Ramakrishna said this approach includes several steps.

“Security should not be an afterthought of delivering a product so we do penetration testing, we do post software analysis of the security of our software, all those are required. But I would say those are not sufficient and security needs to be planned in or designed in, and that needs to happen at the infrastructure level, that needs to happen in the build systems that need to happen in the build processes, and more broadly, in the consciousness and training of the company,” he said. “The learning, or if you want to think of it as the action that we’re taking, is how do we incorporate that across those dimensions within the entire company.”

One way SolarWinds is incorporating secure by design into its entire company is by using more red teams to more rigorously challenge the company’s plans, policies, development systems by using an approach a hacker or other bad actor would use.

“It is important for us to think like threat actors, no matter the size of the company, or the resources of the company, and provide some ability for the team to do synthetic attacks against ourselves to learn and improve on an ongoing basis. Another is that this is specific to the software bill of materials, and software development itself, we have created three parallel build systems, and the three parallel build systems are in different locations, with different permissions. The whole idea, going back to digitally signing a piece of code and delivering it to customers and giving them the confidence that it’s pristine and is coming from us, the goal is to build across three systems and create cross dependencies, and I should take cross checks across those three environments to make sure that the integrity is not compromised in any one of them,” Ramakrishna said. “If you think about a threat actor, even if they’re able to compromise in one environment, they will have to consistently compromise across three different environments in exactly the same way for us to have a compromised delivery to the field. That required a lot of innovation and that will require a lot of investment on our part. Our goal is that as we perfect it to be able to document it and publish it. This is some of the work that I’m working with some of the federal government agencies, including CISA and others, to articulate what we’re doing.”

All of these and the other actions SolarWinds has been taking over the last few months is to create the trust and confidence with its customers, particularly federal agencies.

He said despite revealing in the last few weeks that fewer than 100 customers were compromised by the attack, SolarWinds helped every customer who asked with applying the patch or rebuilding their systems.

“The federal government customers are very important to us and I personally have spoken to many of them at this point. And I continue to do so,” he said. “We’re doing this for multiple reasons. One is touching the customers making sure that they are happy and satisfied with our performance and support. Another is articulating to them what we have learned and what we are doing because many of our federal government customers are also having complex supply chains from a software standpoint, and we are trying to drive our learnings into their environments. The third is a two-way open dialog where we can understand their requirements and their concerns better and take action. So I’m again very grateful, I would say is probably the right word to use in this context, to the government customers who have engaged with us who have helped us and who have been patient with us. And many of them actually have now turned our systems back on and are experiencing the benefits of the solutions that we deliver.”

Ramakrishna said SolarWinds continues to share its lessons learned with the FBI, with CISA and many others.

He said by being transparent, he hopes others can learn from SolarWinds’ experience and not repeat the same challenges or face the same attacks.

“I noticed that some of the agencies may be restricted in what they can share with the private sector. Let’s say as we engage with the FBI, we continue to inform them of what we learn. But sometimes the relationship can be asymmetric. So the more we can make those relationships symmetric, I think the faster information flow will be and knowledge sharing will be,” he said. “If there is a broad recognition that these things can happen to anyone notwithstanding the best intentions, best practices, best tools, then the level of victim shaming goes down. In a strange way, coming out and informing proactively should be rewarded, not punished, so to speak, either by reputational damage or business damage. That’s the other thing that I think as part of awareness building, we all as a community need to do more to help engage equally accountability methods. Therefore, to the degree that we don’t come out and disclose, to the degree that you don’t come out and comply, they should mean some measures between public and private sectors where accountability is both expected and imposed.”