Cybersecurity

Conditions are cooling off for troubled SolarWinds

By any measure software vendor Solar Winds was a high flyer, with many federal customers for its IT managements software. Now the company says it's nearly recov...

Tom Temin@tteminWFED

May 19, 2022 2:20 pm

8 min read

Best listening experience is on Chrome, Firefox or Safari. Subscribe to Federal Drive’s daily audio interviews on Apple Podcasts or PodcastOne.

By any measure software vendor Solar Winds was a high flyer, with many federal customers for its IT managements software. Now the company says it’s nearly recovered from the 2020 Sunburst hack that sent federal agencies fleeing…and became part of the cybersecurity vernacular. The Federal Drive with Tom Temin got more from Solar Winds chief information security officer, Tim Brown.

Interview transcript:

Tom Temin: Mr. Brown, good to have you on.

        Join us May 2 at 2 p.m. ET for a discussion with agency and industry leaders on network modernization strategy, 5G, AI and plans for the future at the Naval Network Warfare Command and NASA, sponsored by T-Mobile. | Register now!

Tim Brown: Thanks, Tom. It’s great to be here.

Tom Temin: And just briefly review what SolarWinds software does. And there’s a lot of products. But why was it so central to the whole supply chain question at that point?

Tim Brown: Yeah, so SolarWinds, you know, it’s the leader in network management, systems management, you know, we have a lot of customers, and we end up being in the middle of network. So that’s why essentially, we were a target for this attack, why it’s called a supply chain attack in this way, is a couple of things. First off, that it didn’t attack our source control system it attacked the build system in line. That’s how we framed it a supply chain attack that our customers framed as a supply chain attack, simply because we were in the middle of, you know, everything that they do in their environment.

Tom Temin: Right. So it’s almost like a utility got hacked in effect.

Tim Brown: Yeah, exactly. Exactly. Because you know, and, you know, we see other things coming up right now, Log4j, for example, and others, so supply chain issues are not going to go away, we just ended up being one of the first out there public supply chain issues like this.

Tom Temin: And did all of the federal customers at that point, just quit using SolarWinds, or did a few of them hang around anyway?

Tim Brown: No, many hung around anyway. So you know, we had great partnership with CISA (Cybersecurity and Infrastructure Security Agency) during the initial cycle. And what we did is we came up with a plan, essentially, and agreed on an approach. You know, the recommendation was that if you’re not running an effected version, if you’re running an older version, then you can upgrade to the latest version. If you are running a version that was tainted, but you didn’t see secondary attacks, then the idea was to look at your environment, make sure that it didn’t see those secondary attacks, and then upgrade. If you were running one of the affected versions, and you saw a secondary attack, then it was investigate your environment and move forward from there. So CISA was a great partner with those recommendations. Then many federal agencies took those recommendations and went with them.

Tom Temin: And what do you feel that you learned about hacking and defenses against this type of thing? Because it seems like every month we learn a new attack vector, such as the Log4j, nobody thought that log files could be the launch pad for attacks.

        Read more: Cybersecurity

Tim Brown: Yeah, one of the things about the threat actor is extremely patient, is extremely thoughtful in what they do, in both the way they attacked us and the code that they put into the environment, things like it wouldn’t run inside of us. Things like it waited 14 days before it started in our environment, very quiet, very stealthy, affected prebuilds and then left. So I think one of the things we have to realize is this level of attack is going to become more common, it’s going to be utilized for ransomware, it’s going to be utilized by nation states, it’s going to be utilized by others, because you know, the payoff at the end can be large. So it’s important to recognize that’s the level of attacker that we have in place.

Tom Temin: And was your first call to Homeland Security, because I’ve heard of this kind of thing before, there was a well known cyber company quite a number of years ago, their first call when hacked was to the National Security Agency because of the nature of what they manufactured.

Tim Brown: Yeah, so you know, it was kind of the first days a little blur, but we had FBI law enforcement on the lines, we had CISA on the lines, we had others on the line, and really just looking at OK, so how do we kind of move forward as the agencies themselves had different missions, right? FBI took a lot of data and were looking at attribution and looking at other things where, you know, CISA was looking on how they can amplify the truth, right, how they can say, this is what you should do. So a lot of good partnership right at the beginning, and it morphed from day one to where we are today.

Tom Temin: We’re speaking with Tim Brown, he’s the chief information security officer at SolarWinds. And you’ve made some corporate readjustments in light of what happened back then, including a new government relationship, government affairs type of person to have on hand. Tell us more about that.

Tim Brown: Yeah, absolutely. So you know, in technology wise, we made a lot of changes, we became much more resilient to these types of attacks. We have a pretty innovative triple build environment, that really no one does. And the reason for that was simply to be able to be more resilient to attack. Now you need three people to collude in order to affect our build. Our new government affairs organization has really helped us get in front of our federal agencies. Talk to them about what we’ve done, talk to them about moving forward, get assistance for those who have not moved forward to move forward. The new institution really opens up additional doors, you know, one of the things that our customers and our general customers really love our products, they’ve done well for them for many, many years. And they’d like to continue using them. So it’s just a matter of letting them know that we are more resilient to attack than most, that we have been able to examine our environment for seven, eight months straight with other folks involved, like KPMG and CrowdStrike in our midst for five months. And then you know, we’ve made changes to product review changes to people, that we just are a lot stronger and more resilient to these types of sophisticated attacks.

Tom Temin: And getting back to that triple build environment that you mentioned, is that a technique or a methodology that you’re making public so that anyone could adopt it?

Tim Brown: Yeah, we actually open sourced a lot of it. So when you look at a triple build, what that means is we build the developer pipeline, that verification pipeline, avalidation pipeline, no one person has access to all three, therefore we compare the results before we ship. So all of those builds need to match. They don’t match, we don’t ship. So collusion would be necessary to affect the build system in the future. So we look at that as basically looking at it assumed breach model. And we did open source that and we open sourced how to compare builds, and how to really go through that process of triple builds.

        Want to stay up to date with the latest federal news and information from all your devices? Download the revamped Federal News Network app

Tom Temin: And what has been the government reaction, I understand some of the agencies that had abandoned the company are saying, well, maybe we should do business again.

Tim Brown: Yeah, absolutely. Absolutely. When they look at alternatives, some have looked at occurrences have come back, some have never left. But just the outreach that we’ve done has been appreciated. We’ve tried to be as transparent and visible about what happened, and about what we’ve been doing. And, you know, that approach has worked well in both the private sector and the public sector. This type of event, it’s terrible when it occurs. And you know, if we can have learnings from it, if we can help both government, public and private sector, learn and move forward and become more resilient, then we all win. So that’s what we’ve been attempting to do. And with that, we have had a good number of customers either come back, turn us back on and those types of things.

Tom Temin: And would you say then, that you didn’t feel like the government was punitive when you notified them and started cooperating with the Cybersecurity and Infrastructure Security Agency, they didn’t send you over to the Federal Trade Commission or something or that type of kind of punitive reaction?

Tim Brown: You know, I think they’ve been kind of respectful for this type of an event can happen with that type of threat actor, right. And it really depends on the actions that you take. And it’s very important to note is that, you know, maybe if our products weren’t so useful, if our products weren’t so good in the environment and so important to the environment. And then they may have decided, OK, well, there’s gonna switch to others. But very difficult to switch to others, you know, with the products that they are integral to what they do, and you know, how they run. So, in general, I think they’re appreciative of the model that we’ve done, and they hope that others would go through and do similar functions.

Tom Temin: Tim Brown is chief information security officer at SolarWinds. Thanks so much for joining me.

Tim Brown: Thank you, Tom. Appreciate it.