State Department’s cyber center reducing noise, trying to prevent the ‘bang’

Time is the cyber defenders’ biggest enemy. Not China or Russia or any hacker group.

How long it takes to detect and mitigate a cyber attack is the difference between having “an incident” and being hacked.

For the Foreign Affairs Cybersecurity Center at the State Department, everything is about cutting days to hours, hours to minutes and minutes to seconds to protect the 270 posts, facilities and offices worldwide.

The 95 employees and contractors running cyber operations aren’t clock watchers by any means. Working 24/7/365 doesn’t allow for that kind of luxury.

When a suspicious email comes into the on-premise or cloud network, they block it, put it in a sandbox and break it open looking for malicious code.

When the FACC needs to block an internet protocol (IP) address, it takes minutes from the time the analyst sends the alert to the firewall team until it’s implemented.

Reports to the U.S. Cyber Emergency Readiness Team (US-CERT) at the Homeland Security Department’s Cybersecurity and Infrastructure Security Agency that used to take 45 to 50 minutes manually now take four to five seconds.

“There are a couple of different ways we measure speed. You can measure time to alerting from when an incident happens to when you receive the alert,” said Gharun Lacy, the deputy assistant secretary and assistant director for cyber and technology security at the Foreign Affairs Cybersecurity Center at the State Department, on Ask the CIO. “I think that’s where we’ve shown the greatest improvement at present; from the time we see alerts to the time we’ve stood up the appropriate response and gotten the right skill sets to bear is minutes. That’s exactly where we want to be, especially sitting here at the Department of State, the center of foreign policy, where we see a lot of action.”

A lot of action may be an understatement. When it comes to foreign adversaries, State may be the highest profile target across government.

“We see things customized for us and our environment,” Lacy said during a tour of the center given exclusively to Federal News Network. “[This is why] one of the things we’re looking at as we mature is trying to get to that almost predictive phase. That’s where we have a robust cyber intelligence ingestion program. We ingest cyber intelligence from multiple sources, internal to the State Department, external vendors, the intelligence community, other federal partners, being able to ingest that real-time threat information to see how the environment is evolving, and then sifting through that noise and figuring out how it’s specific to the State Department’s environment. Then getting that to our network defenders before ‘bang’ is really where we’re looking to go. We’ve got a host of activities in flight where we’re looking to bring more automation to make these things faster, leveraging more cloud capabilities, as well as being able to leverage that internal and external threat feeds.”

The concept of ‘bang’ refers to a successful cyber attack.

The FACC’s workforce is split among five teams:

• Incident response
• Monitoring analysts
• Malware analysis
• Development
• Threat integration

These teams rely on data from open source providers, the intelligence community, CISA and private sector threat feeds.

“We bring in 12 terabytes of data a day in our cyber logs,” said Roy Matthews, the cyber operations chief of the FACC, during the tour. “We have a team that is dedicated to enriching the data, configuring development tools and strategies to parse the data.”

FACC’s role expanding

Data, as most cyber defenders know, is at the heart of defensive operations.

The FACC features a data center with 900 terabytes of storage with another 12T of data coming in each day.

At the same time, the center is leaning on cloud tools to crunch all of that data.

Rob Eickenberg, who leads the cyber incident response team for the FACC, said the every-changing threat and ever-increasing amount of data has required his team to focus on security orchestration, automation and response (SOAR) over the last few years.

“We need to automate as much as possible and get out of doing repetitive tasks,” he said.

Lacy added applying artificial intelligence and machine learning tools are on the horizon given the velocity and volume of data coming into the center.

Lacy said since 2019 the center’s role has increased dramatically.

“Each facility operates their network in a unique manner so getting visibility is difficult,” Lacy said during the tour. “There have been a lot of initiatives the department has put together between Diplomatic Security, Information Resources Management, the Center for Data Analytics and others. So we have to get some consistency in those data logs. We know it’s not a technology issue. It’s a people issue.”

The center’s focus on the people as much as the technology also is trying to address a long standing challenge — whether real or perceived — between Diplomatic Security and Information Resources Management, where State’s chief information officer resides, and over who has ultimate responsibility, accountability and oversight for cyber issues.

Connecting the dots across State

Multiple State CIOs expressed frustration over this systemic disconnect.

Lacy and Al Bowden, the deputy CIO at the State Department, both pointed to recent cyber events to demonstrate the disconnects of data and perspectives have diminished since State brought the FACC to full operational capability in 2016.

Lacy, who has been in his current role for about a year and with State for more than two decades, said he’s seen no difference between employees of DS, IRM or any of the other bureaus, especially when during a cyber emergency.

“Historically, and going back to your comment that there are axes to grind between IRM and DS, a little bit of that is true, but I don’t think it’s as extreme as what the word on the street may be. I think that oftentimes those disconnects were the result of data and perspective,” Bowden said. “One of the things that my colleagues and DS obviously know is threat; this is what they do, and the CIO organization has the heavy weight of operational responsibilities and keeping the department connected around the globe. So yeah, oftentimes, there were instances where those two things conflicted. One of the things that this center has done has been improving the partnership with DS. It has provided a nucleus so that we can take a look at the data and the perspective at the same time and as an agency on the cyber front, make data driven decisions, not decisions that are based on perspectives that may or may not have been armed with all the data in terms of spread, in terms of operational impact.”

He added the center has led to a significant enhancement of State’s cyber defenses to protect operational mission areas because it’s created a shared understanding of what’s going on across the department.

Going after systemic issues

Lacy highlighted two recent examples where the center played a key role in breaking down barriers during a cyber incident.

The first came three months into the job when the cyber task force came together to respond to a cyber incident.

“Let’s be honest, we’ve all heard the rumors and rumblings about the discord. I think bifurcation was the phrase that I heard a lot thrown around. But I sat in that room with that task force and watched that task force work. And this absolutely works,” he said. “I don’t know how anyone that sits in this room and watches this team of people across the entire department go after an adversary and win could get any other notion.”

The second example Lacy pointed to comes from the task force themselves, and his desire to spread this approach beyond just incident response to address more systemic issues across the department.

“The team came up with a great solution: we had our cyber resiliency sprints forum. We’ve done about 20 of them over 100 lines of effort, long standing issues that were able to be knocked out by bringing that same type of team focus, clearly defined goals, and it worked,” he said. “So me being the new guy, I was like, I don’t see what the problem is. We’re getting things done here.”

The cyber resiliency sprints, which started in late 2021, focused on getting more visibility into State’s network, particularly the non-centralized, non-enterprise networks as well as bringing in some more endpoint detection and response tools to areas of the network that they didn’t have them in before.

“The sprints have actually grown in scope to where we can handle technical or even policy driven issues. One of the good things about the sprints is when each sprint objective is thought up, we ask what roadblocks are anticipated? The leadership buttons that need to be pushed to clear those roadblocks are there and it’s something that gets addressed in every sprint scrum, which only last 15 minutes,” Lacy said. “What I love about it with the agility of it is it’s not designed to create meetings. It’s designed to create results in as quick a manner as possible.”

He added there are four or five different cyber resiliency lines of effort that work for two weeks to address a problem. The sprints are agile enough to change as the different threats emerge, and don’t typically require more resources to complete.

Mission-centric approach to cyber

Bowden added part of the issue that the FACC is helping to solve is around mission alignment, from strategic, investment and operational perspectives.

He said the enterprise chief information security officer, which State established in December 2020 to oversee all aspects of the department’s cybersecurity, tries to address that misalignment that has occurred at times over the years.

“When everyone starts to understand how they can contribute to that mission, and to that mission alignment, a lot of things, that turbulence, if you will, and a lot of things that historically were a little rough, started to be not so rough, because I think one thing that we all understand and agree on is with the cyber landscape today there is enough work, anxiety, pain, to be spread across the entirety of the department, and it does absolutely take all of us pulling together to bring things,” Bowden said.

This idea of alignment becomes successful during those cyber resilience sprints. Bowden said if the CIO is trying to get achieved a cyber defense goal that is taking longer than expected, the sprint team can pull the right strings.

“That resiliency sprint framework provides an opportunity to drop that in there and that’s where that absolute mission alignment comes from,” he said. “Every single individual within the various organizations participating in that framework now has one voice, which is saying, ‘do this by this date.’ So it also helps us prioritize the plethora of work that we need to do.”

Lacy added a secondary benefit to the sprint is it lets teams from across the department work together in a way they may not work with each other in the past.

This is also why State rotates which person or organization is in charge of the sprint.

“We make sure that everyone’s getting an opportunity to work on one of those teams, and it’s a great development tool to help our talent really get out, step outside of their traditional lane and learn more about the department,” he said. “I know we also have CISA directives that we ingest into the sprint framework, and we’ve actually presented the sprints to CISA as a method that they can go on the road and show other agencies how to deploy and work with some of this as other directives that come down the pike.”

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.