The Commerce Department’s Economic Development Administration spent almost half of its IT budget last year to remediate a cyber attack that barely happened.

Commerce’s inspector general found in a report released last week a string of errors and miscommunications led to EDA’s overreaction of removing employee email and website access from the main agency network.

EDA’s drastic steps to limit the damage by shutting down much of the access to the main Herbert Hoover Building network ended up costing the agency more than $2.7 million to clean up and reconfigure its network and computers. The IG said the bureau destroyed more than $170,000 in IT equipment, including desktop computers, printers, keyboards and mice.

But the IG found no evidence of a large scale malware attack, and in fact, a series of blunders between EDA and the Commerce Department’s incident response team led to what turned out to be poor decisions by senior leaders, a waste of money and resources and potentially a disruption to EDA’s mission.

The IG’s report on the Economic Development Administration’s year-long saga, however, provides an opportunity for other agencies to learn from and not make the same costly mistakes the next time a cyber attack happens.

“There’s no need to run away and react extremely when a cyber attack occurs,” said Venu Ayala, president and CEO of Zen Strategics, a cybersecurity services firm. “If you have a good infrastructure, if you have the right processes and tools, then you likely are well prepared to handle any cyber attack.”

Ayala, who worked for the Homeland Security Department’s cybersecurity office, said agencies first need to determine which systems are under attack and whether they are critical or non-critical mission-related systems.

Initial determination was wrong

That initial triage determines just how serious the problems are and what steps the organization should take first.

In EDA’s case, that process broke down, the inspector general found. EDA thought the malware attack infected 146 out of 250 components — more than half of their systems.

The problems stemmed from an initial inaccurate analysis by the Commerce incident response team and that led to a misunderstanding by EDA that a widespread malware infection had occurred.

When the incident response team tried to correct its error by saying it was just two components, not 146, EDA IT folks didn’t understand the new notification and the incident response team didn’t adequately address the mistake either, the IG found.

The IG said this misunderstanding wasn’t realized until the following December when auditors completed their validation of the events and found the miscommunication that started EDA down an unnecessary path.

A lack of understanding about administration’s network set up added to EDA’s challenges. The IG found the supposedly infected email server was up-to-date with its anti-virus and other cyber protections. And maybe more importantly, the agency’s outbound traffic doesn’t pass through any other systems before reaching the public Internet, so even if there was an infection it couldn’t spread to other parts of the network, auditor say.

The final straw that pushed EDA to take the drastic measures it did was a history of poor cybersecurity. The chief information officer, who was relatively new to the bureau, thought such an attack was highly probable and possible based on what he knew about EDA’s long-standing cyber challenges. The IG said EDA had outstanding cyber vulnerabilities, which auditors first highlighted four years ago.

Communications breakdown

All of these things conspired to lead EDA to spend money, destroy IT equipment, reconfigure and remediate systems that weren’t infected and overall disrupt its IT network and systems for more than a year.

“EDA accepts the Inspector General’s recommendations regarding its information technology incident,” said an EDA spokesman in an email response to questions from Federal News Radio. “We take the privacy and IT security of all our employees, grantees and other partners seriously, which is why the agency acted out of an abundance of caution based on the information provided to us.”

Joseph Beal, the chief information security officer at CCSi, said agencies need a combination of processes and documentation, but the most important piece is communication.

“We learned in this and many other cases, you just can’t throw people and technology at a problem. More so, you need to sometimes just pick up the phone and say, ‘This is exactly what we are seeing and this is what we have. We need to at least try to define some way of containing what we have and what is the impact of our business mission,'” said Beal, who has worked with several agencies to address cybersecurity challenges, including DHS, Transportation and the Marine Corps. “I think that is where this situation fell short. You see there were multiple emails that went back and forth between multiple constituents. After four or five emails, there’s a rule of thumb for my guys that says if you have four or five emails from someone, you need to pick up the phone so you have clear understanding of the issue and you know what’s going on.”

Beal said agencies should use the National Institute of Standards and Technology’s recently updated publication, SP 800-61, revision 2, to improve how they handle cyber incidents.

The special publication recommends agencies establish clear procedures for prioritizing the handling of incidents and implement effective methods of collecting, analyzing and reporting data. NIST also says agencies need to build relationships and establish suitable means of communication with other internal groups and external groups, including human resources, legal and law enforcement.

Beal said from what he sees from agencies is they don’t test out or are not familiar enough with their incident response plans.

“I think if you look back at everything that happened throughout the evolution of the event for EDA, there was a true break in process, confidence in the people that they had on the ground to get the job done, and when all else failed, they threw everything plus the kitchen sink at it, which wasn’t needed,” he said. “It goes back to saying, ‘What is the true risk of any issue we have across our network? What are the threats that actually are hampering our ability to complete our mission?’ They were in a Catch-22, where they were using a shared network that had multiple entities on it so they had to take direct action. But you should know [that] basically from assessing and being able to understand your network, the ins and outs, and having the right people in place, who are monitoring, protecting and doing the analysis of your system. You should know what type of incident would have the most catastrophic effect on your system.”

Network construction is the problem

Ayala said agencies need to understand the severity of the risk associated with losing a system to a cyber attack, and what the criticality of the systems the attack is infecting is most important.

He said agencies tend to struggle with this concept because of the way networks are constructed.

“The network control systems that we use nowadays have grown very large and very complex, and basically are based on a centralized computer hub or a brain controlling the whole network,” Ayala said. “So what happens if a hack happens into one of the areas, pretty much the whole system is down, and people start talking about whether they should isolate the whole system, the down time and the loss of productivity.”

He added agencies should move their systems toward a distributed network of control systems, which is a system of systems.

“It allows all the various agents, a bunch of mini-brains so to speak, to coordinate their activities,” Ayala said. “So, if a hack happens with one of the systems, it’s only that particular system that is isolated and the rest of the mission and businesses are going on undisturbed.”

It took nearly a year for EDA to recover from the steps took to deal with the cyber attack.

The spokesman said EDA has focused on fully recovering its IT functionality in the most secure, efficient and cost-effective manner possible. The agency is implementing many of the recommendations in the OIG report to ensure that the agency’s IT security is strengthened, including moving to the Commerce-wide shared services.

“EDA’s IT operations, including email, and many of its business applications, are integrated into the Department of Commerce’s IT operating systems which allow us to leverage shared services and ensure a greater level of IT security,” he said. “It is important to note that despite the IT disruptions, at no time did EDA’s important work cease. In fact, EDA continued to provide its grantees and applicants with excellent customer service, and all agency payments were made on time, grant programs were announced and implemented within the planned and required framework, and we continued to make critical investments in distressed communities across the country.”

RELATED STORIES:

Commerce considering managed services to fix cyber weaknesses

Commerce shuts down economic development site due to virus

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.