After reading the Federal Communications Commission’s inspector general report that the commission’s then-chief information officer overstated — maybe even lied — in 2017 about suffering from a distributed denial of service (DDOS) attack when HBO’s John Oliver did a segment on the dangers of ending net neutrality, I had a simple question: Who cares?

It’s not that anyone should condone the hyperbole or the outright attempt to misinform the public by a federal employee. But I just couldn’t get too excited about the report, especially given the fact that David Bray, the FCC CIO at the time, has been gone for more than a year and the net neutrality debate is over for now. On top of that, Bray’s reputation as an honest and caring federal employee leaves me wondering if this was more politics than problem.

So instead of going over all the unpleasant details of the IG’s 106-page report and trying to use our 20/20 hindsight to place blame on Bray and his staff for jumping to conclusions or sticking to a narrative that obviously had gone off course, I’ve asked former federal CIOs to offer some lessons learned to current federal executives about dealing with a similar situation that is likely to happen again.

Lesson No. 1: Stay calm

Sounds obvious, but panicking or jumping to conclusions based on an emotional reaction or political pressure is a real threat. Jonathan Alboum, the former CIO at the Agriculture Department and now chief technology officer at Veritas, said CIOs need space to assess the situation.

“It’s too easy to jump to conclusions, which is a sign that you don’t have a great understanding of your IT environment,” he said. “The better you understand what and where your data is and how your systems function, the faster you can get to the root cause.”

Tony Scott, the former federal CIO, said IT executives need to be careful in what they say without a thorough understanding of the facts because Congress is listening. Be cognizant that stating something as fact could have long-term repercussions.

Lesson No. 2: Have a plan

Several former CIOs highlighted the need to know what to do when something bad happens. It’s more than who takes the system offline or what’s the best number to reach the Homeland Security Department, but you need to know all of the things that happen a few days, a few weeks and a few months after a cyber event.

“Ensure that effective incident response policies and standard operating procedures exist,” said Simon Szykman, the former Commerce Department CIO and now chief technology officer at Attain. “A strong policy/process framework will help ensure that incident response actions and reporting match the actual circumstances surrounding an incident, making it less likely that follow-on activities or communications get off track.”

The plan also should dip into the system architecture environment, said Shawn Kingsberry, the former CIO at the Recovery Accountability and Transparency Board and now vice president for digital government and citizen services for Unisys Global Public Sector.

“It is also clear that a plan should be in place to protect expected demand volumes, and a contingency plan should be also available, should the volumes greatly exceed anticipated demand,” he said. “This should become a part of the DNA of the organization and how they execute.”

Alboum said while outages are not common and can happen for a variety of reasons, having a strong resiliency program will help agencies turn their focus from what happened to the speed with which the organization can respond, correct the issue and resume successful operations.

Lesson No. 3: Obtain multiple points of view

The one big mistake Bray and the FCC made was failing to alert the Homeland Security Department’s U.S. Computer Emergency Response Team (U.S.-CERT) if this was indeed a DDOS attack. Federal law and regulations require agencies to contact US-CERT should a major incident happen, and the taking down of the commission’s comment system would seem to fit the bill. The former CIOs said having that third party review audit logs and data will help put you on more solid ground.

“You need to get multiple points of view or multiple analyses of the problem,” Scott said. “You shouldn’t rely on a single source. Triangulate what actually happened because it could easily be a different technical issue than you first thought.”

Szykman added having that independent, objective analysis, whether it’s through an informal request like a chief information security officer from another agency or through formal channels to US-CERT, puts the CIO in a better place when discussing what happened with agency executive, Office of Management and Budget officials or lawmakers.

Lesson No. 4—Inspire trust

This is the one thing Bray always had going for him, the federal IT community, FCC leadership had confidence in what he did and what he said. So when he said the attack appeared to be a DDOS type of attack, FCC commissioners and lawmakers had every reason to believe him.

Alboum said trust from being correct as much as it does from simply saying “I don’t know right now.”

“At the same time, the leader must be able to articulate the things they are doing to get to the bottom of the situation,” he said. “However, if you don’t have visibility into your data or understand how your systems are working, then it’s much harder to articulate how you are going to resolve a crisis.”

Scott said sometimes you can inspire trust by admitting you were wrong with your first conclusion.

“CIOs have learned over and over again that what things first appear to be often change when the full facts become available,” he said. “Most of the time, your first inclination is probably wrong.”

Lesson No. 5: Data rules the day

This was another shortfall of Bray and his staff. FCC IT staff told auditors they didn’t have enough information from event logs to make any thorough determination of what happened. That led auditors and other experts to question why they were so sure the comment system suffered a DDOS attack versus just a significant spike in usage after the John Oliver show.

Every former CIO said having the right data and being able to present it to agency leadership and lawmakers is how you keep the conversation moving forward.

“Government organizations today need to be prepared and make strategic choices around data collection, data storage, utilization and the location of that data – and how it can be safeguarded against system outages or cyber-attacks. Recovery from an a system or data center outage is exacerbated by the complexity that has been built into our IT environments. Getting to the bottom of a problem in our complex, hybrid, multi-cloud world, is not easy. There too many moving pieces,” Alboum said. “To understand the true cause of a system or data center outage or cyber-attack takes time. It also takes understanding of how all the pieces fit together. If you don’t understand where your data is located, you will be at a loss to answer questions from your leadership or respond to the public in the fastest and best way possible.”

Kingsberry added if agencies are building systems using digital standards then when looking at an incident, you will have consistent development and security operational services to understand the problem.

“Logs from applications, network devices and security tools can provide data to confirm or invalidate assumptions regarding the nature/source of an incident,” Szykman said. “Don’t draw firm conclusions prior to completing a sound analysis, and be open to revising hypotheses if necessary.”

And finally Scott said having the right data also lets you fill all the gaps in communication with auditors, Congress, agency leaders and cyber staff.

In the end, communicating with all the stakeholders from a position of knowledge will ensure you can avoid getting caught up in a political firestorm.

Read more of the Reporter’s Notebook

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.