The government gets a bad rap for failing to meet deadlines and messing up major system overhauls.

The history of federal IT is littered with these examples.

But it’s rare we get insight into how one agency pulls a potential major blunder from the jaws of failure. But that’s what happened to the Federal Communications Commission earlier this week during a major systems upgrade.

The FCC kicked-off its modernization effort Sept. 2, shifting more than 200 servers and transferring more 400 applications associated with those servers to a commercial cloud.

But the effort took two more days than planned to get the systems back online, causing delays for FCC customers in filing reports and other documents.

Instead of the IT problems just becoming another example of government and contractor ineptitude, an email obtained by Federal News Radio sent to FCC employees and partners shows the effort to rescue the project.

“We could have always asked for more time up front, possibly padded our schedules; instead we chose to be ambitious in our timelines because that’s what a startup-mentality culture does,” wrote David Bray, the FCC chief information officer in the email. “We aimed high, adjusted, pivoted, and succeeded in our outcomes. If those in public service take from what we did as a ‘Team’ this summer — and from it and see that it is okay to take risks as long as you are fully committed to seeing them through and getting it done — then we will have helped ‘hack the bureaucracy’ for the better.”

Digging a bit deeper into the efforts, Federal News Radio has learned the FCC and contractor team, led by IBM, worked for 55 hours straight to straighten out the problems.

Sources said the problems stemmed from contractors at the IBM data center had mixed up hundreds of server cables creating a configuration that was unworkable.

Instead of blaming the contractor or blaming the government, FCC team members worked 24/7, and in some cases, sources say, slept in the server room.

By Sept. 10, systems were back online and available to FCC customers.

The FCC said in a release that the move to a commercial cloud service provider would help reduce the costs to maintain the systems, improve their resiliency and allow it to shift legacy applications to cloud solutions in the long term.

Bray told Federal News Radio in August that he plans to get the FCC out of running its own infrastructure. He said the end goal is to create a modern, secure infrastructure using equipment, applications and systems that focus on mission success.

Bray now is plotting phase 3 of this modernization effort, which will include rewriting IT systems to employ reusable “service catalog” of modular components across the FCC.

The FCC’s accomplishment may be something that happens often in government where smart, technical federal employees successfully complete or rescue an IT project. The problem is we don’t hear about it often enough, and instead all we hear the politicals complain about the state of federal IT and how the private sector is so far ahead of the government.

Well, that dialogue could change if the federal IT community does a better job of celebrating successes and stop retreating when there is a failure.

So, send me your successes — on the record, on background, under cover of night in a dark parking garage in Rosslyn. But let’s change the federal IT dialogue together.

This post is part of Jason Miller’s Inside the Reporter’s Notebook feature. Read more from this edition of Jason’s Notebook.

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.