The IRS gets its second thumbs-up from oversight officials regarding its handling of this year’s Tax Day computer system glitch, but the agency and its watchdog office fault contractors for not doing more to mitigate the problem.

In the early morning hours of April 17, the IRS detected a firmware bug that disrupted the single busiest tax filing day of the year. The glitch made 59 tax systems, including the Modernized e-File system, unavailable for about 11 hours, and prompted the agency to delay the tax filing season for an extra day.

The Treasury Inspector General for Tax Administration (TIGTA) found that the IRS handled the systems outage in a “timely” and “effective” manner — detecting, assessing and restoring operation by the afternoon of the outage, and processing nearly 4 million tax returns before midnight.

“Established contingency tools, processes, and procedures for mainframe outages served the IRS well during the Tax Day outage,” TIGTA wrote in a report released Monday.

However, the IG office determined that Unisys and IBM could’ve done more to prevent the firmware glitch. More specifically, TIGTA determined that Unisys didn’t uphold the terms of the Enterprise Storage Services (ESS) contract it has held with the IRS since 2012.

“During the Tax Day outage, the contractor failed to meet several service level objectives, and this failure requires the contractor to pay liquidated damages to the government,” the IG office wrote. “Additionally, the contract lacks detailed service level objectives for performance monitoring and incident management.”

IRS Chief Information Officer Gina Garza told TIGTA that the agency has “sought liquidated damages from Unisys in accordance with ESS contract provisions.”

Communication gaps between IRS, vendors

The report also identified communications gaps between the vendors and the IRS.

Just before 3 a.m. on Tax Day, the storage array affected by the bug pinged IBM with an automated “call home” notification, sending an alert to the vendor. TIGTA found the vendor didn’t give the IRS a heads up about the alert.

“IBM never provided an acknowledgment of the performance issue and did not initiate any incident management procedures because IBM identified the call home as a Severity Level 3,” a threat that warrants a response by the end of the next business day, TIGTA wrote.

Shortly after 2 a.m., the Information Technology Operations Command Center (ITOCC), the IRS’ “first line of defense” for IT issues, got a system-generated error message and began to troubleshoot the problem.

“The ITOCC personnel were not sure about whom to contact at Unisys or IBM to begin triage of the mainframes and tried an 800 number they found online for Unisys storage support,” TIGTA wrote.

The report determined that, under the ESS managed services contract, Unisys or IBM, as the managing contractors, should have been first to identify the outage and contact the IRS.

“During this outage, it was the IRS who initially recognized a problem, and the IRS had to reach out and notify its contractors to prompt action on remediation,” TIGTA wrote. “The contractors did not uphold their contractual agreement.”

IBM first discovered the firmware bug associated with the IRS Tax Day outage in June 2017 and developed a fix, which it released to the public as a microcode-bundle patch in November.

In December 2017, the IRS agreed with IBM’s recommendation to remain on the older microcode bundle for the upcoming tax filing season “because it was considered more stable,” TIGTA wrote.

However, IBM product engineers developed a script for another client in January 2018 that experienced an outage from the same firmware bug that the IRS would suffer three months later.

“However, prior to the IRS outage, IBM did not provide the IRS with any details regarding the other client outage or the availability of a script that would have prevented the Tax Day outage,” TIGTA wrote.

GSA commends IRS on achieving processing goals

Earlier this month, the Government Accountability Office commended the IRS for meeting its return processing goals for the tax filing season, “despite multiple challenges,” including the Tax Day glitch.

Garza, the IRS CIO, said the agency is encouraged by TIGTA’s assessment of how it responded to the systems outage.

“While any service disruption is not ideal, we believe the Tax Day outage demonstrated our improved ability to rapidly and successfully respond to major incidents,” Garza wrote. The quick response to this incident and subsequent processing of almost 4 million individual returns by midnight the same day is a testament to the talented and dedicated staff IRS employs.”

The IRS experienced a similar “hardware failure” on Feb. 3, 2016 that affected several tax processing systems, and took about 30 hours to restore operations.

Looking back on that incident and the most recent outage, Garza said her team has continued to strengthen its response to IT challenges.

“This is significant and demonstrates our commitment to process improvements,” she wrote.

The TIGTA report also reasserted that the outage was not caused by a cyber attack.

“The IRS Computer Security Incident Response Center concluded that the outage fit the pattern of a previously known firmware bug and determined that there was no evidence of any breach or cyber threat activity related to this outage,” the report stated.

Brad Bass, a spokesman for Unisys, said in a statement Wednesday that the company worked with the IRS to quickly address the underlying issues, and added that normal operations were successfully restored that same afternoon.

“Unisys takes its role as a partner to the IRS very seriously. Unisys and the IRS have already implemented program enhancements, and we continue to work closely with the IRS to improve the system,” Bass said.

IBM didn’t immediately respond to requests for comment on Tuesday.

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.