How the cloud saved USCIS from the post-shutdown blues

When the U.S. Citizenship and Immigration Service started modernizing its E-Verify system, little did executives know that decisions made over the last few years would save them from becoming just another federal program facing a backlog of tens of thousands or more.

For all the talk about the cloud and the flexibility and agility it brings, there are few examples in the government where this concept played out in such a dramatic way as it did with USCIS after the government shutdown.

Eric Jeanmaire, the former division chief for Identity, Records and National Security Delivery in U.S. Citizenship and Immigration Service and now the CEO of Finality, said moving E-Verify to the cloud made surviving the government shutdown not only possible, but a success.

The 35-day government shutdown caused a massive backlog of cases from employers running new hires through the E-Verify system, which validates an employee’s lawful ability to work in the United States.

“With E-Verify being appropriated, we had to turn off the service. So all employers across the country were locked out of E-Verify, while they are still going on to hire individuals, they cannot run these individuals through the system to validate employment,” Jeanmaire said on Ask the CIO.

Whereas normally USCIS processes about half a million cases per week, the first week after the shutdown it was processing more than 2 million cases. At its peak, when the agency turned E-Verify back on it saw nearly 1,500 cases per minute, versus the normal rate of 100-to-150 cases per minute—a 15 fold increase of the amount of traffic on the system.

To handle that huge increase in traffic, Jeanmaire’s team didn’t have to buy more servers or figure out how to reconfigure its data center to make better use of underutilized servers. Rather, USCIS turned to its cloud provider for help.

“The technical piece of that preparation was as simple as a few minutes of scaling up additional servers. But that’s the boring part of the discussion and also the easiest part was the technical one. The challenge more so was us trying to estimate what we needed to scale up the application too,” he said. “Every day the shutdown continued was another day we needed to add to our calculation. We were worried about APIs and web services environment. A lot of our large customers have built their own software to interface with E-Verify. During that 35 days, business was usual for those large customers and have 35 days of hires cueing up in their system. And as soon as we turned on the system, we would have that cue starting to drain with thousands upon thousands of cases beginning to run.”

Jeanmaire said USCIS prepared for about 30 times the typical amount of cases coming through the system. While the actual increase was “only” 15 times, Jeanmaire said the agency wanted to make sure they were ready for whatever happened.

“This is one of the things that really benefited us versus being in a data center. The costs leveled out since we were mostly down during the shutdown, we weren’t using a lot of compute power. Ultimately, we did process the same number of cases, just in a very concentrated period of time,” he said. “I think that’s really the value proposition of the cloud, using that throughput when you need it and turning it off when you don’t. This is exactly why doing something similar in a data center would’ve been extremely cost prohibitive because we would’ve had to buy enough hardware all the way up to our maximum peak load, but then there is no turning it off. Once you’ve bought that hardware, you’ve sunk that cost already.”

Jeanmaire said the post-shutdown tsunami of cases was the first time USCIS tested the agility and flexibility of the cloud outside of a simulated environment.

“The system was very stable the entire time,” he said. “That was a great example of us being able to provide value to employers and have a very nice, stable service to the public despite having that very large shutdown period.”

Jeanmaire said USCIS monitored the system’s performance in real time and could’ve added more compute power if necessary.

If E-Verify wasn’t in the cloud, Jeanmaire said the backlog would have lasted for months.

“By the end of the second week, we pretty much had the entire backlog of cases come through and we processed them,” he said. “Really after the first couple of days, we were at a point where we were very comfortable and we knew we could handle pretty much whatever was thrown at us. At the end of week two, we were getting back to our normal numbers [of cases].”

Even though Jeanmaire decided to go back to the private sector, E-Verify has a well designed roadmap for continued modernization.

He said before leaving USCIS that adding more automation and improving the system’s accuracy is a major priority.

USCIS has been improving its person-centric or immigration services, which is a mainframe system from the 1980s that the agency has been modernizing for some time.

“We’ve taken a modern approach to that [central] index [system] and brought in roughly 1 billion forms or records and this system aggregates all of this information and groups it together by individual. It lets us look at a person holistically about their interaction with USCIS,” he said. “That is important to E-Verify because E-Verify is the validation that someone has the status that they say they have. If we have that index pre-aggregated, all we have to do is a quick look up. In our legacy environment, we’d have go through very complicated logic and search through all our systems and then do some internal data calls. This didn’t scale well and was very slow.”

He said the improvements to the person-centric services system will give USCIS more accuracy and require less manual processing.

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.