Poor IT planning got in the way of pandemic payments

Best listening experience is on Chrome, Firefox or Safari. Subscribe to Federal Drive’s daily audio interviews on Apple Podcasts or PodcastOne.

Earlier this year, the CARES Act provided billions of dollars in new unemployment payments to Americans whose jobs were affected by the pandemic. But it also created a huge IT challenge for state unemployment agencies who had to quickly retrofit their systems to administer the pandemic payments. A volunteer group of IT experts, called U.S. Digital Response, sprang into action to help states out, but almost immediately, they ran into a problem common across the public sector: the IT systems were closed, proprietary, and very hard to adapt to new problems. Waldo Jaquith is a member of U.S. Digital Response and a veteran of the General Services Administration’s 18-F organization. He spoke with Federal Drive with Tom Temin about those challenges, and how agencies can prevent them in the future.

Interview transcript:

Advertisement

Waldo Jaquith: When dealing with state unemployment insurance software, overwhelmingly they’re using closed source software, which means that the state doesn’t have access to their own software, other vendors don’t have access to look at the software to figure out what’s wrong or how it can be improved. That’s a huge shortcoming. Often, that software will run on an open source source stack using things like Linux, but it would be an enormous help if these states were using open source software so it’d be easier to debug when things go wrong.

Jared Serbu: Why is open source important rather than just continuing to have the source itself be closed, but having the government own it?

Waldo Jaquith: It’s really important that government be able to look at the source code to their own software, because that’s inherent to procuring goods. If you buy cars, say GSA buys a fleet of cars, they need to be able to open up the hood to make sure that they’re getting what they paid for, they’re getting the V8 engine. Do the brakes work? These basic things. When you’re buying closed source software, software that the government can’t see the source of, you can’t tell if it’s garbage or not. That’s a basic procurement failure.

Jared Serbu: Yeah, you’re making a couple of other arguments I think in this blog post. Tell me if I miss characterizing it, but I think it’s lower cost, better code and more secure software.

Waldo Jaquith: I’m happy to expand into those points.

Jared Serbu: Sure. So let me tackle the cost one first. Because one potential issue that comes to mind is I would think that bidders would come to the table with kind of an attitude of, look you’re going to pay one way or the other. If you want to own the code, or if you want the code to be open, you’re going to need to pay more for it upfront to compensate the vendor for the fact that they can’t recoup some of their development costs through license fees or maintenance fees. I guess my question there is, have you seen enough examples of open source clauses in the real world that would show my theory is wrong?

Waldo Jaquith: Sure. So there’s plenty of competition in the open source space, in the software space generally. And there’s nothing about software being open source that makes it more likely that the customer would want to switch away to another vendor. If anything, the sorts of companies inclined to say, no, you can look at our code, it’s fine, are company’s doing better quality work that people don’t want to switch away from. One of the reasons that open source is important though is to reduce that switching cost for a government agency to be able to move from one vendor to another. A lot of the vendor strategy for these these really big, hairy government software programs is lock in, to make it impossible to switch away. Any business strategy that is contingent on let’s make it impossible for our customers to leave us is probably not a great one in the long run. An argument I like to give to government agencies is this one — who’s going to exist longer the state of Virginia’s employment agency or Oracle? Ultimately, I think the state of Virginia is going outlast Oracle or Microsoft or Apple, let’s look at the long run here. Any strategy that results in lock in with that vendor is gonna end badly at some point. A vendor that says yes, we will let you look at the at the source code is a vendor that’s going to be more difficult to get locked in with. What needs to be open source for these unemployment insurance systems and for government generally isn’t the nuts and bolts of what runs infrastructure. It’s when government pays for custom software. When government says, we need to pay a vendor to build this specialized software for us for unemployment insurance system where you’re paying for the time and materials of the vendor to produce software that should be owned by government. That’s different than buying some COTS, some actual off the shelf software. But I think that’s really different than when government wants something custom. Imagine if you’re going out to buy a house, and you pay the entire cost of having a house built, and then you have to pay to rent it. That doesn’t make sense. Nobody would do that. But that’s what government does when it pays for custom software that then doesn’t own the source code.

Jared Serbu: But another point you make that I’ve frankly never thought of is that open source development incentivizes developers to write better code in the first place and to not cut corners because you have to show your work. Talk about that a little bit

Waldo Jaquith: An experience that I had working with modern software development firms as a federal employee that I never expected was that demanding open source software completely changed the incentive model for who bids on those contracts, and then who winds up getting the work. Normally, the best software developers don’t want to toil in obscurity. They want people to be able to see their work and be proud of it. So when it’s established upfront that the work that vendors are bidding on is going to be open sourced and you get the kind of vendors that are proud of their work and they want people to see it. And then the employees at that vendor fight over who gets to work on that project because their work is going up on GitHub. Their future employers get to see their work, their peers get to see their work, and you’re going to get the best work out of them. It’s a chance to create a portfolio that otherwise is nothing but some references on LinkedIn. Instead, they wind up with tens or hundreds of thousands of lines of code that they can show others they can take with them they can be proud of.

Jared Serbu: There are or historically have been debates around security in the open source world in government in the past. A lot of those have been debunked, I think. But let me just kind of try and make a devil’s argument here. I can see the value of open source as far as security goes and kind of mass scale or potentially mass scale software applications. Is it different when you have a super boutique system made or tailored for one state? I mean, they’re obviously big incentives. It’s not hard to imagine a boiler room filled with Russian hackers trying to find vulnerabilities in the code that runs the North Dakota UI system so they can funnel money out of the system. Doesn’t the open source model only work if there is an equally large number of good guys looking at the same code? And is there a developer community that’s large enough, that cares enough about what’s under the hood of North Dakota’s UI system to make a difference?

Waldo Jaquith: States need to insist that the software that drive their UI systems and other crucial systems be open source so that state employees can inspect that code and make sure that it’s any good. When a state procures a new building, there are state employees whose job it is to go in on a regular basis and make sure, are these studs adequate to spec? Are we putting the ceiling joists and the floor joist at the appropriate spacing? Can they bear the appropriate weight? The wood floor, is it dried adequately? Is the moisture level appropriate and so on. There’s no reason why we shouldn’t extend this normal practice of inspecting purchase goods to software, but we often don’t. So it’s incumbent on states to be good purchasers, to be good procurers to have software developers inspect the goods. They can’t rely on white hat hackers just spending their spare time inspecting government software. It is on government to be good customers.

Jared Serbu: There are a lot of open source licenses out there obviously, MIT, GPL, Apache. For government purposes, if I’m a contracting officer that wants to go the open source route, does it matter much which one you use? Or are there any particular characteristics you want to be looking for?

Waldo Jaquith: The question of software licensing can be philosophical with government. I’ll just say that the side I come down on philosophically is that government has no right putting any license on software. A product of government is something that should be in the public domain, and any license is inherently restrictive beyond that. Software should be released into the public domain, and ideally released under Creative Commons zero license because public domain is really a US concept. But the Creative Commons zero license is even more public domain than public domain. Any license is more restrictive than that.

Jared Serbu: I guess one argument against that is if it’s purely public domain, you’ve you’ve opened the door to allow some commercial developer to then make that code proprietary, once they’ve built some extra layers on top of it.

Waldo Jaquith: That would be great for commercial developers to be able to make money on public goods. I think that’s wonderful. If the US or if states can spend money on software, taxpayer dollars, and somebody can then take that good and make money on it, that’s great the same way that government spends a fortune on highways, and then people can use those highways to make money by shipping goods over them, or they can see the specs for highways and use those same specs in order to build private roads. This is a good and natural thing for people to do. I think it’s great for people to be able to build on top of public intellectual property and make money on it. If somebody can take something for free and make it better and sell it, that’s capitalism. That’s great.

Jared Serbu: As we wrap up here, you want to plug US Digital Response real quick. It’s a really interesting and fascinating organization. Can you tell us a little bit about what the organization does?

Waldo Jaquith: Yeah, I’d love to. US Digital Response is an all volunteer organization, just started in March, of a bunch of us, myself included, a lot of technologists across the country who suddenly found themselves desperate to help with the COVID response. We’ve got almost 6000 volunteers who have showed up who want to spend their own time helping local, state and federal government deal with the effects of COVID through nonpartisan fast free assistance. So it’s about quickly delivering critical services and infrastructure to support the needs of the public. So we’ll provide consultation, we’ll provide some project based assistance, basically show up for a while and fix a problem that’s been created or exacerbated by COVID. And then we go away. US Digital Response is not long term staffing, we’re not replacements for government employees. But we do recognize that there are instances where government and the public really needs help, and getting some volunteers to step up to provide technical assistance is what we do best. Anybody who wants help from US Digital Response, any government employee can go to usdigitalresponse.org to find out more. There’s no catch. We have nothing to sell. There’s no strings attached.

Jared Serbu: Yeah, that sounds very much like the digital services model that’s been operating for a few years in the federal government space, except you guys don’t live in the government?.

Waldo Jaquith: It’s like that, I’ll say digital services model, which is taking off at a state level as well, is about a dedicated team with longevity working over the course of years with a bunch of agencies in order to change how government operates. And compared to that, US Digital Response, we’re tourists, we show up, we help with a particular problem, and then we go away. I hope that one of our legacies can be creating support, particularly at a state and local level, for the creation of digital services groups, because that’s really what government needs.

Jared Serbu: Lastly, what kind of takers have you gotten so far? Has it mostly been at the state level?

Waldo Jaquith: Yes. Thanks to the federal laws around this, overwhelmingly federal agencies can’t violate the anti deficiency act to get help from us. So there’s only narrow ways in which we can benefit the federal government. But state and local governments, we’ve worked with many dozens, I think I might be able to say hundreds at this point, across the country. Some projects are as simple as they just need an hour’s advice, and we help them understand how to better procure something or what their technical options are. And in other cases, we have persistent teams building software that can be used and reproduced between states in order to deal with the effects of COVID.

Find out more about U.S. Digital Response.