Big Data

NSF develops predictive model to flag improper payments in grants

NSF’s Payments and Analytics Branch has created an open-source predictive model to flag potential improper payments, and is looking to share its work with oth...

Jory Heckman@jheckmanWFED

January 6, 2021 8:08 am

3 min read

The National Science Foundation is responsible for one in four federal grant dollars that academic institutions receive to pursue basic research.

To get a better handle on this spending, the NSF’s Payments and Analytics Branch has created an open-source predictive model to flag potential improper payments, and is looking to share its work with other grant-giving agencies.

The NSF Payments and Analytics Branch handles the daily operations of the agency’s grant-giving operations, such as processing grant payments and providing technical support to grant recipients for post-award financial processes. It also handles NSF’s obligations under the 2019 Payment Integrity Information Act, which sets requirements for agencies to cut down on improper payments.

More recently, however, branch chief Jesse Simons told Federal News Network that his office is experimenting in the “higher-growth” field of payment analytics and compliance.

“We take a very punk-rock aesthetic to some of this, in that we’re still learning our instruments and we get together and play. Sometimes it sounds great, sometimes it can just be noise. That’s just part of the iterative process that we’re going through,” Simons said in an interview.

While much of this office’s analytics work remains in development, it gained attention in November, when the team behind NSF’s improper payments predictive model won the Association of Government Accountants’ 2020 Innovation Challenge.

AGA presented the award at its Technology & Transformation Summit, held virtually this year. AGA gives the award to a state, local or federal agency that demonstrates “technical solutions to nagging problems or challenges that governments face.”

The model stemmed from NSF exploring several use cases of how to use grantee single audit data from the Federal Audit Clearinghouse, including whether a grantee incurred any kind of “question cost” or improper payment, and combine it with other agency data sets.

Any grant recipient who has spent $750,000 or more in federal funds has to undergo a single audit – criteria that apply to a good portion of NSF’s grant recipients.

“What we wanted to try to do is determine a subset of variables that we could use to accurately forecast whether one of these auditees would have a question cost,” Simons said. “And if we could do that successfully, then we could apply it in more of a forward-looking way.”

NSF narrowed the universe of predictive variables down to 10, and after testing the statistical validity of this subset of variables, found it accurately predicted whether a grantee had a question cost of their audit report 87% of the time.

Simons said that NSF’s grant program hasn’t been susceptible to significant improper payments, but given the higher volume of overall federal spending during the COVID-19 pandemic environment, he said the predictive model gives NSF and other agencies another tool to minimize risk.

“We see it as a valuable tool to potentially provide some business assistance to grantees that might need help. Nevertheless, I think that it’s just another lens for us to take over how we view improper payments. We conduct annual testing, we conduct qualitative risk assessments within our agency by interviewing various stakeholders — so I think the more different views that we can put into some sort of composite viewpoint, the better for us going forward,” he said.

NSF developed its predictive model in R programming language, and has made it available to other agencies on the open-source development platform GitHub.

“When it comes to payment integrity, we all have a shared responsibility to monitor the problem. It’s just at different levels, based on what agency you’re at. So why not try to share, in crowdsourcing, the development of a common solution, because everyone’s required to comply to some extent? But each agency might have its own nuances and complexities, so that’s why I’m excited about getting the opportunity to share this with other agencies and see what other ideas they might be able to add to this,” Simons said.

Simons said his office is looking to improve NSF’s data visualization capabilities and stand up dashboards that can give leadership real-time information about agency operations.