Study of GAO bid protest data shows why it’s hard to interpret the numbers
One of the great parlor games in federal procurement is adding up the number of award protests every year. But because of the way the Government Accountability ...
Best listening experience is on Chrome, Firefox or Safari. Subscribe to Federal Drive’s daily audio interviews on Apple Podcasts or PodcastOne.
One of the great parlor games in federal procurement is adding up the number of award protests every year. But because of the way the Government Accountability Office compiles the numbers, it’s hard to understand the real patterns. This is the topic of a recently published paper. The Federal Drive with Tom Temin got a summary from Smith Pachter McWhorter attorney Joseph Petrillo.
Interview transcript:
Tom Temin: Joe, I tried to read the paper, but you lawyers keep everything behind the Bar Association paywall, but you have downloaded and compiled some of the findings for us. And what’s this all about?
Joseph Petrillo: Well, this is the result of a study of GAO bid protest data and an analysis of how useful the data is, which was published in the public contract law journal. And it’s called “Data Scarcity in Bid Protests, Problems and Proposed Solutions.” And the author is a third year law student at George Washington University named Will Dawson. And he’s done a very good job, I think, of looking at the data, doing an analysis of it, and providing more interestingly, some ideas about how to improve the system to get better policy outcomes from better data.
There are some well known inadequacies in the GAO bid protest statistics. First of all, the statistics count protests, not protested solicitations or even protesters. So what that means is, you may very well have amended protests filed by a protester. And each of those is counted as a separate protest. So it’s not at all uncommon for a protester to amend its protest after reviewing the report. And there may be other amendments as well. So we start from a number that’s not really very helpful.
Tom Temin: In other words, there’s double counting.
Joseph Petrillo: Yeah, multiple counting double, triple. And then sometimes there are four protests filed by a single protester in the course of a protest. And then, of course, on top of that the issue that there are maybe multiple protesters filing protests on an individual solicitation. So we don’t really have a good handle on how many solicitations are being protested at GAO. In addition to that, the other main problem is that GAO tends to bundle dissimilar things in the same category that they’re reporting on. For instance, dismissals, protests are listed as having been dismissed a certain number each year. But those include both protests that GAO dismisses as untimely, or legally inadequate. And also protests that are voluntarily withdrawn by the protester. Which I guess looks at the administrative report says, “we’re not going to win this one,” and pulls it back. So those are two very, very dissimilar things with different policy implications.
Tom Temin: Right. So therefore, you can’t really draw safe conclusions from the yearly data that comes out,
Joseph Petrillo: Right, and there are other things that aren’t counted at all. So because of this, from time to time, folks have done an analysis of this data and gone into the actual decisions and tried to tease out some more information. Very good report in that area was done by Dan Gordon, my colleague, Dan Ramish, did another but Will Dawson did his own. He did a deep dive into GAO protests and data. And I think he found some interesting things. When he looked at the actual protest decisions from 2000 through 2020. He looked at 26 categories of why protests were sustained. And three of those categories accounted for half of the sustained protest. They were unreasonable application of the evaluation criteria, including situations where the protester was treated in a disparate or unfair fashion. Secondly, award decisions based on reasons not stated in the solicitation. So you’ve got evaluation criteria, you’ve got requirements in the solicitation, but the decision is based on something else. And thirdly, there is an important issue relating to the award, but there’s no support in the record for it. For example, the awarding official has advice coming up from advisory boards and panels in the evaluation doesn’t take the advice, but doesn’t explain why, or he doesn’t do it in inadequate fashion. So those three reasons account for fully half of the sustained protests in this 20 year period. There are some other interesting things that he chose, for instance, 24% of protests that allege an organizational conflict of interest are sustained, and that’s a very high rate of sustains.
Tom Temin: We’re speaking with Joe Petrillo. He’s a procurement attorney with Smith Pachter McWhorter. And so there’s one unifying theme to those causes of protest sustainment, and that seems to be over use of human discretion by contracting officers. These are not technical issues. These are discretionary issues.
Joseph Petrillo: They’re discretionary issues where the discretion either isn’t explained, or it isn’t grounded in the rules of the protest process. I think you’re right. That’s a good observation. So this is the type of data, if it were accumulated and recorded on a regular basis over a long period of time, would help better inform policy decisions,
Tom Temin: How did Dawson get the data in that case?
Joseph Petrillo: In this case, he was able to look at the actual protest decisions, he ran a computer program, he wrote that analyzed those decisions, and he got results from that. But what we don’t have data on are situations where the GAO doesn’t issue a written decision. And a lot of dismissals aren’t published in terms of the reason for them. And we’ve got a big gap in the record there.
Tom Temin: Was he able to FOIA that gap and get hold of those documents?
Joseph Petrillo: That information is available from GAO, those are publicly available on the GAO website. And being a student he’d probably have access to computer legal research resources, like Lexus that would provide that information.
Tom Temin: So basically, he accumulated what is out there, but has never been compiled in exactly that lens, so to speak.
Joseph Petrillo: Exactly. But even with that, there are lots of data elements that aren’t available. A couple of years ago, Congress required DoD to conduct a study of bid protests of DoD procurements. And it engaged RAND Corporation, a highly respected research organization, to answer those. And RAND could only find enough data to answer five of the 14 questions posed by Congress could not even get the data to answer nine of the questions. So that I think shows that the lack of this data is hindering policy decisions. It’s hindering decisions by agencies about how to reorganize their evaluation process, how to train their personnel. It’s hindering decisions by protesters about what protests to bring.
Tom Temin: Is the methodology and the code that was developed by the student, Mr. Dawson, is that available to publish that?
Joseph Petrillo: He did, in fact, publish his entire program, as an appendix to his article. And I, I praise the editors for including it, because for those who can understand such things, it’s probably useful information.
Tom Temin: Yes, somebody could probably clip it and then charge the government a million dollars to do a study against it.
Joseph Petrillo: Well, I hope not. But, you know, Dawson proposes a public disclosure of a 15 item data set to provide better data for policy formation and other decision making. And if you look at the information he’s proposing, the sources that are available that he discusses, all but four or five of those can be drawn from other data sources and accumulated from GAO and agency data sources and things like that. And then there would have to be some additional data entry and analysis for four or five of those items. But it doesn’t seem like by any means an overwhelming task. So perhaps this will spur the policymakers out there to or maybe Congress to move ahead and get this data accumulated.
Tom Temin: Yeah, it sounds like Mr. Dawson is right in line with the foundations for evidence-based policymaking, which is the law of the land and also the policy of the government.
Joseph Petrillo: Well, when you know what the situation is, and you know what you’re trying to achieve, you’re probably going to get a better result.
Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.
Tom Temin
Tom Temin is host of the Federal Drive and has been providing insight on federal technology and management issues for more than 30 years.
Follow @tteminWFED
