Artificial Intelligence

What happens if a hacker manages to mess up an artificial intelligence algorithm

DARPA has launched a program to develop defenses against that possibility. It's called Guaranteeing AI Robustness against Deception, or GARD.

Tom Temin@tteminWFED

January 11, 2022 11:51 am

10 min read

Best listening experience is on Chrome, Firefox or Safari. Subscribe to Federal Drive’s daily audio interviews on Apple Podcasts or PodcastOne.

Artificial intelligence might be the greatest thing since the half slip. But its software. And that means it can be hacked. That’s a concern for the Defense Advanced Research Projects Agency. DARPA has launched a program to develop defenses against that possibility. It’s called Guaranteeing AI Robustness against Deception, or GARD. For more, the GARD program manager, Dr. Bruce Draper, joined the Federal Drive with Tom Temin.

Interview transcript:

Tom Temin: Dr. Draper Good to have you on.

        Join us May 6 at 1 p.m. EST for Federal News Network's Industry Exchange Data where we'll explore how you can employ data to modernize your agency. | Register today!

Bruce Draper: Good to be here.

Tom Temin: Let’s begin with what are the threats, what can be done to a artificial or machine learning program that you don’t want to have happen to it?

Bruce Draper: Well, you know, it’s really funny because machine learning programs have come a long way in the last 10 years. And they’ve become very powerful; we can train them to do all kinds of things. But it turns out for all that power, they can often be fooled by very little changes that would affect you or I. So for example, if you think about a computer vision program that recognizes objects in an image, it can recognize it under all kinds of conditions and scales and lighting conditions. But it turns out, sometimes if you just tweak a few pixels, something you or I wouldn’t notice, it’ll cause the AI system to produce the wrong output. And we call those adversarial attacks, and then what we’re worried about.

Tom Temin: In other words, it’s not an attack on the algorithm code itself, but what it is you’re feeding the algorithm. And if someone knows what it’s trying to do, then you can figure out how to fool it, in other words.

Bruce Draper: That’s right. It’s sort of like camouflage for machines. And what’s interesting is it’s things that you are I would never think of or notice.

Tom Temin: Yes, because that brings to mind the example of, say, the NGA — the National Geospatial Intelligence Agency — is looking to use machine vision from space and looking at images. And they’re looking for specific things that are maybe unnatural, like squares and circles. So then that would say that in that particular application, someone could make something into a circle that wasn’t and fool the machine vision program.

Bruce Draper: That’s right, you could either make it think that there were objects, you know, in the world that aren’t there. Or conversely, you could hide the objects that are there. And if you’re in the intelligence community, both of those are obviously concerning.

Tom Temin: And it’s really, then, not a classic cybersecurity issue then, is it? Because what is being manipulated is external to the program.

        Read more: Artificial Intelligence

Bruce Draper: That’s right. And so if you think about cybersecurity, in cyber someone’s attacking your your operating system over a network or an application, which means any system can be attached that’s on the network. In this case, it’s more limited. Only machine learning programs get attacked in this way. But it’s more general, because that program doesn’t have to be on the web at all, because it can be attacked purely through its inputs and output.

Tom Temin: And do we have any evidence this is actually happened?

Bruce Draper: MITRE Corporation has a website where they track known cases of adversarial AI attacks. There aren’t very many, not nearly as widely spread as cybersecurity cyberattacks are at this moment. And most of them go in the category of mischief, rather than anything life-threatening, but the fact that they’re starting to occur and that they do happen in these mischievous settings is obviously concerning. And it means going forward, as we use more AI, both in the military and civilian context, it’s important that we be able to trust these systems, so you’ll be able to defend them.

Tom Temin: And even though it’s not classic cybersecurity, in some ways it’s also an insider threat, because someone inside could feed data that is extraneous, and therefore cause an issue.

Bruce Draper: That’s right, we’re worried about things of insider threat where people can get to the data stream and change the images or the input data to the system. We also worry about what we call physical attacks, where people can just get to, for example, the scene — put a sticker on a stop sign, so it’s not recognized anymore. And lastly, poisoning attacks: They can get to the training data when you’re developing the system. They can then make it so that it’s fooled down the road.

Tom Temin: We’re speaking with Dr. Bruce Draper. He’s a computer science professor and program manager at DARPA. And tell us about the garden program, what it is you’re trying to accomplish. And there’s a lot of parties involved in this whole thing.

Bruce Draper: There are a lot of parties involved, we’ve got 13 different performers working on aspects of it. There are three things we’re really trying to do. One is obviously develop better algorithms that are harder to attack, trying to come up with systems to defend machine learning systems so they cannot be attacked, so the known attacks don’t fool them. Second of all, we’re trying to develop the theory behind this, because we’re always worried about the unknown attack — the attack we haven’t thought of yet. So we want to have a theory that defends against all possible effects. And then third, and one of the things that we’ve talked about most lately, right now there’s no sort of system out there, either in the commercial world or in the military world, for when you develop an AI system or machine learning system, making sure that it’s well defended. So we’ve been creating tools and techniques for evaluating how vulnerable an ML system is. And we’re trying to push those tools out into the broader community, in the hope that it will become a standard so that the AI systems that we all use in the future are things we can trust.

        Sign up for our daily newsletter so you never miss a beat on all things federal

Tom Temin: So let me make up an example. You mentioned a stop sign that is recognized as a stop sign. They’re pretty distinctive. They’re red, they’ve got a white band around the edge and they’re eight sided. But when someone puts a sticker on there, then maybe it doesn’t look like a stop sign to the Machine Vision Program, even though it does, as you mention, to the human eye. So could the algorithms be designed more robustly so that well, eight sides — red mostly — and a white ribbon around the edge, even with a sticker in the middle, it’s still a stop sign.

Bruce Draper: Right. So what we’re trying to do with the with the defenses is come up with systems that will not be fooled by any minor change. Obviously, if you chop the stop, sign down and take it away, you’re not going to see it. But the idea is to make sure that no minor change — no sticker applied to it, no spatter of paint, no small clever change — could defeat it. And that’s what we mean by coming up with better defenses. And then we’re also trying to make sure that if someone says, hey, I’ve got this great AI system, I want to put it on a self-driving car, that you can evaluate it and know whether or not it can be fooled in this way, because obviously if your self-driving car doesn’t recognize the stop sign, that’s a real problem.

Tom Temin: Got it. And so what will the output of the GARD program be. Guidance? Or will it be a software products? Or what do you envision being made available?

Bruce Draper: So what we’re making available first are these tools for evaluating how vulnerable this system is. And we’re making those publicly available, open source, trying to get this out in the community, along with tutorials of how to use them, and how best to defend an AI system and data sets and things like that. And then we’re using those tools so that as we develop new algorithms, as we develop new defenses, we put those into those tools and therefore disseminate them, not only to the military community, but to the commercial community at large.

Tom Temin: And was this type of work going on in the AI community, this kind of collaboration in the open source? Or were they just too focused in general on the wonderful things you can do with AI and ML?

Bruce Draper: Well, certainly up until recently, you know, the challenge for those of us in the AI research field was simply to make it work. And so we were not worried about things like adversarial. Rcently, it has become more of a concern. And before the DARPA GARD program started, there were some academic projects, but they were mostly focused on the scenario, whereof the sort of insider threat scenario where someone can play with the exact inputs. What we’ve done is expanded that work, made sure it’s all out in the public, but also expanded it to these scenarios, like the stickers on stop signs and the poisoning threats. So that all the sort of real practical threat vectors are covered.

Tom Temin: Yes, we’ve been discussing mostly the idea of machine vision and visual pixels that can be altered. But are there other domains besides visual that this is a threat for?

Bruce Draper: This is pretty much a threat for most forms of AI. We’ve been actively working on all the sensors, so audio, visual, all these sorts of things. You could also imagine it for systems that do things like analyze, oh, I don’t know, financial records, right? I mean, there’s there’s still an AI system with still lots of input, except there, because the input is from a database, someone has to get access to the database. But if they can do that, it’s the same sort of thing.

Tom Temin: I mean, for many decades, programmers have been saying garbage in, garbage out. In some ways, this is just the latest iteration of a very old problem in software systems.

Bruce Draper: Right. But it’s this malicious, you’re not throwing lots of garbage at it, you’re throwing one grain of sand in just the right spot to mess up the machinery.

Tom Temin: So it’s a pretty subtle issue in many applications.

Bruce Draper: It is, in fact, when the attackers do well, you don’t know you’re being attacked.

Tom Temin: And is the GARD Program itself, does it have an end? It sounds like something that would be a never-ending need to keep working on.

Bruce Draper: So the GARD program itself started about two years ago and has about two years remaining. What we’re trying to do — but you’re right, this threat is not going to go away in two years. This is why we’re trying to do two things. One is build up this open source user community, so that even when GARD is gone, the work, the research, new defenses continue to be developed. And then the second is we’re trying to work within the military community to make sure that when the military either trains their own systems, or more often acquires systems from other sources, that these systems are properly vetted and tested before they’re put into service.

Tom Temin: Dr. Bruce Draper is a computer science professor and program manager at DARPA. Thanks so much for joining me.

Bruce Draper: Absolutely. It was a pleasure, Tom.