Mitre puts cybersecurity vendors to the test

Best listening experience is on Chrome, Firefox or Safari. Subscribe to Federal Drive’s daily audio interviews on Apple Podcasts or PodcastOne.

Just as there’s no shortage of cybersecurity threats there’s no shortage of companies and products to try and help you counteract them. The non-profit Mitre Corporation tested products from 21 vendors using an attack emulation. For what researchers discovered, Mitre’s principal cyber operations engineer, Frank Duff, joined Federal Drive with Tom Temin. 

Interview transcript:

Tom Temin: Mr. Duff, good to have you on.

Frank Duff: I really appreciate you having me.

Tom Temin: Now you were testing for a specific threat. So tell us what it was and what the methodology was to test these products. Let’s start there by setting the scene.

Frank Duff: Absolutely. So every round, we choose a different adversary to emulate and this is our second round. First one, we did APT3, which is a group that’s attributed to China. This round, we actually did something with APT29. So this is the group that is attributed with the DNC compromise a couple years ago, and so it’s a Russian actor that offers a very sophisticated toolkit and so offered a very unique chance to emulate.

Tom Temin: So this was the APT or the advanced personal system threat that allowed hacking of the Democratic National Committee’s emails, correct?

Frank Duff: Correct. Yeah. And at least that’s what’s been attributed to. And so they’re heavily reported on, there’s a lot of intel available on what their behaviors look like. So therefore, when we created the emulation, it’s something that we can do with a lot of confidence that we’re emulating it correctly, right. We’re adhering to the behaviors that have been exhibited by that actor.

Tom Temin: And 21 companies participated, and they paid to have the products tested, correct?

Frank Duff: Correct. So vendors do pay to participate. They’re the ones that are at the end of the day footing the bill and that allows mitre is the nonprofit to be able to execute this. We open it up, we do an open call for participation and any vendor can sign up under that. And 21 stepped into the fold this year to participate in it.

        Read more: Cybersecurity

Tom Temin: And you’ve got all of the big names Blackberry, Cylance, Broadcom, CrowdStrike, McAfee, VMware, all the well known companies. And what products do they submit to be tested against this particular APT?

Frank Duff: So from like a marketing endpoint most of these products would fall into what a Gartner or Forrester would consider endpoint protection platforms, EPPS, or endpoint detect and respond, EDR technologies. What these technologies are, are the technologies that would focus on detecting the threat once they’re in. So let’s say your traditional anti viruses have failed. How do you find out that the adversaries in your environment what they did, where they went, etc.

Tom Temin: So when you say endpoint, this would be something that could come in from a mobile device or on a laptop computer.

Frank Duff: So the focus right now is on Windows Enterprise for this round. So whether that’s something like your laptop or the actual servers, file servers, or the domain controllers, things that are attributed to the enterprise itself, but it’s Microsoft Windows focused.

Tom Temin: And is the vector, a phishing email or do they get in some other way?

Frank Duff: Are they getting in is something that we hand wave. We claim that the adversary came in through a spear phish but our evaluations start with the file on the desktop. That the users double clicking, how the access was achieved is kind of beyond the scope of the evaluation. And our focus is once that double click happens, so whether it was a witting user, or whether it came in from a spear fish or some other platform, the witting user in this case would have double clicked the file, and that would have launched the evaluation at that point.

Tom Temin: And this is not something that they plant and then nine months later, it activates. This is something once you click on it, you release some routine?

        Sign up for our daily newsletter so you never miss a beat on all things federal

Frank Duff: Yeah, I mean, so in reality, it could happen a number of ways. But for the sake of this evaluation, we choose to start the evaluation immediately at that double click, and that’s the first activity that we’re representing. One of the important aspects to recognize for this evaluation is it’s also open book. So we tell them when we double click that file, we tell them what that file was. They have to show us the detections that are associated with that. So what were they able to alert on with that double click or subsequent activities.

Tom Temin: So these software products then operate as agents on each person’s machine?

Frank Duff: Correct. So the sensors that are deployed are on each of the endpoints that are part of the test, whether again, that’s a server or whether that’s the representative of the laptop or personal computer. So the solutions deploy on that they’re gathering data related to what processes are running, what files are being touched, what changes to the underlying operating system are happening. And then all that data is fed back into a central repository where an analyst could go and query it to be able to analyze it to understand whether the behavior is normal or abnormal is something that they should be concerned with an investigate or just something that’s happening on the system.

Tom Temin: And in running these tests, what did you discover about the products?

Frank Duff: So it’s really interesting to watch the evolution of these products. Since this is now our second round, we’ve been able to see how some of these products are advancing. If you look back at the first For instance, one of the things that we noticed was PowerShell logging. PowerShell is a administrative tool that is on Windows operating systems by default that allows users to be able to do certain types of scripting to make their lives a little bit easier. It’s something that admins here is also used heavily. In the first round. With APT3, we actually did do PowerShell scripting, and largely vendors missed it, there was there’s on average, two out of 12 vendors that participated, were able to perform a detection. The second round, we focused heavily on PowerShell scripting, a lot of the activity that we were doing was centered around it, because that’s what APT29 does. And it was very good to see that these products were for the majority had visibility in district lock logging, understanding what was in the contents of the script that PowerShell was executing, so that you could extract the behaviors from that and leverage it. So PowerShell visibility is definitely one of those things that’s improving which sense It’s so prevalent with so many different actors that are out there, it’s a really positive thing to see across the industry from other standpoints, right, data sources in general are continuing to expand as products learn how to deal with the big data problem. They’re trying to figure out how to leverage some of this information that is just voluminous in in the network, how they can actually pull in the relevant details so that you can still make sense of it. So seeing things like the integrity level of a processor, you know, whether it was just a normal user, or whether that user that was executing that process had privileges to the entire system, for instance, as well as other data sources. So sure, to see expansion in terms of the capabilities of these products.

Tom Temin: Do the products stop the process or do they simply alert the administrator or the security operations center that something is going on that you better look at?

Frank Duff: So this evaluation is focused on the detection, we in fact, we tell them that they have to either turn off their protections or they have to set the detection to alert only mode. So that way we can ensure that when we do our repeatable process, our methodology that it can be the exact same methodology across all vendors, if we hit a block, we don’t have to worry about it because that block doesn’t exist, we can just go from end to end, say how all these products were able to deal with the detection problem, this coming round that we’ve opened up, we’ll start looking into the protection aspects. So the process was killed because the file that was being executed was deemed malicious for whatever reason, for instance.

Tom Temin: Alright. And so of the 21. Companies, would you say that if a agency chose any one of them, they would have a good sense of when the wrong process is happening, and could do something about it? And did any really stand out as hey, you really should look at this one. And I understand you can’t endorse any of them, give us some sense of relativity.

Frank Duff: Yeah. So from my perspective, as somebody that’s done research in this domain long before we were doing evaluations in this domain, and then an understanding what the mitre attack knowledge basics and how to leverage that a lot of that requires you to have visibility into input behavior. And so from that standpoint, these types of products are, from my standpoint necessary to understand what the adversary is doing to minimize their time with once they get in, how long they’re on your network, the amount of damage that they’re doing all these products, I think what you can say about them is they’re going through this process, our evaluation is threatened formed. And so they’re trying to improve themselves based on the real threat. And I think that that can mean a lot. There’s definitely very capabilities in terms of both what data they’re capturing how they’re able to leverage that data to create detections, whether those are alerts or just looking at more of hey you should look at this. But at the end of the day, this type of product from my standpoint is definitely one of those that should be necessary in any environment.

Tom Temin: And the vendors then can use this information to go back and improve their products, it sounds like.

Frank Duff: Absolutely, and many of them do. Some are more upfront and relaying information feedback back to us in terms of the improvements that they’ve made. What we’ve identified in between the the couple of rounds that we’ve done, we’ve identified flaws in their detection logic that could have had serious ramifications to potential deployments. We’ve been able to inform tech roadmaps, verify certain findings, inform other ones figure out ways that they can improve the usability for their end users. So my hope is, is that the vendors really come away from this feeling, not just that they have something that they can talk to in public domain in terms of coverage going through yet another evaluation, but that they can actually get technical value out of this in terms of better capability.

Tom Temin: And do they see one another results, that is to say, Can FireEye see how Blackberry Cylance did and can Broadcom see how Malwarebytes did?

Frank Duff: Absolutely. So they don’t see it up through the evaluation itself. However, once we’ve released the results, everybody can see everybody’s so all the results are publicly released. We don’t hold things back. The methodologies released so you can see all 21 vendors, you can compare them side by sides if you want to. And and we know that right that that empowers both the end user community so that when you’re going off and buying a tool, you know, the true capabilities that exist for these solutions, but it also helps kind of propel the research forward. So if I was a vendor, I could understand how other vendors are approaching this problem. And you can kind of lift all boats up with that evaluation so that they’re all improving by going through this process.

Tom Temin: Frank Duff is principal cyber operations engineer at the MITRE Corporation. Thanks so much for joining me.

Frank Duff: Absolutely. Again, thanks for having me and I appreciate the chance to talk about this.

Check out the results here.

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.