New DoD cyber test procedures emphasize resilience over compliance

To listen to On DoD on your phone or mobile device, subscribe on PodcastOne or Apple Podcasts. The best listening experience on desktop can be found using Chrome, Firefox or Safari. 

In the three decades since Congress first ordered DoD to create an independent director of Operational Test and Evaluation, software has always had something to do with that office’s mission: assessing the degree to which a particular weapon system will work in real-world combat.

The national security landscape has changed a fair amount since 1984. So has the process of developing software: it’s now as vital to the development of the F-35 as traditional performance measures like thrust and fuel consumption were back then.

But the ways in which DOT&E assesses the fitness of those systems hasn’t quite reflected those changes, according to the office’s newest director, Robert Behler.

“Now our weapons systems are really defined by software,” he said. “It is the building material of choice, and we have to be able to make sure that we do a proper assessment of that from a cybersecurity standpoint, and from the standpoint of the functionality of the software.”

New cybersecurity guidance issued

In April, Behler’s office published a new revision of one of the few documents DOT&E releases to the public, directing changes in how the department will assess the IT systems which increasingly represent a vital component of its military readiness.

Among other changes, for the first time, the new cybersecurity guidance tells DoD’s operational testers and, consequently, the program managers who must prepare for those tests to closely examine the provenance of the IT subsystems that go into a given weapons system.

“If you look at all the things that touch the cyber and software aspects of a weapon system, it gets pretty large pretty quickly,” Behler said in an interview for Federal News Radio’s On DoD. “They may not all be using the same standards of security that the main contractor may be. My feeling is it could very well be the soft underbelly of cybersecurity for a weapon system, so the guidance is to look at the supply chain and to make sure that you, the program manager, is comfortable with the cybersecurity of that supply chain.”

In another change, DOT&E’s new procedures tell the department’s operational testers to examine not just whether a given system is susceptible to cyber attack, since virtually all systems fall into that category. Testers should also assess how they would actually perform in a battlespace where an enemy succeeds in shutting some systems down, at least for a time.

“We can’t test everything,” Behler said. “What is really needed, to continue to operate, is to be resilient and be able to recover from a cyber attack.”

The previous version of the DOT&E guidance, replaced by the April update, included a detailed appendix, instructing DoD’s operational test agencies on specific cybersecurity measures they should inspect as part of their work to determine whether a given system is operationally suitable for its mission. It included a list of security controls, developed by the National Institute of Standards and Technology (NIST) as pointers to determine whether a system was in compliance with federal standards.

By contrast, the word “compliance” appears nowhere in the new guidance.

“We cannot take a checklist mentality to solve all of our cyber problems,” Behler said. “I don’t want any tester to walk away saying, ‘If I fill all those squares in those compliance checklists, the system is going to be capable of doing its mission in a cyber-intensive environment.’ We can listen to folks say that they’ve complied with all the checklists, we can listen to contractors saying that the systems are safe and cannot be intruded into through the supply chain, for example. Those are very good words, but in the end, it’s my responsibility to make sure that’s correct. We have to do the assessment.”

New testing paradigms envisioned

Over the longer term, Behler has broader changes in mind.

In a world of weapons systems increasingly defined by software, it will be almost impossible to find enough humans beings, irrespective of their expertise, to accurately test how those systems will perform in real-world situations.

So, he said he’s attempting to push the department toward new procedures that would break the traditional paradigm in which a system is tested for operational suitability or effectiveness at one particular point in time, and then moves onto the next stage in DoD’s acquisition process.

“My thrust is to be able to come up with automated testing, vulnerability discovery and vulnerability patching in an automated way,” Behler said.

The process will need to continue to evolve as DoD gradually alters its software procurement strategies toward agile software development.

“We want to get to a point where we’re building software at night and testing it during the day,” he said. “That’s the end state, and it’s the model they’re using in places like Silicon Valley. We want to get to the point where the difference between a developer and an operator or a tester is almost nonexistent.”

Behler said an integrated approach along those lines is vital to accelerating the acquisition process. But it does present new challenges to DoD’s traditional system, in which operational testers proffer their views on the suitability of a system before a program manager or more senior officials give it the green light to proceed to the next stage.

As one example of the existing process, DoD’s operational test community recently reported that the department’s forthcoming electronic health record, MHS Genesis, was neither operationally suitable nor effective, after the department had already implemented the EHR at three test sites in the Pacific Northwest.

Going forward, at least when it comes to software, DoD needs to combine traditional notions of developmental testing with operational testing, collapse the two, and conduct them on an ongoing basis, Behler said.

“I don’t think we’re ever going to see a single point where everything is done, where we hand it over to the OT&E community and we go out and test it in an operational environment,” he said, ”We can test in an operational environment as we’re going through the development stages. What we’re going to call it is ‘capability-based test and evaluation.’ We want to test the capabilities, and we want to make sure that we get those capabilities early, including cybersecurity.”

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.