The Defense Department has nearly 90 pages of guidance that defines what DevSecOps for the DoD. Red Hat recently took this guidance and developed REDSORD, a reference implementation of their trusted software supply chain specifically for DoD customers. REDSORD came out of an internal question, “Instead of a build-your-own flashlight kit, what would a ‘flashlight with batteries included’ services approach look like for DoD customers to help achieve the DoD DevSecOps guidance?
“When many think about DevSecOps problems, they think in terms of toolsets. We are changing that paradigm and thinking about workflow first “ said Bill Bensing, managing architect of Red HatⓇ Software Factory. “Why? It’s a forest-through-the-trees problem. A workflow first approach identifies the organizational behaviors and expected outcomes to define what quality is. The workflow codifies and enforces these to ensure all aspects of security, compliance, trust, and privacy are addressed. The tools, well the tools are selected to implement the workflow. Our approach ensures tools are not the limiting factor for the organization’s journey into a DevSecOps culture. Some great side effects of a workflow-first approach is making once hard things, such as real-time auditing capability for an authority to operate, easier. “This perspective pushed us to create REDSORD, it’s a highly transparent and opinionated DevSecOps approach that focuses on these additional capabilities such as collecting, validating, and warehousing auditable data throughout the build, test, and deploy phases.”
A change to DevSecOps begins with introspection on the DoD’s part. The organization has to understand what it does and doesn’t understand about DevSecOps. The organization needs a vision. And it needs to know the baseline from which it’s starting. All that is necessary before it can know where to go next.
Bensing said that despite the size and variance within the DoD and all its disparate components, there are some common themes. Getting rid of that “wall of confusion,” as it’s referred to in DevOps, tends to be the start. That’s a question of how to take traditionally siloed organizations, and integrate them appropriately. But that is a cultural change more than anything else, and organizations often look for technological solutions before looking to their own culture.
“People tend to focus on the tools. And that’s where they get bogged down. It’s common that an organization becomes engrossed in these tool-based-religious-menial spats that makes the big picture of expected DevSecOps behaviors and outcomes very ambiguous,” Bensing said. “It doesn’t matter what the tool is. The workflow is the most important; it’s the rules, policies, and approach which dictates how one gets from idea to production. That’s why we created our trusted software supply chain (TSSC). REDSORD is the TSSC for the DoD which helps them enable, accelerate, and enforce the processes and expected behaviors for software development and deploy to enable the DoD DevSecOps culture. Tools are simply an implementation detail.”
And the goal of automation is to remove the subjectivity and variability that leads to quality variances caused by human interaction. But that assumes the organization is starting with a common practice in the first place. And standardization can be a problem for many software organizations.
“It’s looking at your workflow and assessing those critical behaviours for quality, functionality, and compliance validations you need inorder to take your software from idea to production. Automated builds and testing are table stakes for DevSecOps. It’s about automated compliance and vulnerability scanning during build time and runtime. It’s about automated validating deployment manifests to ensure software is being deployed as expected. It’s about establishing provenance and verified historical data to ensure complete transparency. Bensing stated. “DevSecOps requires us to accept a new set of rules to operate by. The old ruleset was ‘have your domain expert complete a manual review’, while the new ruleset is ‘have your domain expert codify their validations so automation can perform the review in real-time.’ Automations biggest asset is removing variability from a process by re-applying the same approach, the same way, every time. It results in time savings, because now, what was once a manual process that took a long time due primarily to que time, just waiting for someone to perform a review, is now near real-time. This is how you ensure the highest quality software gets to market in the least amount of time.
The biggest opportunities for automation are when humans have to do manual reviews. For example, authorities to operate are very thorough, very in depth processes. But if they could be strategized better, a process could be built to automate the real-time collection and evaluation of critical data for complete transparency.
“The trusted software supply chain cannot build quality into components. It will not take something that’s low quality and make it high quality. You can only engineer quality into your product. The trusted software supply chain enforces objective quality and security standards to ensure any defects are found as far left as possible in our environment of ever increasing and complex quality or security requirements,” Bensing said. “That’s why REDSORD helps the DoD, because it drives desired behaviors by enforcing a high-trust approach. The people who once were responsible for manual review, now define and codify how the review should be done. This achieves two specific outcomes. First, it helps eliminate issues which come from those one-off-judgement-based mistakes. Second, those once reviewers are now freed to focus on higher quality automation efforts.”
Standardization and automation of workflows may be a significant culture change, but it can actually help DoD retain some of its more traditional aspects, like separation of duties and responsibilities, without retaining the organizational silos that make it more difficult to get from idea to market as quickly.
“The concept around the trusted software supply chain came from a lot of our Red Hat consultants and architects seeing a lot of the same themes over and over,” Bensing said. “As the world goes cloud native, the industry standard are situations where one can push a button and have IT systems spin up at a glance. We want to facilitate this industry trend so we can focus on the questions of ‘How can we help the government focus more on building their mission critical applications,’ as opposed to just installing a bunch of stuff that could be accessed on-demand.”