What happens when you bring together some of the nation’s leading hackers, the Pentagon’s chief of training and an Air Force Academy professor who teaches cyber skills to cadets? They all agree on one thing: The government’s approach to cyber security is coming up short.

They sat on the dais, an unusual assortment of experts at a conference for military simulation and training experts. No prepared speeches, just a wide open Q&A.

Their message in three bullets:

• You can’t teach cyber defense without a thorough understanding and expertise in cyber offense
• Cyber is all about breaking the rules. If you try to break cyber defense into a series of check-box requirements, you will fail
• The Fifth Domain, as cyber is sometimes called in the military (joining air, land, sea and space) is not like the others. There is no high ground and the weapon you wield today may not even exist tomorrow

In the center was Frank DiGiovanni, director of Force Readiness and Training at the Pentagon, joined on his far left by Martin Carlisle, professor of computer science at the Air Force Academy. Sharing that stage were three of the best-known ethical hackers in the business: Jeff Moss, founder of Black Hat and DefCon, the two best-known annual hacker conferences; John Rigney, co-founder of Point3 Security, a Maryland cyber firm, who says he made his first hack at age 8; and Brian Markus, CEO of Aries Security, best known for his “Wall of Sheep” – an annual rite at the DefCon event, where he posts the names of all who have exposed themselves to security cyber hacks while attending the conference, which brings together some of the world’s top hacking talent.

What these five know about cyber security – or how to defeat it – can’t be cataloged. Indeed, part of their message is that cyber security, or cyber warfare, is so fluid, so rapidly evolving, that trying to define it or contain it is essentially impossible.

The government and industry are both in a quandary over the cyber challenge, partly because it’s unclear where their missions start and stop. America is fighting its cyber battles like the British fought in the American Revolution, he said. Back then, the British fought out in the open, following a well-drilled formula for combat. The Americans countered with guerilla warfare, fighting from the woods.

By limiting most of our defenders to defense-only approaches, the United States is effectively fighting while hand-cuffed. Cyber attackers, on the other hand, whether criminals or nation states, are playing without rules.

Said Markus: “We’re going up against a 300-pound fighter with one hand behind our back. We are going in with too many limitations.”

That’s the first thing cyber training needs to take into account: Cyber warriors have to be able to think like their attackers, and to do that they need to train like their attackers. Instead of focusing on rules and process, they need to focus on puzzle- and problem-solving.

Certifications are useful in understanding what people know, Carlisle said, but they are of limited use in fighting the active cyber attacker. Hackers buy cyber defense technology and then work on their own to defeat it. So one can’t be satisfied that having the best tools will be enough to protect your network.

The key to developing cyber talent isn’t to teach people to do well on certification tests, Carlisle and the others said, but rather to teach them to think and problem solve.

Said DiGiovanni: “If you think you can catalog every known thing that can happen to you, you’re wrong from the beginning…. To do this right, the training environ needs to be able to go beyond the square where you know exactly what you’re doing. The minute you do that, it’s exploitable. Someone will find a weakness in that training regimen and attack it.”

Similarly, Rigney questioned military efforts to standardize network design. “One attack profile means one target,” he said.

Cyber security is complex and fluid. Everything is changing, all the time, the panelists said. The military has the capability and the mission to develop the right tools. But to be successful, its policies and approach will have to change – and not just in how people are trained. One problem several panelists mentioned is that even when the military gets the training right, it sometimes mishandles the talent it produces.

This problem is not the military’s alone. Industry also makes mistakes. Markus described training programs to teach cyber skills that were highly successful, only to fail when it came to retaining talent. “When you train a bunch of people, and they get really excited, and amped up, and they have all this great knowledge of effects and warfare, and then you say, ‘Go watch the SOC [cyber operations center] logs,’ they say, ‘Fine, I quit. I want to go do something else.’ That’s why the industry is bleeding out people. They train them to be offensive warfare personnel, and then they have them go watch a gate.”

The military, everyone agreed, needs to be careful not to follow that model.

Carlisle, who emphasized he was speaking on his own behalf, and not for the Air Force Academy or the Air Force itself, said he rejects two common notions in military circles: First, that it’s ok to train for defense only out of concern about the risks involved with teaching people how to hack. “The military has certain fields, SEALs for example, who we accept that we train them to act with a certain degree of lethality. We should treat cyber the same way,” he said. Second, there are leaders who are satisfied to train cyber personnel only to lose them to industry after five or six years. Those leaders say what they really need are managers, not technical experts. “I reject that hypothesis,” he said.

The heart of cyber warfare, the panel agreed, is offensive operations. These are essential military skills they said, which need to be developed and nurtured – in order to ensure a sound cyber defense.