VA cyber saga continues as audit shows continued holes in network security

Jason Miller, executive editor, Federal News Radio

wfedstaff |

As the Veterans Affairs Department tries to reassure and explain to House lawmakers the steps it’s taking to protect veterans’ data and agency networks, a new audit shows long-standing holes in the network continue to put veterans’ data at risk.

Documents obtained by Federal News Radio show VA failed for the 15th year in a row its consolidated financial statement audit with regard to security controls.

View all of the documents here.

Documents show VA...


As the Veterans Affairs Department tries to reassure and explain to House lawmakers the steps it’s taking to protect veterans’ data and agency networks, a new audit shows long-standing holes in the network continue to put veterans’ data at risk.

Documents obtained by Federal News Radio show VA failed for the 15th year in a row its consolidated financial statement audit with regard to security controls.

View all of the documents here.

Documents show VA ended up with material weaknesses, including the failure to remove terminated employees from accessing the network, and the lack of a formal process for monitoring, preventing installation and removing unauthorized application software on agency systems.

And these latest findings by VA’s independent auditor CliftonLarsonAllen further support the House Veterans Affairs Committee’s constant drum beat for VA to do a better job at securing its systems and data.

The Office of Management and Budget through Circular A-130 and under the Federal Manager’s Financial Integrity Act require auditors to look at the network security attributes that the financial system runs on to ensure financial data is protected.

A VA spokesperson said VA’s commitment to securing veterans’ data hasn’t wavered.

“We take seriously our obligation to properly safeguard any personal information within our possession. VA has in place a strong, multi-layered defense to combat evolving cybersecurity threats,” the spokesperson said in email responses to questions from Federal News Radio. “In the event a veteran’s personal information is compromised, we take the matter very seriously and take immediate action, such as notification to every veteran who might have been affected, and offering free credit monitoring services. VA goes well beyond what the law requires in providing veterans with credit monitoring protection when their personal information has been potentially exposed.”

Documents show, however, VA struggled more in 2013 than in 2012 with removing terminated employees’ access to the financial management database.

Domain controllers disagreement continues

VA struggled with password standards, including having nine sites that have “network service accounts with passwords that have not been changed in over three years,” and one site where the “domain controller settings were not in compliance with the Windows 2003 Server Security Plan.”

The ability of VA to secure its domain controller settings became a major area of disagreement between the agency and the committee during a June 2013 hearing.

Committee members say as a result of VA’s poor control of its domain controllers, bad actors have a wide open door to steal data and access all parts of the network.

VA’s acting Chief Information Officer Stephen Warren disagreed with that assessment. At the hearing, he said a bad actor with access to the domain controllers can go anywhere they like in the network, but they do not own and control the network, meaning the hacker doesn’t control what anyone does or can do or where the traffic goes.

But security experts and lawmakers disagreed.

“We know that the way these individuals work that it is a typical tactic for them to, if they compromise something such as a domain controller as was said earlier, or particularly the domain controllers, the domain controller has a file on it called the SAM file and that file is the securities accounts manager,” said Jerry Davis, the former VA chief information security officer, at the hearing. “In that file are all the password accounts for the users in the network. So, if they have got the domain controller, they will grab the SAM file. When they encrypt the information, generally if it is going out and it is encrypted, I know they hit a domain controller. I guarantee they probably took the SAM file. They are going to go back, crack it later and are going to take every password that was on that system.”

Now that the financial management system audit shows that at least at one site where the domain controller is potentially exploitable, some government sources are questioning how many others are in the same state of disrepair.

“If you have one domain controller compromised, all domains talk to each other so they are all vulnerable,” said one government source, who requested anonymity, because they weren’t authorized to speak to the press. “It’s easy to move from one domain to the next if you control one of them.”

A former government official said the inability of VA to remove terminated employees’ access to the network is dangerous.

“If you have separated employees and the biggest threat is the insider threat and they still have active access to the networks, that’s really bad,” the former official said. “I’d be worried about that because there are a lot of disgruntled VA employees out there.”

A year in VA’s network?

Another government source, who also requested anonymity, said the primary goal of a bad actor is to maintain presence in the network. They build multiple back doors to come and go as they please by compromising the domain controllers. And VA’s network is huge, with more than a million devices, which makes closing those doors or finding the openings that much harder.

In a Nov. 22 response to questions from the committee, VA said it removed and cleaned all computers that could have been part of the incidents that compromised the domain controllers.

But the second government source said, “Reclaiming one device does not ensure that the entire environment is free and clear of the actors. With such a large network hosting hundreds of thousands of devices with superfluous pathways, actors have many places to hide and go undetected for months or years at a time.”

The source also said the average length of time a bad actor is in a network before being detected is a year, so based on congressional inquiry and VA’s testimony, these hackers already may have 10 years of combined access in their network through the nine separate incidents.

“A rule of thumb that security professionals follow is that you always assume that your network is compromised, that your communications are being read and attackers have access to unprotected resources,” said the source. “In fact, this is the same rule of thumb that the National Security Agency follows in managing their own networks.VA has had unprotected resources for years. Databases holding sensitive data on veterans, unlatched servers, unlatched desktops, poor code in Web applications combined with an attacker pretty much guarantees that the network is not under the control of VA.”

And if bad actors are in VA’s network, they can access sensitive data fairly easily, auditors found. The financial audit documents say the baseline configuration of Web applications show a critical or high vulnerabilities for “weblogon” systems, including “cross frame scripting, blank and easily guessed passwords and credit card and social security number disclosures.”

“The vulnerabilities were detected during internal scanning of one Web application. VA’s Web application firewall for the data centers blocked the scanning of the Web applications from the outside,” the documents noted.

The first government source said the problems with the baseline configurations and the domain controllers validate how bad the situation is at VA and why the House Veterans Affairs Committee continues to press them to fix the problems.

House committee holds another briefing

The latest attempt by the committee to get VA to act more quickly to fix what lawmakers say are serious security holes came Dec. 3.

A committee staff member said the briefing included representatives from VA, the agency’s inspector general office, the Government Accountability Office and Democrat and Republican lawmakers. The briefing included a panel of cyber experts who offered their opinions on VA’s IT security vulnerabilities.

“The House Committee on Veterans’ Affairs has been working for nearly a year to convince the department to resolve a number of serious IT security vulnerabilities identified by Oversight and Investigations Subcommittee staff and substantiated by GAO and VA OIG,” the staff member said. “The Dec. 3 briefing was a continuation of this effort. Following the briefing, representatives from VA declined to respond when given the chance to comment on the security vulnerabilities presented by HVAC staff and various IT experts. Committee members are hopeful that VA, having been presented with a detailed list of its specific network security vulnerabilities, will take this opportunity to work with HVAC to resolve these IT challenges.”

The staff member confirmed that VA sent assistant secretary for congressional and legislative affairs Joan Mooney and chief of staff Jose Riojas, but not anyone from the CIO’s office.

“VA appreciates the committee’s interest in this topic and always welcomes the opportunity to work with Congress to better serve veterans,” the VA spokesperson said.

A third government source with knowledge of VA said the fact that CIO’s office didn’t send anyone to the briefing is part of the “siege mentality” that has taken over the office as the House Veterans Affairs Committee has become more aggressive.

“I find it disingenuous in how they are responding to this and the degree of contempt they have in how they are approaching this. They feel it’s a witch hunt,” said the source. “There is a marked lack of respect for the committee by the IT leadership. How they are managing the process is indicative of the lack of respect for Congress and particularly the Veterans Affairs Committee. They think it’s a game so they will evade, obfuscate and they will basically come back with just the bare minimum so as not to be out of compliance.”

The source said the cybersecurity environment in VA hasn’t improved since the summer’s hearings, and more vital cybersecurity employees have left the agency.

In November, Don Sheehan, who was the acting associate deputy assistant secretary for cyber operations, resigned and now works at NetApp.

The third source said Sheehan was one of the talented people who kept the place together, but grew frustrated and looked for a better opportunity.

“If you look back at Continuous Readiness in Information Security Program (CRISP) initiative, it’s been two years now and it was effective because [former VA Deputy Secretary] Scott Gould got in there and put skin in game, and they went hospital by hospital and did site visits to look at and fix the material weaknesses,” the source said. “Now VA doesn’t do any of that and has a zero level of engagement. It’s all conference calls and blogs now.”

VA’s struggle with cyber isn’t just an issue the House Veterans Affairs is looking at. A spokesman for the sister committee in the Senate said, “The committee is continuing ongoing discussion with the VA on its implementation of the Continuous Readiness Information Security [Program] and its overall cybersecurity posture. No veteran should feel that they cannot trust VA to keep their personal information secure when accessing the health care and benefits they have earned.”

To date, VA has taken several steps to improve its network.

“VA has in place a strong, multi-layered defense to combat evolving cybersecurity threats. These defenses include monitoring outside our network by external partners, active scanning of Web applications and source code, and protection of servers, workstations, network and gateways, among other security efforts,” the spokeswoman said. “Nonetheless, VA is always looking to identify new ways to strengthen its systems, and in recent months has implemented Einstein 3A network monitoring capabilities with the Department of Homeland Security, upgraded nearly all virtual private network connections to the more secure Rescue or Citrix Access Gateway, completed deployment of Windows 7 with built-in encryption to over 95 percent of VA computers, and implemented two-factor authentication for system administrators.”

Warren also detailed steps VA is or has taken in the Nov. 22 letter to the committee.

He said the agency on Nov. 21 increased its information operations condition to an “elevated” level from a “guarded” level. This means there has been an increased number of incidents reported to VA by the Homeland Security Department’s U.S. Computer Emergency Readiness Team (U.S. CERT).

Warren told the committee the VA is taking several steps to improve its network and data security. These include:

  • VA’s network security operations center conducts vulnerability, Federal Desktop Core Configuration, content and payment card industry scans.
  • VA uses penetration testing on custom developed and commercial-off-the-shelf software it’s deploying. It’s comprised of manual penetration testing and automated tools.
  • The product development staff uses static code analysis tools to test code for security weaknesses. The security operations center also tests code as part of the independent assessment before the approval to put a system on a network or before renewing an existing systems authority to operate.
  • VA performs manual testing of Web application security, looking for among other things enumerate business logic flaws or errors.

Additionally, Warren said VA is encrypting desktop and laptops, and implementing Microsoft Windows 7. He said Windows 7 migration will be completed for the majority of VA systems by Jan. 31.

Interestingly, former VA CIO Roger Baker said in August 2012 that 99 percent of VA laptops were encrypted. So it’s unclear if Warren was referring to those laptops, which VA has been working on since the infamous 2006 data loss incident, or a new effort.

“VA OIT began several initiatives to increase visibility to VA assets across the enterprise to include upgrade of the Active Directory infrastructure and two- factor authentication implementation enterprisewide for all accounts,” Warren wrote in the letter.

Missing out on the basics

One VA source said the two-factor authentication implementation under Homeland Security Presidential Directive-12 is happening, albeit slowly.

“It’s a big job in a big department like this and we are struggling with it,” the VA official said. “We are encouraged to encourage the need to do this, but we’ve come to some technical problems. It’s not the card, but the software on the PC. We experienced some lengthy access calls and are trying to work them out.”

Warren added two-thirds of all wide-area network circuits now are encrypted to protect against data exposure to non-VA employees. Warren said the rest will be completed by the second quarter of 2014.

Despite all these steps and progress, VA continues to get dinged by auditors.

VA’s inspector general in June issued the fiscal 2012 Federal Information Security Management Act report and found significant deficiencies related to access controls, configuration management controls, continuous monitoring controls and service continuity practices designed to protect mission-critical systems.

The second government source said VA has lacked basic hygiene for a long time, which makes its environment easy to penetrate and steal data from.

“In 2009 or 2010, VA requested that NSA perform an assessment of the VA network security posture. NSA refused saying that VA didn’t even do basic hygiene and that it would be trivial to penetrate the network,” the source said. “This proved to be true as VA was consistently penetrated multiple time over the following few years or were already penetrated and just detected the actors later in the timeline. I can put all of this together and assume with reasonable assurances that veteran data has been hauled out of VA. Information of value is monetized by being sold on the underground black markets or bartered and traded for other goods and services between bad actors.”


VA’s security shortcuts put millions of veterans’ data at risk, former VA cyber official alleges

Lawmakers, IG expose further vulnerabilities in VA’s cybersecurity

Serious doubts remain about VA’s ability to secure veterans’ data