The Department of Health and Human Services readied for Hurricane Sandy with a mix of high-tech and low-tech preparations.
All of the approaches were part of the lessons learned from this summer’s Derecho that knocked out power for days in the Washington, D.C. area.
“As we saw the hurricane heading our way late last week, we began reviewing all of our concept of operations, standard operating procedures and made sure everyone was well prepared and knew what to do in the case of an emergency,” said Frank Baitman, the HHS chief information officer. “We used some old technology too. We began printing out paper contact lists in case people lost access to their electronic contact lists.”
Along with the paper lists, Baitman said HHS increased the size of its help desk staff over the weekend to handle questions from employees about how to log onto their laptops and the network. But maybe, most importantly, Baitman said HHS ensured a key data center wouldn’t fail like it did in July.
“We learned that some things didn’t work quite well. One of our core data centers went down and although it rolled over to be on battery and diesel, it didn’t come back up immediately and we lost email,” he said. “Thankfully, that occurred during the middle of the night and very few employees were aware that it happened. We have been hard at work working with our facilities folks to retest all of our backup systems. We have tested that data center under load and it works just fine going over to backup now.”
Prepare to telework
Secretary Kathleen Sebelius also sent a note to employees late last week reminding them to prepare for the storm. Among the recommendations Sebelius made was for employees to take their laptops home in case they needed to telework on Monday.
“I would be very surprised based on the calls I’ve been a part of this morning if remote access weren’t dramatically up today,” Baitman said Monday. “There are so many employees who are taking advantage of the technology to stay connected to the network even though they are working from home today.”
An online Federal News Radio poll found a wide split among federal employees who answered whether they were working from home or not during the storm.
32 percent of respondents said they were working from home, 13 percent said they were mainly checking email and 40 percent they were not working and were “hunkered down.”
“We are getting hourly reports. We are having all of our operating divisions report into our computer security incident response center, which is in Atlanta,” he said. “Right now, in all operating divisions, everything is operational and no damage at this point.”
Should a data center or network go down, Baitman said he would first see how critical the applications and data are in the center and then decide how best to proceed. HHS is running more than 200 data centers across the country.
“For instance, our data center for the Office of the Secretary gets tested monthly,” he said. “We discovered with the July storm we weren’t testing it properly and we changed our testing protocols. Others that are more critical get tested more frequently.”
Baitman said HHS’ headquarters in Washington has some minor flooding.
“I think we are all expecting various power failures as a result of this storm. It’s kind of inevitable given the size and power of the storm,” he said. “Our hope is that all of our preparations and our testing will result in our data centers, in particular, staying up, allowing people to continue to be connected to those networks remotely.”