The network consolidation and modernization work that started almost two years ago turned out to have truly saved the Transportation Department when the coronavirus pandemic hit.
If DOT had not consolidated eight separate networks and reduced more than 4,000 devices spread out across hundreds of locations, the surge in remote workers would’ve all but paralyzed the agency.
Ryan Cote, the chief information officer of the Transportation Department, said he shuddered to think what might have happened if not for the previous two years’ worth of work.
“In a matter of about 18 months we took that network from 8 into 1 and from over 4,000 devices to under 1,500, and more importantly, we standardized it all under one vendor so we knew what we had in terms of hardware and got everything patched and up to a level of readiness and security we found acceptable,” Cote said on Ask the CIO. “We reduced the support of that tremendously. Just eliminating hundreds of circuits that we didn’t need, consolidating into a managed trusted internet protocol services (MTIPS) solution and a multiprotocol label switching (MPLS) cloud that allowed us to have this visibility and manageability of the network that allowed us to be quick and agile when it came to working on the network.”
Cote said the network modernization and consolidation made it less difficult to know what it would take to get DOT to 100% telework. He said DOT knew the number of licenses it needed to increase or the bandwidth capacity that the agency required to support so many thousands of teleworkers because they knew what the current network capabilities were.
But it took more than just a modernized network. Cote personally got involved in nearly every aspect of the effort to get the network ready for the telework surge.
“I said to my folks, ‘let’s make a 100% telework readiness plan,’ and we had that in less than a day,” Cote said. “I took that back to senior leadership, the budget and acquisition folks, and said here is what it would take. In a matter of days, around the first week of March, people who looked at the plan read it and realized we needed to get there. They said here is the money, and I took that stamp of approval and went to the acquisition folks and said we have approval, go, go, go.”
Becoming personally involved
It was at that time that Cote became personally involved. He said he knew the normal acquisition process of 30-to-60 days wasn’t fast enough and started making calls to contracting officers and vendors.
“I communicated that this is not a normal routine acquisition and we needed to expedite this in a way never expedited before,” he said. “That is why in a matter of five days we went from 40% telework enabled to 100% telework enabled. People picked up the phone, circumvented the normal channels, called the contracting officer or the vendor and said ‘I need you to personally own this. I need license keys emailed back to me by this afternoon, is that possible? If not, why?’ It was really fun to see government could be pushed a little bit and made to execute in such a timely fashion. It was really encouraging to our team to see that when chips are down and our backs were up against the wall and we needed to execute, we executed fantastically.”
The end results was DOT upgraded its virtual private networks, virtual desktop interface, tripled its bandwidth to 6 gigabits from 2 gigabits and sent employees home with 15,000 laptops.
“Pre-COVID, our network was designed and built for an inside out usage and architecture where everyone comes to work, they are inside the building and work from inside out, while services comes in. When we reversed that, we were faced with a network that was built to be inside out and a workforce that was outside in,” he said. “So we were routing all the traffic from our home users via VPN or VDI internally and then back out externally to our application or solution providers. We looked at this and said there are better ways. We can bypass some of our network traffic and do split tunneling to the Microsoft cloud because that’s secure and encrypted end-to-end. We continue to make tweaks like that on our network, and certainly they are more application specific, asking if this is the best way to give this to our customers?”
The results of these efforts became evident very quickly, and something Cote said wouldn’t have been possible without a lot of extra work without the network consolidation and modernization.
“Pre-COVID-19, we never had more than 3,000 or 4,000 folks telework in a day. Now we are nearing 10,000 a day and have had zero issues,” Cote said. “We are looking at the usage of video teleconference solutions where again, pre-COVID we had a couple of hundred users at any one time, but now we have thousands of people using it and having VTC meetings. It’s cool to see how our user community has quickly adapted to the situation and are using these tools that we’ve presented to them to work with.”
Cote said the biggest lesson that’s been reinforced from this entire experience is the importance of being personally invested and accountable. He said picking up the phone and having a conversation means you are personally invested and not just passing the issue along and following normal channels.
“Those channels are important for various reasons. You need checks and balances. You need oversight and ensure you are following proper processes and procedures, but it doesn’t mean you can’t move at speed if you personally own something end-to-end,” he said.