Cloud Exchange 2024: NOAA’s Adrienne Simonson, Patrick Keown on sharing massive datasets with the public

The National Oceanic and Atmospheric Administration collects tens of terabytes of data every day from satellites, radars, ships, weather models and more.

All that data adds up. NOAA currently holds about 56 petabytes of data across hundreds of datasets.

The agency’s mission involves gather and sharing information about everything from the surface of the sun to the bottom of the ocean, and a wide array of industries rely on its datasets to do business.

That’s why it created the NOAA Open Data Dissemination (NODD) Program, which provides public access to the agency’s open data through a partnership with major commercial cloud platforms.

“NODD makes NOAA assets available to the public at no charge. We publish those data assets that are machine-readable. We maintain the data inventory. We engage with the public on the use of the data in order to expand the use of the data. And we provide the public with an opportunity to ask for the datasets they might want,” NODD Director Adrienne Simonson said during Federal News Network’s Cloud Exchange 2024. “Users can write in and talk to us about their challenges, and we’re going to answer their questions.”

Consumers of NOAA data includes researchers and scientists, who rely on it to analyze climate change and disaster recovery, for instance.

“We’ve gotten overwhelmingly positive support for data dissemination that we’re doing, the data that is out in the cloud and the access that we’re providing. That comes because we’re deeply engaged with those users,” NODD Program Manager Patrick Keown said.

“It certainly is a challenge to store just the sheer volume of the data,” he added. “One of our biggest issues is the fact that that’s rapidly changing, and we don’t know the volumes of the data that are coming. That has been one of our bigger challenges.”

Simonson said agriculture, insurance, renewable energy, retail and transportation industries all rely on NOAA data.

“Those folks are using the data for things like strategic operational planning. They’re also using it for risk management and supply chain management, and to create innovative products for their own customers,” she said. “So the scalability and the access really allow NOAA to essentially support the economy and spur innovation.”

The NODD Program supports the data needs of a wide range of businesses.

“We have heard from a lot of folks who are startups that say that they wouldn’t otherwise be able to have started the companies, except for that the data was free. We also hear from Fortune 500 companies, so we’ve got quite a spread there going on,” Simonson said.

The wide range of uses of the data is often staggering, Keown added, recounting being on a panel when someone from a large appliance maker discussed the importance of public access to NOAA data. The panel member pointed out that the NODD data helps it ensure that its refrigerators run at the right temperatures.

The panelist told Keown: “The only way for us to do that efficiently and to reduce our greenhouse gas emissions, and all of these other things, is to understand what the weather’s going to be like. And we can only do that if data is unfettered.”

How NODD makes ‘data democratization’ possible

NODD started out in 2015 as the agency’s Big Data Project, with agreements with Amazon Web Services, Google, IBM, Microsoft and a nonprofit called Open Commons Consortium.

In 2021, the project was rebranded as NODD as it began to focus on providing the public with free and easy data access via the cloud.

Keown said NOAA continuously improves NODD based on feedback from its users and addresses concerns around access and search.

“Some users really enjoyed that kind of user interface, feeling like they could find what they needed. But others, especially large private industry, large research industry, they needed completely open access. And they were the ones that were providing early on overwhelmingly positive feedback to us,” he said.

NODD is in its fifth year of essentially a 10-year contract. The contract is a two-year base contract, with four option periods of two years each.

“We’re looking to take on more datasets. We’re also looking to continue to engage with our users, to make sure they understand how to use the data. While we speak, though, there are teams within NOAA that are working with each other and with the cloud service providers to develop the future data dissemination,” Simonson said.

Among the program’s goals, Simonson said NODD supports “data democratization.”

NOAA wants to keeping expanding the access to the data to more people, she said.

“One of the reasons why this got started initially was because NOAA has a lot of data already that people were struggling to get to, and we were seeing the exponential rise of data volumes,” she said.

NOAA’s evolving cloud infrastructure

Keown said NOAA has been expanding its cloud presence over the past five years.

“All of the line offices within NOAA and the organization have worked at a different pace — at the pace that they felt they could be both innovative but also support operations. The National Weather Service is a very operational component. There’s life and property that are at stake, so making major infrastructure changes take a little bit longer. Others like the National Ocean Science Service, who really are often at the forefront of open science and trying to find new innovative ways to do things, they were able to maybe adopt cloud a little bit differently,” Keown said.

As NODD has evolved, the program office’s goal has been “to help those offices get to the cloud and give them this infrastructure to at least say, ‘Here, we’re going to utilize this for dissemination of data until we can figure out how to maybe do this internally.’ ”

All the agency’s offices now “are coming up to speed of how to build this cloud infrastructure, this cloud environment to support their needs and ultimately support the needs of the users — many of them new and different types of users than they originally had,” he added.

Keown said NOAA ensures all of the datasets in the NODD Program have full and accurate descriptions and that data users can follow up with the NOAA component agencies that generate the data.

“It also gives them the ability to get back to the owner of the data. For instance, if it’s a dataset that belongs to the National Marine Fisheries Service, it can directly link them back there. It could also directly link them to the cloud offering where they can access that data,” he said.

Keown said NOAA continues to refine NODD based on user feedback, and the agency is looking at ways to make its data more accessible and valuable to the private sector and the public.

“When we start to talk about data in five years, or data in 10 years, we’re reaching out to every portion of the organization, every program. We work with the data owners to better understand what new datasets they’re going to bring online,” he said.

“Then, we have to work to create strategies, whether they’re cloud strategies, or data strategies, or strategies for AI — things that would give us the ability to make those decisions long term for the organization. And I think we have to be a little more flexible than probably the government has been in the past. We used to develop 10- and 20-year plans, and now we see ourselves developing two- to five-year plans that often change in a year or two.”

Discover more articles and videos now on Federal News Network’s Cloud Exchange 2024 event page.

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.