Inaccessible Data

[Nick Januario]

Dear OpenDap,

My name is Nicholas Januario, and I am emailing you regarding potentially broken links to access data. For my master's project/thesis, I have been downloading temperature data to calculate heatwave occurrences. I was doing this by downloading the tile subsets from the Daymet database where my study regions resided. As of late, I cannot download those files/tile data anymore, and it produces files with 0 kb of data. I investigated this, and it led me to check the actual links where the data would normally reside. This link, for example: https://thredds.daac.ornl.gov/thredds/fileServer/ornldaac/2129/tiles, leads to a bad request 400. Going even further and trying to directly download specific tiles or specific years all lead to the same issue. I can't remember when I started not being able to download the data/not access the links directly to download it. If you could look into this issue with the links, that would be greatly appreciated. Thanks.

Sincerely,

Nicholas Januario

[Miguel Jimenez]

Hi Nicholas, 

The issue you are experience is frustrating and I would like to help you access the data as soon as possible. Thanks for providing the url, I can certainly confirm I cannot open that in the browser. Just so you know, NASA is in the process of migrating their data archives to the cloud, and with it the migration of APIs to the cloud. So many opendap servers are experiencing some potentially disruptive changes, for example their URL and redirecting to the new endpoint...

Here are a few things you can do, that NASA recommends for finding data, and you can accommodate to your data needs

1) Identify the collection concept id or DOI of your dataset. You can search for it using the Earthdata search engine. For example, 

https://www.earthdata.nasa.gov/data/projects/daymet/data-access-tools

Provides several collections options, clicking on one of them will provide the necessary information (both DOI and Concept ID). You can see them for example in the following link

https://www.earthdata.nasa.gov/data/catalog/ornl-cloud-daymet-daily-v4r1-2129-4.1

2) With this information, you can now use several resources to identify the most up-to-date location for your files and their opendap API. For example the Earth Data search engine. But

if you are familiarized or interested in python tools, we recommend pydap to search for your opendap urls. Pydap has a relevant function to find urls: get_cmr_urls. This takes the DOI, Concept ID (official named collection concept ID) as required arguments, and can also take optional arguments like dates, etc, and provides a list of opendap urls.

There are several examples for searching opendap urls in the following only resources:

a)  https://github.com/OPENDAP/NASA-tutorials/blob/main/binder/CMR_queries.ipynb

b) https://github.com/OPENDAP/NASA-tutorials/blob/main/binder/MERRA-2_Access.ipynb

You can run these examples interactively on your browser by clicking on the launch | binder button on this page if all you need it so find the urls. Clicking on each of those should take you to the familiar OPeNDAP data requests form.

 For example, for a DayMet file I got (notice the .dmr at the end of the url):

https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_pr_dayl_2023.nc.dmr

Let me know if you need further help. We also recommend using pydap with xarray (Python) specific subset and aggregated across many files. This documentation has relevant information, and will experience a major update to best practices that yield performant access to opendap data (even behind authentication).

Thank you for reaching out,

Miguel Jimenez-Urias, PhD
Computational Oceanographer | Sc. Community Director
OPeNDAP, Inc

https://www.linkedin.com/in/MiguelJimenezUrias/