Python HYCOM Data Attraction (Attn: Michael McDonald)

[Christopher Edwards]

Dear Michael McDonald,

I hope you are doing well. My name is Christopher Edwards, and I am a student currently working on a marine science research project focused on coral reef environments. As part of my research, I am attempting to download and process daily HYCOM hindcast data from the Gulf of Mexico (GOMu0.04) via the THREDDS data server.

Specifically, I am accessing data from the following URL pattern for multiple reef sites and years (2019–2023):
https://tds.hycom.org/thredds/dodsC/GOMu0.04/expt_90.1m000/DAILY/archv.{year}_{doy:03d}.nc

However, during the download and processing using Python and xarray, I consistently encounter errors such as:

• NetCDF: Malformed or inaccessible DAP2 DDS or DAP4 DMR response
• unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR

These errors occur for multiple daily files (e.g., days 12, 13, 14, etc. in 2019) and result in incomplete data retrieval for my analysis. I have attempted to handle these issues by skipping bad files, but the problem persists across many days and reef locations.

I would appreciate any guidance you can provide on:

• Whether these errors reflect temporary server or file issues, or if certain daily files are known to be incomplete or corrupted
• Recommended best practices for bulk downloading or accessing this dataset through the THREDDS/OpenDAP services
• Any alternative data archives or methods to obtain continuous, reliable data for the Gulf of Mexico region
• Relevant access policies, rate limits, or maintenance schedules that I should be aware of

Thank you very much for your time and assistance. Please let me know if you need any additional information about my project or data access methods.

Best regards,
Christopher Edwards

University of Arkansas-Fort Smith

Biology Major: Environmental Focus

[Michael McDonald]

Christopher,

Please show the steps you took to obtain this mystery OPENDAP URL via. Show the starting THREDDS URL that you are on, where this OPENDAP URL can be obtained? Also include any screenshots if they could help in tracking down the issue you are having. 

?  https://tds.hycom.org/thredds/dodsC/GOMu0.04/expt_90.1m000/DAILY/archv.{year}_{doy:03d}.nc