HYCOM download issues - attn: Michael McDonald

[Nowak Heather]

Hi Michael,

 

We have been downloading the HYCOM GLBv0.08 expt 53.x. version of Matlab for research purposes for the last several years. We have internal scripts that utilize the OPeNDAP site to download data for specific location. Usually we do this without much issue, although at times it can be a bit slow. However, we have noticed significant issues trying to download HYCOM over the past week. We usually download the entire 22 year time series in about 2-3 days, but recently we’ve gotten a lot of errors and have only successfully downloaded 1 year worth of data, which took about 48 hours to complete. Can you advise if there are any known issues with this version of HYCOM currently or the platform we use to obtain the data? Do you have any suggestions on how to get this rectified so we can continue to download data for our client’s needs?

 

Any information would be greatly appreciated.

 

Regards,

Heather Nowak

 

Heather Nowak Oceanographer Tel: +1 281 545 2322   Mob: +1 888 954 5644 115 Serrato Lane Bldg C Richmond, TX 77469 - USA woodsholegroup.com       energyoffshore-cls.com

 

 

 

[Michael McDonald]

Heather,

This is a known issue. Please reduce your data requests to smaller chunks (single time requests, non-aggregated) and extend your client timeouts (3~10x to be safe). 

https://groups.google.com/a/hycom.org/g/forum/c/B_dbzaCHy60/m/j09YnhAkBQAJ

--
--
You received this message because you are a member of HYCOM.org
To ask a question, send an email to fo...@hycom.org

--

Michael McDonald
HYCOM.org Administrator

[Richard Arena]

Hi Michael,

 

Just a follow up on this question/ response. I wrote a few lines of code in Python to extract data from the .nc files on data.hycom.org using xarray. Each call requests 36 values of water_u and water_v. There is no issue with water_u, although when I try to get water_v I get a timeout. I investigated this more and tried extracting single values for water_v, everything is fine until I get to depth level 33 and then I get a timeout. This is a strange behavior that I have never run into before while downloading any data. I was wondering if you have any thoughts about this.

 

Thanks,

Rich

 

From: Michael McDonald <michael....@hycom.org>
Sent: Saturday, October 14, 2023 3:08 PM
To: Nowak Heather <hno...@woodsholegroup.com>
Cc: fo...@hycom.org; Richard Arena <rar...@woodsholegroup.com>
Subject: Re: HYCOM download issues - attn: Michael McDonald

 

You don't often get email from michael....@hycom.org. Learn why this is important

CAUTION: This message comes from an external server, do not click on links or open attachments unless you know the sender and are sure the content is safe.

[Michael McDonald]

Rich,

Please share your code making these xarray calls. The "data.hycom.org" service is for whole/complete file downloads. You should likely be using the pydap/opendap calls to "tds.hycom.org" (thredds/opendap) data URLs to make these subsets/queries more efficient. Otherwise, your xarray calls might be downloading the entire snapshot from the component files (ts3z, uv3z, ssh, etc) via a regular curl request. 

[Richard Arena]

Hi Michael,

 

First, I just want to clarify that my experiments with downloading do not reflect the method that Heather has been using for years.

 

As part of a project we are working on in the Gulf of Mexico, Alexandra Bozec from FSU shared some high resolution model results with us.

 

Index of /pub/BOEM/GOMb0.01/data (hycom.org)

 

I used python requests and Beautiful Soup to parse the webpage and get a list of the file names to loop through. I did not need all the results and downloading the individual files would have been expensive. Then a couple of lines of code,

 

data =  xr.open_dataset(file+'#mode=bytes', decode_times=False)

 

D = data.sel(lon=lon, lat=lat, depth = Depths, method="nearest")

 

I then used the variables in D, u and v, to do my calculations and make some plots. This worked just fine with no issues.

 

When I tried this same method with GLBv0.08 EXPT 51.X I ran into issues, but only with the water_v variable which I thought was strange.

 

I then tried working from the thredds server,

 

https://tds.hycom.org/thredds/dodsC/datasets/GLBv0.08/expt_53.X/data/1994/

 

Again, I also had issues with the water_v variable not with water_u. I have used pydap in the past but found no significant difference in either method. I will give pydap a go with this issue and see what the results are. I will let you know what I find out.