Data (GOFS 3.1 expt 53.X)

81 views
Skip to first unread message

Ashneel Chandra

unread,
Jan 22, 2021, 10:50:53 AM1/22/21
to fo...@hycom.org

Hi

 

I am planning to use the HYCOM GOFS 3.1 (expt 53.X) product.

I have tried to obtain the data from the NetcdfSubset link but since the data I need is so large, I need to download multiple small chucks of it, which is taking a lot of time.

Is it possible for a unique link (or fewer links) to be generated for the data I require so I can download the data I need with this one or at least fewer links?

Specifically, I needed the temperature, salinity and horizontal current profiles for the region, longitudes: 40°E - 80°W and latitudes 15°S - 15°N (equatorial Indian and Pacific Ocean) at the full spatial resolution (0.08°) and 6 hourly temporal resolution for the full period the data is available (1994 – 2015).

Your assistance or suggestions to obtain the data more efficiently will be highly appreciated.

 

Sincerely

cidimage001.png@01D66FDD.D4F80D50

Ashneel Chandra
PhD Candidate in Climate Dynamics
Geophysical Institute
University of Bergen
Phone: +47 463 63094
uib.no/en/persons/Ashneel.Chandra

 

 

Michael McDonald

unread,
Jan 26, 2021, 7:18:47 PM1/26/21
to Ashneel Chandra, fo...@hycom.org
Please share what progress you have made thus far (downloading data) and we can go from there to help determine your issues and maybe find a more optimized method.

Ashneel Chandra

unread,
Jan 26, 2021, 7:54:43 PM1/26/21
to Michael McDonald, fo...@hycom.org

Hi Michael

 

Thanks for getting back to me.

I have written a script which downloads every second file (6 hourly) on the ftp site. After a file is downloaded it extracts the region I need and saves it to a predefined running variable and then the downloaded file is deleted to save storage.

While this method works, it takes about a week for me to get one year of data, as I am downloading the whole file when I only need a part of it. The NetcdfSubset  option is an alternative but it only allows me to download about a few days of data at a time which seems tedious.

Any suggestion to obtaining the data more efficiently will be appreciated.

 

Cheers

Ashneel

Michael McDonald

unread,
Feb 1, 2021, 5:59:13 PM2/1/21
to Ashneel Chandra, fo...@hycom.org
Ashneel,
If you substitute the NCSS method for acquiring the same file you are requesting now from FTP and apply the same subsetting options you are currently performing no the server-side then that should be all you need to speed things up. You will still need to get "one time slice aka hourly" at a time via NCSS in a for loop (a tripple nested for loop, one for the year, one for the day of the year (use date command in Linux to convert DoY to the expected value YYYYMMDD, and one for the hour tHHH), which you seem to already have worked (correct)? Making these simple/quick subsets to a single NetCDF object via NCSS will be quicker than getting the whole file, subsetting local and then discarding the original (depending on how much info you are eliminating with the subset).

To get a similar URL from NCSS using the non-aggregated data files you start at the top level THREDDS catalog and then work your way down until you get a single NCSS form page for *one file*, the URL at the bottom of the NCSS page is what you will feed into your for loop script and just change update the time/path in the URL (note that the YEAR will be in two places as it is in the file/dir path).

the file format is e.g.,
YYYYMMDD12_tHHH



e.g.,
* Unaggregated * 
"All Data/"


then choose the "NetcdfSubset" in the Access list,

in the "wget"-able URL (see bottom of NCSS form once you apply your subset options) you can test this in any browser to verify that the output NetCDF file is what you want, then place this in a for loop and replace the YYYY in the path with a variable, and the YYYYMMDD12_tHHH part of the file with your 

e.g, this URL 


can be simplified to the following since there is only one temporal value available in the NCSS dataset.

https://ncss.hycom.org/thredds/ncss/datasets/GLBv0.08/expt_53.X/data/2015/hycom_GLBv0.08_539_2015010112_t000.nc?var=surf_el&temporal=all&accept=netcdf4




Please read the NCSS manual here,
--
Michael McDonald
HYCOM.org Administrator

Ashneel Chandra

unread,
Feb 10, 2021, 4:29:53 AM2/10/21
to Michael McDonald, fo...@hycom.org

Thanks for the detailed walkthrough Michael.

I am still implementing this method, but it does seem to be faster and more efficient.

Thanks again for all your assistance.

Reply all
Reply to author
Forward
0 new messages