wget script

[Venkat Kolluru]

Hi,

Is there any wget shell script available for downloading HYCOM output?

Thanks

Venkat

 

Venkat S. Kolluru

Technical Director, Surfacewater Modeling Group

 

ERM

350 Eagleview Boulevard, Suite 200

Exton, PA 19341-1180

 

Tel: 610 524 3654

Mob: 610 764 0579

 

www.erm.com

venkat....@erm.com

 

---

This message contains information which may be confidential, proprietary, privileged, or otherwise protected by law from disclosure or use by a third party. If you have received this message in error, please contact us immediately and take the steps necessary to delete the message completely from your computer system. Thank you.

Please visit ERM's web site: http://www.erm.com

[Michael McDonald]

Venkat,

What format do you want the data in (is netCDF your desired output format)?

What method(s) have you tried to download the HYCOM data (e.g., netcdfsubset, opendap, ftp, opendap, etc.)?

[Michael McDonald]

Venkat,

Have you looked at the unidata documentation for creating custom
netcdfsubset URLs (these should be "wget friendly")?

http://www.unidata.ucar.edu/projects/THREDDS/tech/interfaceSpec/NetcdfSubsetService.html

The key to this is not requesting to much data, i.e., only request 1
day of data at a time, max (netcdfsubset service by default selects
the entire time series). You should be able top easily loop through a
time series via separate netcdfsubset+wget requests to create your
dataset (collection of netcdf files with only the domain and variables
you need). The more data you request (whether that be using additional
variables or a longer date range) the longer the request will take to
process (if it ever complets). We reboot the THREDDS service on a
daily basis at noon EST as preventative maintenance and also to update
our legacy catalogs. So, if your request is taking longer than 24
hours to complete/stream, all that time will be for nothing (e.g., you
do not get a partial netcdf file). If you incrementally make small
manageable requests then you will have restart points, in the likely
event of a failure/timeout.

--
Michael McDonald
HYCOM.org Administrator
http://hycom.org


On Wed, Oct 10, 2012 at 10:58 AM, HYCOM.org Forum <fo...@hycom.org> wrote:
> Hi,
>
> I have been downloading netcdf format using netcdf subset approach.
>
> This seems to be more efficient for my spill modeling than downloading for
> entire world.
>
> I am wondering if there is any wget or perl script available to download
> netcdf subset data.
>
> If you recommend some other approach of fast downloading, that will be
> great.
>
>
>
> Thanks
>
> Venkat

> --
> You received this message because you are a member of HYCOM.org
> To ask a question, send an email to fo...@hycom.org

[JDTi...@aol.com]

I have a python class I might be willing to share if you are interested. It might need to be modified to your needs, but I have it downloading batches of temp, sal, and current vectors for me.

[venkat....@erm.com]

Hi,

Many thanks for your response.

Please send me your python code and I will modify to suit my needs.

Regards,

Venkat

[JDTi...@aol.com]

class download_netcdf:

    def __init__(self, date, data):

        

        HYCOM_date = netcdftime.DateFromJulianDay(date+2415384.5)

        year_of_date = HYCOM_date.timetuple().tm_year

        day_of_year = HYCOM_date.timetuple().tm_yday

        file = ('archv.' + str(year_of_date) + '_' + 

                str(day_of_year).zfill(3) + '_00_3z' + data + '.nc')

        

        if os.path.isfile(file):

            print 'File exists!'

        

        else: 

            ftp = ftplib.FTP('ftp.hycom.org')

            ftp.login('anonymous', 'youremailaddress')

            if data == 't':

                directory = ('/datasets/GOMl0.04/expt_20.1/' + 

                             str(year_of_date) + '/' + data + 'emp')            

            else:

                directory = ('/datasets/GOMl0.04/expt_20.1/' + 

                             str(year_of_date) + '/' + data + 'vel')

            ftp.cwd(directory)

            print 'Retrieving ' + file + '...'

            ftp.retrbinary('RETR ' + file, open(file, 'wb').write)

[JDTi...@aol.com]

Let me add... I am not a programmer, so this code might have mistakes in it. To get the HYCOM date from mm/dd/YYYY:

date = datetime.strptime(str(end_date), '%Y-%m-%d')

date = netcdftime.JulianDayFromDate(end_date) - 2415384.5

Then it is simply called as:

download_netcdf(date, 'u')

if you want the u data (also accepts 't', 'v', and 'w'...sorry no salinity in there). Let me know if I left anything important out. You'll need to import ftplib, os, and netcdftime at least.

[JDTi...@aol.com]

Sorry, I missed you are trying to get a spatial subset also. Maybe you're lucky enough to only be dealing with the GOM as in my class. Otherwise, I hope you might still find the code useful.

[Flav]

Venkat,

I use this kind of wget script to get some MED data easily readable by ncview after.
hope this helps.

Flav


----------------------------------------------------------------
#!/bin/bash
# Flav's script to get hycom data - 2012

#http://ncss.hycom.org/thredds/ncss/grid/GLBa0.08/expt_90.9/dataset.html
#http://ncss.hycom.org/thredds/ncss/grid/glb_analysis
#options qui existent:
#
#var2d=emp,mld,mlp,qtot,ssh
#var3d=salinity, temperature, u, v
#spatial=bb&north=46&south=30&west=-7.5&east=37 pour la med
#time_start=present ou du type: 2011-01-03T00:00:00Z
#time_end=present   ou du type: 2011-01-04T00:00:00Z
#addLatLon          on ajoute lon et lat au fichier
#horizStride=2      on prend 1 point sur ...

# pour horizStride=3, 1 seul temps et ssh (2D) il faut 1s de download - 188K
# pour horizStride=1, 1 seul temps et ssh (2D)  il faut 4.7s de download - 1.6M
# pour horizStride=3, 1 seul temps et temperature (3D) il faut 6.5s de download - 2.1M
# pour horizStride=1, 1 seul temps et temperature (3D) il faut 60s de download - 19M

mmth=1
dday=1
cpt=1

while [ $mmth -le 12 ] ; do  # loop sur le mois
  while [ $dday -lt 32 ] ; do  # loop sur le jour
     
    if [ ${mmth} -lt 10 ] ; then
      if [ ${dday} -lt 10 ] ; then
        if [ ${cpt} -lt 10 ] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-0${mmth}-0${dday}T00:00:00Z&time_end=2010-0${mmth}-0${dday}T00:00:00Z&horizStride=4&addLatLon" -O med00${cpt}.nc 
        fi
        if [[ ${cpt} -ge 10 && ${cpt} -lt 100 ]] ; then
            wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-0${mmth}-0${dday}T00:00:00Z&time_end=2010-0${mmth}-0${dday}T00:00:00Z&horizStride=4&addLatLon" -O med0${cpt}.nc 
        fi     
        if [ ${cpt} -ge 100 ] ; then
            wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-0${mmth}-0${dday}T00:00:00Z&time_end=2010-0${mmth}-0${dday}T00:00:00Z&horizStride=4&addLatLon" -O med${cpt}.nc 
        fi          
      fi
      if [ ${dday} -ge 10 ] ; then
        if [ ${cpt} -lt 10 ] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-0${mmth}-${dday}T00:00:00Z&time_end=2010-0${mmth}-${dday}T00:00:00Z&horizStride=4&addLatLon" -O med00${cpt}.nc 
        fi
        if [[ ${cpt} -ge 10 && ${cpt} -lt 100 ]] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-0${mmth}-${dday}T00:00:00Z&time_end=2010-0${mmth}-${dday}T00:00:00Z&horizStride=4&addLatLon" -O med0${cpt}.nc 
        fi
        if [ ${cpt} -ge 100 ] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-0${mmth}-${dday}T00:00:00Z&time_end=2010-0${mmth}-${dday}T00:00:00Z&horizStride=4&addLatLon" -O med${cpt}.nc 
        fi
      fi
      let "dday += 1"
      let "cpt += 1"
    fi
   
    if [ ${mmth} -ge 10 ] ; then
      if [ ${dday} -lt 10 ] ; then
        if [ ${cpt} -lt 10 ] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-${mmth}-0${dday}T00:00:00Z&time_end=2010-${mmth}-0${dday}T00:00:00Z&horizStride=4&addLatLon" -O med00${cpt}.nc 
        fi
        if [[ ${cpt} -ge 10 && ${cpt} -lt 100 ]] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-${mmth}-0${dday}T00:00:00Z&time_end=2010-${mmth}-0${dday}T00:00:00Z&horizStride=4&addLatLon" -O med0${cpt}.nc 
        fi 
        if [ ${cpt} -ge 100 ] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-${mmth}-0${dday}T00:00:00Z&time_end=2010-${mmth}-0${dday}T00:00:00Z&horizStride=4&addLatLon" -O med${cpt}.nc 
        fi 
      fi
      if [ ${dday} -ge 10 ] ; then
        if [ ${cpt} -lt 10 ] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-${mmth}-${dday}T00:00:00Z&time_end=2010-${mmth}-${dday}T00:00:00Z&horizStride=4&addLatLon" -O med00${cpt}.nc 
        fi
        if [[ ${cpt} -ge 10 && ${cpt} -lt 100 ]] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-${mmth}-${dday}T00:00:00Z&time_end=2010-${mmth}-${dday}T00:00:00Z&horizStride=4&addLatLon" -O med0${cpt}.nc 
        fi
        if [ ${cpt} -ge 100 ] ; then
      wget "http://ncss.hycom.org/thredds/ncss/grid/glb_analysis?var=ssh&spatial=bb&north=46&south=30&west=-7.5&east=36&time_start=2010-${mmth}-${dday}T00:00:00Z&time_end=2010-${mmth}-${dday}T00:00:00Z&horizStride=4&addLatLon" -O med${cpt}.nc 
        fi
      fi
      let "dday += 1"
      let "cpt += 1"
    fi
  done # end loop sur jour
  let "mmth += 1"
  let "dday = 1"
done # end loop sur mois

#EOF