Downloading in Python

[Katherine Hedstrom]

Hi all,

My python scripts for downloading files are no longer working and I am hoping that you can tell me how to fix them. Namely, the URL I am attempting to use is:

http://ncss.hycom.org/thredds/ncss/GLBa0.08/expt_91.1/2015/salt?var=salinity&north=89.9777&south=44&east=1019.1199&west=74.1199&disableProjSubset=on&horizStride=1&time_start=2015-01-01T00%3A00%3A00Z&time_end=2015-01-01T00%3A00%3A00Z&timeStride=1&vertCoord=&addLatLon=true&accept=netcdf

The error I get is:

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: Validation^ errors: No variables requested --
Traceback (most recent call last):
  File "get_hycom_GLBa0.08_salt_2015.py", line 78, in <module>
    dataset = netCDF4.Dataset(url)
  File "netCDF4/_netCDF4.pyx", line 2111, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1731, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -77] NetCDF: Access failure: b'http://ncss.hycom.org/thredds/ncss/GLBa0.08/expt_91.1/2015/salt?var=salinity&north=89.9777&south=44&east=1019.1199&west=74.1199&disableProjSubset=on&horizStride=1&time_start=2015-01-01T00%3A00%3A00Z&time_end=2015-01-01T00%3A00%3A00Z&timeStride=1&vertCoord=&addLatLon=true&accept=netcdf'

Cheers,

Kate

[Michael McDonald]

Kate,
Your NCSS URL just worked for me subsetting & downloading a ~560MB
output file. Note: if you change the URL to "accept=netcdf4" then this
reduces the NCSS request to only 66MB, which helps (significantly)
lessen the NCSS server load and the data download between hycom
servers and your client.

Please share *more* (ideally all) of your python code that is making
these requests. We typically see NCSS requests in a separate
wget/curl/browser client and then read into the applicaiton.
Alternatively, you can use the native python modules to natively read
and subset (client-side) the data directly via OPeNDAP and elimate the
NCSS (server-side) bottleneck entirely.

--
Michael McDonald
HYCOM.org Administrator

[Katherine Hedstrom]

Hi Michael,

I tried the opendap and failed there too. I attach the script as it stands. It used to work with Python 2 and the old web site. I am now trying with Python 3. I know there's work to be done to switch to the month/day style, but so far I was just trying to get the first chunk of it working. This was not written by me...

Kate

[jason....@usm.edu]

Does something like this work for you?

lon = dataset['Longitude'][:,:]

[jason....@usm.edu]

Sorry. I didn't look close enough. It seems you are trying to read your netcdf in the code I first suspected. If your trouble is downloading, I do something similar:

from pydap.client import open_url

data = open_url(url)

# this part may be formatted differently depending on the data structure

temp = data['temp'][:][..., temp_indices]

I'm not sure if it is the same on Python 3. If you could share the error output, someone may be able to be more helpful.