wget for bulk data download

[Clare Ostle (ENV)]

Hello,

I usually use wget within the command line to download data in bulk, and I have been trying to download the ECCO2 mixed layer depth data at this address:

http://ecco2.jpl.nasa.gov/opendap/data1/cube/cube92/lat_lon/quart_90S_90N/MXLDEPTH.nc/contents.html

but because each .nc file requires further action the wget command is not working, can you point me in the right direction to a bulk download URL of this data? Or advise me on using wget for the above URL, I have also tried using wget at the following data URL for this dataset:

wget -q -O - http://ecco2.jpl.nasa.gov:80/opendap/data1/cube/cube92/lat_lon/quart_90S_90N/MXLDEPTH.nc

Thank you for your help,
Clare

[Daniel Holloway]

Clare,

    It appears (to me anyway) that they decided to use the name 'MXLDEPTH.nc' as a directory name, not specifically a file.   I used a browser to go to that location and found a directory full of netcdf files, one per day:

    http://ecco2.jpl.nasa.gov/opendap/data1/cube/cube92/lat_lon/quart_90S_90N/MXLDEPTH.nc/MXLDEPTH.1440x720.19920101.nc

    My expectation would be that you need to iterate over the set of daily files using wget.  First time I've seen a directory named as such, it might definitely cause strange behavior if used like a filename for our server.

    Dan

[Daniel Holloway]

First time I've seen a directory path named with a suffix like this.   We should ensure our handlers are capable of discerning that this is a directory and not a supported filetype.

Dan

Begin forwarded message:

[Nathan Potter]

Clare,

The site in question does not allow (or simply has not enabled) direct (download) access to the underlying data files. You can make DAP data requests using the DAP2 query interface, and wget will do this you just have to provide the DAP2 query string. Probably one of two things is going on:

1) The data provider didn't mean to configure it this way (the default configuration is to deny this type of access and admin has to change the default server configuration in order to allow direct access to the underlying data resource.)

2) They intended to restrict access to the underlying data resources because they want to stop people from doing exactly what you are trying to do. Instead they wish for you to construct DAP2 queries that restrict the requests to a specific subset area/time of interest.

I suspect if you go here:

http://ecco2.jpl.nasa.gov

You'll find links to the administrators of that site and they may be able to clarify their intent, and fix their configuration if it's not inline with their intended access policies.


I hope that helps,


Nathan

= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317

[Nathan Potter]

The handlers work fine (surprising really) with that dir name, but there are other issues. See my reply to Clare.


N