reader for OpenDAP binary object

[Ed Zaron]

Hello All,

Can anyone suggest a client for reading the OpenDAP "binary" data served
from the default OpenDAP web interface? These files have extension .dods
and they are returned by clicking the "Get Binary" button at the top of
the dataset .html url form.

-Ed

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Edward D. Zaron
Research Assistant Professor
Department of Civil and Environmental Engineering
Portland State University
Portland, OR 97207-0751
Phone: (503)-725-2435
FAX: (503)-725-5950
za...@cecs.pdx.edu
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Gallagher James]

On Jan 2, 2011, at 10:04 PM, Ed Zaron wrote:

>
> Hello All,
>
> Can anyone suggest a client for reading the OpenDAP "binary" data served from the default OpenDAP web interface? These files have extension .dods and they are returned by clicking the "Get Binary" button at the top of the dataset .html url form.

Ed, You can point anything that reads netCDF and is using the newest version of that library (4.1) at those files and read them. Otherwise, you can use the getdap tool that's bundled with libdap to read them, albeit as ASCII, and use a little creative piping to get the data into binary form. Note that starting with version 1.5, our server can return a netCDF file in addition to the .dods response.

James

>
> -Ed
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Edward D. Zaron
> Research Assistant Professor
> Department of Civil and Environmental Engineering
> Portland State University
> Portland, OR 97207-0751
> Phone: (503)-725-2435
> FAX: (503)-725-5950
> za...@cecs.pdx.edu
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>

--
James Gallagher
jgallagher at opendap.org
406.723.8663

[ed zaron]

Thanks, Ellyn and James,

My usual tools are ncdump, ncview, nco, and snctools.

I'm using netcdf 4.1.2-beta2 currently:

ezaron$ nc-config --version

netCDF 4.1.2-beta2-snapshot2010111822

Here are the results when I try to open the .dods file:

ezaron$ ncdump -k dataset-duacs-nrt-over30d-global-merged-madt-h.dods 

ncdump: dataset-duacs-nrt-over30d-global-merged-madt-h.dods: NetCDF: Unknown file format

ezaron$ ncks -A -x dataset-duacs-nrt-over30d-global-merged-madt-h.dods tmp.nc

ERROR NC_ENOTNC Not a netCDF file

Etc.

My goal is to have the latest netcdf functionality (ability to read and write in compressed file format). And it would be convenient to download chunks of data form the OpenDAP server in the binary or netcdf format.

I haven't tried the java-based tools in a while, but I may look at ncBrowse again.

Could the OpenDAP server be old? I don't know how to query its version, but the download data web form states,

"Tested on Netscape 4.61 and Internet Explorer 5.00".

Thanks,

Ed

[Roy Mendelssohn]

nco, ncbrowse, ferret, and number of tools (including the present version of the netcdf library) should be able to bring in the data directly, you do not need to download the .dods file and then access it. I assume you are going to the OPeNDAP html response page, deciding on the subset, and then selecting "Download binary". Instead, decide on the subset, copy the URL that is shown toward the top of the form,. and feed that to either nco or snctools. I am not certain I still have snctools installe din my matlab, but it should be able to import the data from the URL directly into Matlab (make certain you use the OpeNDAP enabled version).

Or, since you use Matlab, too other toolboxes that can access remote data directly include njtbx (http://sourceforge.net/apps/trac/njtbx/wiki/DownloadNjtbx-current) and nctoolbox (http://code.google.com/p/nctoolbox/).

There also is "loaddap" from OpeNDAP (http://www.opendap.org).

If this isn't clear I can provide and example.

-Roy M.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Men...@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

[John Caron]

Hi Ed:

You dont want to work directly with the .dods file, rather use the opendap URL. So you would do

  ncdump: -k http://server/something/whatever.nc

this provides remote access. For local access, download a netcdf file, then work with that.

Also, you can use the netcdf-Java library, the TOOLSUI GUI for it is at

   http://www.unidata.ucar.edu/packages/netcdf-java/

Or lots of other tools as Roy says. The key is to use the URL.

John

[Gallagher James]

On Jan 3, 2011, at 6:41 PM, ed zaron wrote:

Thanks, Ellyn and James,

My usual tools are ncdump, ncview, nco, and snctools.

I'm using netcdf 4.1.2-beta2 currently:

ezaron$ nc-config --version

netCDF 4.1.2-beta2-snapshot2010111822

Here are the results when I try to open the .dods file:

ezaron$ ncdump -k dataset-duacs-nrt-over30d-global-merged-madt-h.dods 

ncdump: dataset-duacs-nrt-over30d-global-merged-madt-h.dods: NetCDF: Unknown file format

ezaron$ ncks -A -x dataset-duacs-nrt-over30d-global-merged-madt-h.dods tmp.nc

ERROR NC_ENOTNC Not a netCDF file

Ed,

Sorry for the confusion. I'm CC'ing Dennis Heimbigner on this since he is the main developer of the netcdf 4.x opendap code. I'm pretty sure that you can read a .dods file using netcdf 4.1 with a file:/... kind of a URL. 

Dennis: Am I correct or do I have this confused with something else?

FWIW: Both Roy and John are correct that you can use the http://... urls with any of those applications that use netCDF, and that is really what we intended to be the main way. I believe the file:/ form of the URL was added primarily for testing.

[Ed Zaron]

Hi Dennis,

I have attached the .dods file for you. It is just a subset of the
complete file and only contains the coordinate variables and dimensions.

Regarding Roy's comment that I should access the data through the usual
http schema: I agree, but I'm making lots of plots with the data and I'm
not sure how many times I will need to re-load it to figure out the
precise subset to display, etc. I think the provider will be changing
the url for the data shortly, so I'd rather spend the bandwidth getting
files now.

By the way, my question is not at all urgent -- I found a way to get the
netcdf files via wget/ftp. But this is the second time in as many months
that I wanted to get the netcdf files from the OpenDAP server, and I
couldn't find any documentation about how to use the .dods file. The
workaround with wget/ftp is, to me, inelegant compared to the nice
interface available through OpenDAP-enabled netcdf.

Thanks for having a look at this.

Regards,

Ed

Dennis Heimbigner wrote:
> Can one of you send me a copy of
> the file: dataset-duacs-nrt-over30d-global-merged-madt-h.dods
> =Dennis Heimbignefr

[Ed Zaron]

-- oops. Here it is ...

Dennis Heimbigner wrote:
> Can one of you send me a copy of
> the file: dataset-duacs-nrt-over30d-global-merged-madt-h.dods
> =Dennis Heimbignefr
>

[Roy Mendelssohn]

Hi Ed:

What I think you are missing is the that .dods structure is what is sent by the OpeNDAP protocol, and the tools everyone has mention knows how to read this structure and transform it into files. So if you use nco, you can access it and save it into netcdf. Same with many of the other tools. If you use one of the matlab tools, you can save it into any one of a number of formats.

HTH,

-Roy

> --
> You received this message because you are subscribed to the Google Groups "OPeNDAP" group.
> To post to this group, send email to ope...@opendap.org.
> To unsubscribe from this group, send email to opendap+u...@opendap.org.
> For more options, visit this group at http://groups.google.com/a/opendap.org/group/opendap/?hl=en.

[James Gallagher]

On Jan 3, 2011, at 7:16 PM, Gallagher James wrote:

On Jan 3, 2011, at 6:41 PM, ed zaron wrote:

Thanks, Ellyn and James,

My usual tools are ncdump, ncview, nco, and snctools.

I'm using netcdf 4.1.2-beta2 currently:

ezaron$ nc-config --version

netCDF 4.1.2-beta2-snapshot2010111822

Here are the results when I try to open the .dods file:

ezaron$ ncdump -k dataset-duacs-nrt-over30d-global-merged-madt-h.dods 

ncdump: dataset-duacs-nrt-over30d-global-merged-madt-h.dods: NetCDF: Unknown file format

ezaron$ ncks -A -x dataset-duacs-nrt-over30d-global-merged-madt-h.dods tmp.nc

ERROR NC_ENOTNC Not a netCDF file

Ed,

Sorry for the confusion. I'm CC'ing Dennis Heimbigner on this since he is the main developer of the netcdf 4.x opendap code. I'm pretty sure that you can read a .dods file using netcdf 4.1 with a file:/... kind of a URL. 

Dennis: Am I correct or do I have this confused with something else?

...a quick update: 

The OC library, which is used by netCDF 4.x to read from DAP servers, does indeed know how to read from those saved binary files but a bug crept into the version used in the netcdf 4.1 distribution that broke that feature. If it's important for your work (i.e., you have a ton of data in those .dods files already), we can help you build netcdf 4.1 from source using a new version of OC.

Let me know,

James

--

James Gallagher

jhrg at mac.com

[ed zaron]

Hi James,

Thanks a lot. If you can point me to a netcdf + OC source tarball, I'd appreciate it. I am using the compression features of the latest netcdf, so I'd like to build with these, if possible.

My desire to open a .dods object is more of a convenience issue than anything else. So far I have been able to find other ways to make a local copy of the desired netcdf file, or I just subset with the usual opendap server functionality.

Thanks again,

Ed