unable to read downloaded files from OPeNDAP

[Prater, Mark]

Hello,

 

I've downloaded a file using the OPeNDAP Dataset Access Form, with the following Data URL:

 

http://sift.pmel.noaa.gov/thredds/dodsC/uncompressed/ki_026_a_ha.nc?      lon[0:1:645],lat[0:1:719],time[0:10:1440],ha[0:10:1440][0:1:719][0:1:645]

 

and the data (seems to) download without problem.  The resulting file size is about 263MB.  However, I cannot read this file - which I assume is a standard netCDF file - regardless of the software I use.

 

I've tried Matlab's function

 

   nc = netcdf.open( 'ki_026_a_ha.nc', 'NC_NOWRITE' )

 

and get the error

 

   Unable to open netCDF file, library error message "NetCDF: Unknown file format"

 

I tried the third party functions from nctoolbox

 

   nc = ncdataset( 'ki_026_a_ha.nc' )

 

with the error

 

   java.io.IOException: Cant read ki_026_a_ha.nc: not a valid CDM file.

 

I tried to open the file in IDV, and got the error message

   There was an error loading the data:

  Error creating data source:file.grid with: C:\Work\Projects\Tsunami\MOSTdata\ki_026_a_ha.nc

   Grid data source failed making data set: C:\Work\Projects\Tsunami\MOSTdata\ki_026_a_ha.nc

   Cant read C:/Work/Projects/Tsunami/MOSTdata/ki_026_a_ha.nc:  not a valid CDM file.

 

I've made the same attempts with a similar data set, with the same results.

 

Any suggestion? Thanks, and best regards,

 

-- Mark

 

_______________________________

Mark Prater

Research Analyst

WeatherPredict Consulting, Inc.

401-792-4672 (Direct)

 

[Daniel Holloway]

Hi Mark,

    A few things to consider:

    1:  The server you are accessing the data from is actually a THREDDS data server (.i.e., TDS), and sup...@unidata.ucar.edu might have some insight on how best to approach the problem.

    2:   I followed a similar path I assume as you when you say you 'downloaded a file using the OPeNDAP data access form' from that server but the options on that form do not return a NetCDF file, they'll return a DAP binary response, or ascii but not NetCDF so whatever you saved will not be compatible with the netcdf tools you're using.

    3:  That server does provide a simple 'http' access to the file, but you might not be able to subset the file remotely.   Not sure how big that file is but downloading the complete file should be compatible with the netcdf tools.

    4:  If you only needed Matlab access to the data you could use our 'loaddap' client to access the subset of data using the URL you've provided, then save the result in a .mat file or such.  

    5:  Rumor has it that soon Matlab will support DAP access through the NetCDF tools, supposedly it's in beta currently though I haven't tried it myself.

    Hope this helps, if not please ask more questions.

           Dan

    p.s.  Remember that sup...@unidata.ucar.edu can help with TDS questions as well, though I think I covered the issues.

[Daniel Holloway]

On Feb 22, 2012, at 2:05 PM, Prater, Mark wrote:

Thanks Dan!

 

I naively thought that the *.nc file was a binary netCDF file.  Actual the file has the suffix *.nc.dods  - is there a standard mechanism to read/load/unpack those files?  Or is that question best left to the support@unidata people? 

    Hi Mark,

          Not that I'm that familiar with, I expect that any DAP-enabled client could read it though typically they communicate directly with the remote server.

          There are some questions you could ask the Unidata support folks, one might be if their subset service could be deployed on that particular site to assist end-users like yourself who want to perform remote subsetting operations and return a NetCDF file, not a DAP response to the requesting user.

           There might be other options for returning a subsetted NetCDF file, other than their subset service, but I'm not aware of it.

            Unfortunately we don't have a 64-bit windows client available for loaddap, we were in the works to complete that but funding ran out though we're still working on it piecemeal.

            I'd bounce a question off of sup...@unidata.ucar.edu, requesting help with making a subsetted request to a large, TDS served dataset, and returning a NetCDF file to the end-user.  Most likely they'll say that the particular site should install their subsetting service, which for these datasets might work.   Of course you'd have to convince the NOAA site-admin to do it... easier said than done.

            Dan

 

I did download loaddap from the OPenDAP site, but am struggling to get that to work.  There is no detailed information on how to use it, and placing the installation directory in my Matlab path causes the loaddap.m file (which is nothing but an error-filled function statement followed by comments) to be executed, rather than the loaddap.mexw32 file.  I *think* this may be because the mex file is 32bit, while my matlab is 64bit (that is, maybe I require a loaddap.mexw64?).

 

The original data is very large (~2.6 GB), so I was using the data access form to subsample the data down to a size for easier analysis.

 

Anyway, thanks for your detailed response,

 

Best regards,

 

-- Mark