Any plans to extend GridType to CF Convention?

[Richard Signell]

Pydap folks,

I may have this wrong, but it seems that GridType is limited to COARDS-compliant grids (e.g. 1D lat/lon/z arrays).   If this is true, are are there plans to extend GridType (or perhaps have another class) for CF-Compliant grids?

I ask this because many coastal and estuarine grids are curvilinear structured grids with stretched coordinate systems.  Here's a typical small example:
http://schumann.coastal.udel.edu:8080/thredds/dodsC/cb/eutr_0204.html

 In these grids, the lon, lat coordinates are 2D arrays, and the z values are computed from formulae expressed in a standard way via the CF conventions.

As far as I know, the only library that currently represents the CF grid class is the NetCDF-Java library, but it would be great to have this capability in python.

If we had this capability in pydap, a user could, for example, request a 3D spatial subset of temperature at a specific time step, and get back the 2D lon,lat and the 3D z array that went with it.

And that would be cool.

Thanks,
Rich

[Roberto De Almeida]

Hi, Rich!

On Tue, Feb 5, 2013 at 5:24 AM, Richard Signell <rsig...@gmail.com> wrote:

I may have this wrong, but it seems that GridType is limited to COARDS-compliant grids (e.g. 1D lat/lon/z arrays).   If this is true, are are there plans to extend GridType (or perhaps have another class) for CF-Compliant grids?

I ask this because many coastal and estuarine grids are curvilinear structured grids with stretched coordinate systems.  Here's a typical small example:
http://schumann.coastal.udel.edu:8080/thredds/dodsC/cb/eutr_0204.html

In these grids, the lon, lat coordinates are 2D arrays, and the z values are computed from formulae expressed in a standard way via the CF conventions.

You're right. As far as I know, curvilinear grids are usually represented in Opendap/NetCDF by defining the data in terms of x and y, and then also defining lat and lon in terms of (x, y), eg:

Float32 sst[x = 200][y = 100];

Float32 lat[x = 200][y = 100];

Float32 lon[x = 200][y = 100];

Where x and y are just indexes. In the example you linked we have:

Float32 LAT[NY = 124][NX = 84];

Float32 LON[NY = 124][NX = 84];

 

As far as I know, the only library that currently represents the CF grid class is the NetCDF-Java library, but it would be great to have this capability in python.

If we had this capability in pydap, a user could, for example, request a 3D spatial subset of temperature at a specific time step, and get back the 2D lon,lat and the 3D z array that went with it.

Chris Barker mentioned a few days ago how they are handling selection of unstructured grids; there's a presentation here:

  http://www.opendap.org/node/265 

Basically they're using Opendap server-side functions to specify a bounding box:

http://…/model.nc?ugrid(SST,45,-80, 45,-60, 20,-60, 20,-80)

In this example we're defining a square bounding box where we want the data, but any number of points can be specified. Is this what you had in mind? It would be fairly simple to implement in Pydap, I think. 

Cheers,

--Rob

 

And that would be cool.

Thanks,
Rich

--
You received this message because you are subscribed to the Google Groups "pydap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydap+un...@googlegroups.com.
To post to this group, send email to py...@googlegroups.com.
Visit this group at http://groups.google.com/group/pydap?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Roberto De Almeida, PhD

http://dealmeida.net/
:wq

[Chris Barker - NOAA Federal]

On Tue, Feb 5, 2013 at 2:04 PM, Roberto De Almeida
<rob...@dealmeida.net> wrote:
>> As far as I know, the only library that currently represents the CF grid
>> class is the NetCDF-Java library, but it would be great to have this
>> capability in python.

What can NetCDF-Java do with the grid class? What would you want
OpenDAP to do with it?

>> If we had this capability in pydap, a user could, for example, request a
>> 3D spatial subset of temperature at a specific time step, and get back the
>> 2D lon,lat and the 3D z array that went with it.

so this isn't subsetting according to the grid (i.e. subsetting time,
not in lat-lon space), that should be starighforward. Is it a matter
of preserving the grid info?

> Chris Barker mentioned a few days ago how they are handling selection of
> unstructured grids; there's a presentation here:
>
> http://www.opendap.org/node/265
>
> Basically they're using Opendap server-side functions to specify a bounding
> box:
>
> http://…/model.nc?ugrid(SST,45,-80, 45,-60, 20,-60, 20,-80)
>
>
> In this example we're defining a square bounding box where we want the data,
> but any number of points can be specified. Is this what you had in mind? It
> would be fairly simple to implement in Pydap, I think.

This brings up another "can of worms" -- with a curvilinear (but
structured) grid, a subset is not necessarily any longer a logically
rectangular grid, even if the you are subsetting to a rectangle in
lat-lon space. So what would one want back?

- a UGRID ? -- that would be cool, but has hardly any )any at all?)
tools know how to deal with UGRIDS now, probably not the most useful
option.

- The logically-rectangular unstructured grid that holds all of the
requested region -- io.e. generally larger than it needs to be.

- The above, but with the parts of the grid that are outside the region masked.

There are also interesting issues with multiple grids in one file --
like the ROMS rho and psi grids.

But in any case, it would be nice to supply the same API for both
unstructured grids and curvilinear grids.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Roberto De Almeida]

On Tue, Feb 5, 2013 at 2:59 PM, Chris Barker - NOAA Federal <chris....@noaa.gov> wrote:

This brings up another "can of worms" -- with a curvilinear (but

structured) grid, a subset is not necessarily any longer a logically
rectangular grid, even if the you are subsetting to a rectangle in
lat-lon space. So what would one want back?

 - a UGRID ? -- that would be cool, but has hardly any )any at all?)
tools know how to deal with UGRIDS now, probably not the most useful
option.

 - The logically-rectangular unstructured grid that holds all of the
requested region -- io.e. generally larger than it needs to be.

- The above, but with the parts of the grid that are outside the region masked.

You could also return the data as a logical 1D vector:

Float32 SST[time = 12][depth = 50][space = 100];

Float32 LAT[space = 100];

Float32 LON[space = 100];

Float32 time[time = 12];

Float32 depth[depth = 50];

This is how Numpy does, so it would be the easier option to implement in Pydap. But I agree that there should be a single API for all Opendap servers, and both for unstructured and curvilinear grids.

Or you could interpolate the data into a new grid (though I wouldn't do that). But given the name of the function, it should return an ugrid and we should build tools to deal with them, if they're missing.

--Rob

 

There are also interesting issues with multiple grids in one file --
like the ROMS rho and psi grids.

But in any case, it would be nice to supply the same API for both
unstructured grids and curvilinear grids.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

--
You received this message because you are subscribed to the Google Groups "pydap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydap+un...@googlegroups.com.
To post to this group, send email to py...@googlegroups.com.
Visit this group at http://groups.google.com/group/pydap?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Rich Signell]

Rob,

Really the main functionality I was looking for was just to have
lon/lat/depth/time values returned when a subset of data from a
curvilinear structured grid is requested.

To do this one needs to follow the CF conventions to determine the
coordinate variables (since they aren't determined by 1D variables
with the same name as dimensions)
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/cf-conventions.html#coordinate-types

I think the rules are basically:
1) parse the "coordinates" attribute, which specifies the coordinate variables
2) for each then parsing the attributes of the coordinate variables
to determine which attributes they are.
- check to see if there is a coordinate with the attribute
"positive" and CF compliant units of height or pressure. If "yes",
then it's a Z coordinate.
- check to see if the are CF compliant units of longitude
"degrees_east, degree_east, ..." if so, then it's a LON coordinate.
- check to see if the are CF compliant units of latitude
"degrees_east, degree_east, ..." if so, then it's a LATcoordinate.
- check to see if the are CF compliant units of time
"minutes/hours/days since, ..." if so, then it's a TIME coordinate.


The method for subsetting curvilinear grids based on a lon/lat
bounding box in NetCDF/Java is to find the index ranges that will
completely cover the bounding box (so the request might bring back a
larger portion of the grid that extends outside the grid, but
completely covering it. I believe this is nearly always the behavior
wanted.

Thanks,
Rich

Dr. Richard P. Signell (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598

[Chris Barker - NOAA Federal]

On Tue, Feb 5, 2013 at 7:37 PM, Rich Signell <rsig...@usgs.gov> wrote:

> Really the main functionality I was looking for was just to have
> lon/lat/depth/time values returned when a subset of data from a
> curvilinear structured grid is requested.

yup -- that would be good.

> To do this one needs to follow the CF conventions to determine the
> coordinate variables (since they aren't determined by 1D variables
> with the same name as dimensions)
> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/cf-conventions.html#coordinate-types

hmm -- this reminds me that I"ve been thinking for ages that a Python
package that provided an API for working with CF-compliant data would
be great.

A port of netcdf-Java ( does it have an API we'd want to use ) ?

Is the "Python implementation of the CF data model" a good basis?

Something else?

I'd been thinking it should be an add-on for the NetCDF4 python
package -- Rib do you use that in pyDAP?

Anyway, something that could be used in pyDAP and other projects would be great.

> The method for subsetting curvilinear grids based on a lon/lat
> bounding box in NetCDF/Java is to find the index ranges that will
> completely cover the bounding box (so the request might bring back a
> larger portion of the grid that extends outside the grid, but
> completely covering it. I believe this is nearly always the behavior
> wanted.

Given the tools for working with the results, yes -- that's what we do
with GOODS, too.

[Sean Arms]

Greetings all!

On 2/6/13 10:54 AM, Chris Barker - NOAA Federal wrote:
> On Tue, Feb 5, 2013 at 7:37 PM, Rich Signell <rsig...@usgs.gov> wrote:
>
>> Really the main functionality I was looking for was just to have
>> lon/lat/depth/time values returned when a subset of data from a
>> curvilinear structured grid is requested.
> yup -- that would be good.
>
>> To do this one needs to follow the CF conventions to determine the
>> coordinate variables (since they aren't determined by 1D variables
>> with the same name as dimensions)
>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/cf-conventions.html#coordinate-types
> hmm -- this reminds me that I"ve been thinking for ages that a Python
> package that provided an API for working with CF-compliant data would
> be great.
>
> A port of netcdf-Java ( does it have an API we'd want to use ) ?

We are talking about doing this exact thing here at Unidata. We are not
sure the best way to go about doing so, but we'd definitely put it out
on github, just like netCDF-Java! However, as Rich knows well, we need
the feedback from the community that indicates we should put our effort
into this area of development. While I see the need, we need to hear it
from you all. That said, Rich is on the Users Committee, and we do have
a committee meeting coming up soon, so if this is something you'd like
us to do, let him know and he will bring it to the table (note that
there are people here who would like to work on this).

Cheers!

Sean

[Gallagher James]

On Feb 7, 2013, at 8:08 AM, Sean Arms wrote:

> Greetings all!
>
> On 2/6/13 10:54 AM, Chris Barker - NOAA Federal wrote:
>> On Tue, Feb 5, 2013 at 7:37 PM, Rich Signell <rsig...@usgs.gov> wrote:
>>
>>> Really the main functionality I was looking for was just to have
>>> lon/lat/depth/time values returned when a subset of data from a
>>> curvilinear structured grid is requested.
>> yup -- that would be good.
>>
>>> To do this one needs to follow the CF conventions to determine the
>>> coordinate variables (since they aren't determined by 1D variables
>>> with the same name as dimensions)
>>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/cf-conventions.html#coordinate-types
>> hmm -- this reminds me that I"ve been thinking for ages that a Python
>> package that provided an API for working with CF-compliant data would
>> be great.
>>
>> A port of netcdf-Java ( does it have an API we'd want to use ) ?
> We are talking about doing this exact thing here at Unidata. We are not sure the best way to go about doing so, but we'd definitely put it out on github, just like netCDF-Java! However, as Rich knows well, we need the feedback from the community that indicates we should put our effort into this area of development. While I see the need, we need to hear it from you all. That said, Rich is on the Users Committee, and we do have a committee meeting coming up soon, so if this is something you'd like us to do, let him know and he will bring it to the table (note that there are people here who would like to work on this).

Along the same lines, we are also talking about these issues in the OPULS/DAP4 project. In that project we are looking at making sure the DAP4 data model supports the higher level models like CDM with very high (or perfect) fidelity. We don't envision DAP4 moving away from it's domain-neutral stance, but expanding the Grid type to encompass more kinds of grids (which I think are equivalent to OGC coverage model) is a goal of DAP4/OPULS.

We are also interested in a set of extra capabilities (that we're calling 'extensions') that will be documented as part of OPULS and those will contain geospatial and temporal subsetting. So, for that reason, and sorry for being so long-winded about it, we're interested in this as well. And Rich is on our advisory board… ;-)

James

>
> Cheers!
>
> Sean
>>
>> Is the "Python implementation of the CF data model" a good basis?
>>
>> Something else?
>>
>> I'd been thinking it should be an add-on for the NetCDF4 python
>> package -- Rib do you use that in pyDAP?
>>
>> Anyway, something that could be used in pyDAP and other projects would be great.
>>
>>> The method for subsetting curvilinear grids based on a lon/lat
>>> bounding box in NetCDF/Java is to find the index ranges that will
>>> completely cover the bounding box (so the request might bring back a
>>> larger portion of the grid that extends outside the grid, but
>>> completely covering it. I believe this is nearly always the behavior
>>> wanted.
>> Given the tools for working with the results, yes -- that's what we do
>> with GOODS, too.
>>
>> -Chris
>>
>>
>

> --
> You received this message because you are subscribed to the Google Groups "pydap" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pydap+un...@googlegroups.com.
> To post to this group, send email to py...@googlegroups.com.
> Visit this group at http://groups.google.com/group/pydap?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--

James Gallagher
jgallagher at opendap.org
406.723.8663