Wrking with station data

[dan]

I'm still trying to determine how to work with the data returned from a .collect() call.
This thread is a bit of a documentation process as well.
.collect() returns an OmObservation class. The station data is contained within the .feature member. For a single station query, it is a Station class.
Here is a bit of hierarchy of the Station class with the functions that deal with getting bits out(getters, not setters):

Station(inherits PointCollection):
    def get_location(self):
    def get_uid(self):
    def get_name(self):
    def get_unique_members(self):
    def get_description(self):
    def properties(self):
    def get_property(self, prop):


PointCollection(inherits Feature collection):
    def filter_by_time(self, starting, ending):
    def filter_by_variable(self, variable, elements=None):
    def __filter_depths(self, list_to_filter):

FeatureCollection:
   def get_type(self):
    def get_time_range(self):
    def get_depth_range(self):
    def lower_left(self):
    def upper_right(self):
    def get_bbox(self):
    def get_size(self):
    def get_elements(self):
   
   
Point(the elements portion of a FeatureCollection):

    def get_location(self):
    def get_time(self):
    def get_member(self, **kwargs):
    def get_member_names(self):
    def get_members(self):

[Emilio Mayorga]

Hi Dan et al,

Dan, thanks for that summary documentation. It will come in handy.

To add to it, and to help you along in understanding IOOS SOS GetObservation responses (since, unlike the rest of us on this list, you weren't involved in the development of the Reference Implementation), I'll summarize here some important concepts; I'll also mention the extent to which the existing SOS servers and demo endpoints span the full range of possible GetObservation responses in IOOS SOS Milestone 1.

First, seeing the Milestone 1 template XML responses can be pretty useful, since they represent what pyoos is targeting and what the SOS servers are supposed to deliver. All the templates are here:
http://code.google.com/p/ioostech/source/browse/#svn%2Ftrunk%2Ftemplates%2FMilestone1.0
The inline comments are pretty rich and helpful.

However, as I mentioned in another email recently, the GetObservation response is currently split into two component XML's: the enclosing OM-GetObservation.xml and various possible SWE-* responses. The merged response I created exercises most (but not all) of the capabilities available in a Milestone 1 GetObs response:
https://github.com/ioos/ioossos2kml/blob/master/SWETesting/OM-GetObservation-SWE-MultiStation-TimeSeries-Corrected.xml
Take a look at the inline comments. This is the GetObs response parsed in my Wakari notebook at
https://www.wakari.io/sharing/bundle/emayorga/pyoos1_test_3

Now, to summarize the structure and capabilities:

* A response (to a network offering) can consist of multiple stations, each made up of 1 or 2 "feature types", with each feature type in a station having 1 or more sensors, and each sensor having 1 or more observed properties. Currently only timeSeries and timeSeriesProfile feature types are implemented in the templates or in the wild; the third one, "trajectories" (or something like that), is not implemented yet.

* Different feature types are encoded as separate, parallel blocks (om:ObservationCollection/om:member/om:Observation elements). A station having sensors of two different feature types (eg, a depth profiling buoy that has weather sensors -- we have several of those in NANOOS) would then be split up into two parallel blocks, by feature type. In contrast, a station having >1 sensors of the same feature type (say, a weather sensor package and a fixed-depth water sensor package, both of which produce "timeSeries" data) would be encoded in a single om:member/om:Observation element, as shown in OM-GetObservation-SWE-MultiStation-TimeSeries-Corrected.xml. There's currently no merged template file showing a response with two feature types AND multiple stations AND multiple sensors per station. I can create one, eventually.

* ncSOS is constrained to return only one one feature type per response, having only one station; but that response can have multiple sensors. This restriction is due to a core limitation in the CF/CDM conventions. So, the most complex response you'll find from ncsos would look like this:
timeSeries > station1
    > sensor1 > air_temperature
    > sensor1 > wind_speed
    > sensor2 > sea_water_temperature
    > sensor2 > salinity
and so on ...

* 52North IOOS SOS does not have the built-in ncsos constrain. However, the stable demos are fixed to return only one kind of limited response. Specifically, each station offering has two sensors, each sensor belonging to a different feature type and having just one observed property. So, the most complex GetObs response to a network offering request looks like this:
timeSeries
  > station1 > sensor1 > air_temperature
  > station2 > sensor1 > air_temperature
timeSeriesProfile
  > station1 > sensor2 > sea_water_temperature
  > station2 > sensor2 > sea_water_temperature

At this time, we can fully flex pyoos only with hard-wired merged XML templates like mine. Between my template, existing ncsos servers, and the 52n stable demo, we do have a reasonably good set of tests, though.

Dave, Derrick et al, please chime in if I butchered anything!

Hope that helps. Cheers,

-Emilio

--
You received this message because you are subscribed to the Google Groups "pyoos" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyoos+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dave Foster]

Minor note, I also was not involved in the Reference Implementation development, and this email offers some more good perspective for me, esp with regards to the limitations of the existing implementations.  It will help make Pyoos better.

[Dan Ramage]

Emilio,

I'm sitting here looking at your notebook, comparing the IoosGetObservation you are using and the IoosSweSos object I am using.

One thing that stands out for me is the way elements is accessed. In your code, elements turns out to be a dictionary, in mine it is a list.

I do understand that you are getting multiple stations, while I am retrieving one.

I know I sound like a broken record, but I just think I am guessing about how I should use PyOOS. For example, when you're gathering up the observations in the notebook in "In [69]" you are drilling down directly to the 'value', 'units', and 'standard'. Are these keys fixed across all the different data collectors, USGS, HADS…? My concern is coding making an assumption that is wrong. My thought with the POint.get_member() function is that there are a set of known ids that correspond to value, units, etc, and that insulates us from the underlying structure.

Anyway, I think out dual pronged approach is leading to some useful questions.

DAn

[Emilio Mayorga]

Dave: I'm really glad my notes helped you, too. I assume only by virtue of you being a pyoos and ncsos developer that you were fully inducted into the SOS Reference Implementation inner sanctum. But I know it can be confusing. My bad.

Dan: my notebook is very sprawling and is hitting on multiple parallel capabilities. I'm going to try to create a new one that's much more focused and smaller, to specifically compare extraction of the time series data from my XML template, an ncsos endpoint, and the Axiom 52N stable demo endpoint (single-station vs multi-station requests). Hopefully by tomorrow.

As for this:

Are these keys fixed across all the different data collectors, USGS, HADS…?

That's for Dave (and Kyle?) to answer. My guess is that the collectors behave differently, but I understand what you're getting: if pyoos converts all harvested data into Paegan objects, shouldn't those objects be accessible in very similar ways independent of the data source? I assume that's where you're coming from.

Cheers,

-Emilio

[Dan Ramage]

Emilio,

You are correct about my concern on the collectors. I was hoping PyOOS would allow a single interface to multiple datasources in a harmonized manner. 

Dan

[Dave Foster]

Dan,

This is absolutely the goal - a normalized form of access.  I am not fully aware of the other collectors behavior at this point as I've been so focused on the IoosSosSwe one, but having other formats is rather irrational for this library.

[Kyle Wilcox]

Just joined this group yesterday...

Dan - You are correct in your hopes for Pyoos.  The general goal of Pyoos is to provide a single interface to collect observational data.

Pick sources, apply filters, and get back a normalized result.

There should be tests for all supported sources.  That is the best place to look.  Of something does not seem to be working, make a new thread here with the specific problem and I can take a look.  Or better yet, dive into the code and start improving it and the documentation!

More will get accomplished of we work on this together.  Dave and I will never be able to keep up at current demand.

[Emilio Mayorga]

Here are some hopefully encouraging results.

https://www.wakari.io/sharing/bundle/emayorga/pyoos_tsaccess_diffsources

(note: I didn't bundle the environment with this notebook)

In this notebook I access Rich's WHOI ncSOS end point and the Axiom 52North IOOS SOS Stable Demo end point, using the same approach. The main conceptual difference between the two is that ncSOS returns a single station (as requested, and as constrained by ncSOS) while the 52N example is based on a network offering request returning multiple stations.

I've borrowed from Dave's example and previous iterations from Dan and me.

Later I'll rework my example (https://www.wakari.io/sharing/bundle/emayorga/pyoos1_test_3) that uses the multi-station multi-sensor timeseries Milestone 1 XML template, but using the same time series parsing approach used in the other notebook with ncSOS and 52N. Dave: I'll see if I can test your new pyoos branch that implements multi-sensor parsing.

-Emilio

[Rich Signell]

Emilio,
These are indeed encouraging results!

One thing: I see that you (and Dave before you) requested data by
netcdf variable name ('u_1205')

collector.filter(variables=['u_1205'])

instead of by standard_name ('eastward_sea_water_velocity').


collector.filter(variables=['eastward_sea_water_velocity'])

But I just tried it and it seems that the latter works also! This is
great, since netcdf variable names are arbitrary in CF conventions.


Thanks,
Rich

>>>> HADS...?

>>>
>>>
>>> That's for Dave (and Kyle?) to answer. My guess is that the collectors
>>> behave differently, but I understand what you're getting: if pyoos converts
>>> all harvested data into Paegan objects, shouldn't those objects be
>>> accessible in very similar ways independent of the data source? I assume
>>> that's where you're coming from.
>>>
>>> Cheers,
>>> -Emilio
>>>
>>>
>>> On Mon, Feb 3, 2014 at 10:10 AM, Dan Ramage <d...@inlet.geol.sc.edu>
>>> wrote:
>>>>
>>>> Emilio,
>>>>
>>>> I'm sitting here looking at your notebook, comparing the
>>>> IoosGetObservation you are using and the IoosSweSos object I am using.
>>>> One thing that stands out for me is the way elements is accessed. In
>>>> your code, elements turns out to be a dictionary, in mine it is a list.
>>>> I do understand that you are getting multiple stations, while I am
>>>> retrieving one.
>>>>
>>>> I know I sound like a broken record, but I just think I am guessing
>>>> about how I should use PyOOS. For example, when you're gathering up the
>>>> observations in the notebook in "In [69]" you are drilling down directly to
>>>> the 'value', 'units', and 'standard'. Are these keys fixed across all the

>>>> different data collectors, USGS, HADS...? My concern is coding making an

Rich Signell
81 Queen St
Falmouth, MA 02540

[Dan Ramage]

Kyle,

Can you point me to one of the tests that deal with handling the returned data? I see various .collect() calls but I am not seeing what I do if I want to loop through the data elements and get their values for instance.

Dan

[Emilio Mayorga]

Thanks for testing that, Rich.

Now, regarding this:

collector.filter(variables=['eastward_sea_water_velocity'])
But I just tried it and it seems that the latter works also!

I hate to be a party pooper, but from what I can tell this should *not* work! The SOS offering does not advertise the CF standard names; it only advertises the variable names, as ioos sensor urn's. Here's what the GetCapabilities doc has for the observed properties in that offering:
<sos:observedProperty xlink:href="urn:ioos:sensor:gov.usgs:1211-A1H:u_1205"/>
<sos:observedProperty xlink:href="urn:ioos:sensor:gov.usgs:1211-A1H:v_1206"/>
<sos:observedProperty xlink:href="urn:ioos:sensor:gov.usgs:1211-A1H:CD_310"/>
<sos:observedProperty xlink:href="urn:ioos:sensor:gov.usgs:1211-A1H:CS_300"/>
<sos:observedProperty xlink:href="urn:ioos:sensor:gov.usgs:1211-A1H:upr_4001"/>

ncSOS (or pyoos? I doubt it) may be doing some handy magic behind the scenes accepting CF names and matching them to these variable names (or ioos sensor urn's), but that magic is unadvertised, so it shouldn't be happening as magic. According to the IOOS SOS specs, CF standard name URL's should be advertised in the sos:observedProperty elements, not sensor urns.

Hugs and kisses,

-Emilio

[Dave Foster]

I think we have this captured in ncSOS's issue tracker as https://github.com/asascience-open/ncSOS/issues/107

[Emilio Mayorga]

Thanks, Dave.

Actually, this issue is related, but not identical. Issue 107 is about ncSOS accepting a shorthand of the observedproperty -- specifically, the last string token. While that's probably not strictly legal (I think, but I can't remember), at least it's using a logical component of the advertised observedproperty string; I can see the convenience, and kind of like it myself (though it's probably not compliant behavior).

What I'm pointing out is off the left field, I think. A user or client software could not possibly make a linkage between "urn:ioos:sensor:gov.usgs:1211-A1H:u_1205" and "eastward_sea_water_velocity" without external information not present in the GetCapabilities document. I'm guessing the matchup is being done via a parallel DescribeSensor request; if not, it's deeper magic.

-Emilio

[Richard Signell]

Uh...

> Thanks for testing that, Rich.
>

> Hugs and kisses,
> -Emilio

... I just worked up to hugging in my men's group. Not sure I'm ready
for kisses yet. ;-)

[Emilio Mayorga]

And here's my final entry in the troika started by
https://www.wakari.io/sharing/bundle/emayorga/pyoos_tsaccess_diffsources

That notebook illustrates access to ncSOS and 52N IOOS SOS end points (first as the collection, then parsing the time series data) using the same approach.

The new notebook
https://www.wakari.io/sharing/bundle/emayorga/pyoos_tsaccess_xmltemplates

does the same but using the multi-station, multi-sensor timeseries Milestone 1 XML template, and necessarily starting with an IoosGetObservation class rather than with IoosSweSos. But after that, everything is identical.

I haven't had a chance to test Dave's github branch that addresses parsing of multi-sensor content within a station. Hopefully I'll try it next week.

Cheers,

-Emilio

[dan]

For a Point in elements, calling get_members() returns a list of objects. Calling get_member_names() returns an empty list.
Calling get_member() errors out, the function is trying to use the values returned from get_member_names() to access self.members.

Is this a bug?

I've added some class diagrams to the document page here: https://github.com/asascience-open/pyoos/wiki/IoosSweSos-first-pass-documentation
PyCharm has some nice features!