Re: PyOOS Issue Missing Multiple Sensor DataRecords

[Emilio Mayorga]

Dan: Thanks for following up on that important point! I hadn't had a chance to look at the pyoos code to track the problem.

Dave: My notebook is here:
https://www.wakari.io/sharing/bundle/emayorga/pyoos1_test_3

See block "In [16]" and ones around it.

The source XML is basically the Milestone 1 XML template (SWE-MultiStation-TimeSeries-Corrected merged with OM-GetObservation); see my email yesterday to ioostech_dev (I assume you're subscribed to ioostech_dev?). The URL is in the notebook, and it's this:
https://raw2.github.com/ioos/ioossos2kml/master/SWETesting/OM-GetObservation-SWE-MultiStation-TimeSeries-Corrected.xml

I'm cc'ing the pyoos list, to give this issue greater exposure.

Thanks!

-Emilio

On Wed, Jan 22, 2014 at 10:08 AM, Dave Foster <dave....@gmail.com> wrote:

Can you link the notebook here please?  Or more importantly, the source XML being processed with the multiple sensor data records.  Just to keep everything together.

On Wed, Jan 22, 2014 at 1:04 PM, Dan Ramage <d...@inlet.geol.sc.edu> wrote:

In Emilio's latest notebook, he made note that for the wmo_41001 station, the second set of sensors was not getting processed. I think I may have tracked down why that is, although I am not too sure so I don't want to create a bogus bug report in GitHub.

In timeseries.py in _init_ line 113 is: stations[s.uid] = s

Stations is a dict and when there are multiple sensor blocks the loop there keeps using the same key to assign the data. I am not 100% sure this is the issue, but I'd been stepping through the code and this stood out.

Dan

[mayorga]

I'm trying to follow up on this topic. From what I can tell, I replied to the pyoos list (so here are the previous exchanges), but Dan's last reply only went to Dave and I. I've pasted Dan's reply below.

I haven't tried fully track what's going on, but the issues is definitely in parsers/ioos/one/timeseries.py

It seems to me that feature.elements need to be abstracted out further into "sensors", which can have their own separate vertical local frame. Or at the very least each variable ("element member") needs to retain its linkage to the sensor identifier. Right now that linkage is lost, and as Dan said other sensors are ignored and therefore their associated variables and data are not returned by TimeSeries().feature.

Dave, can you confirm that this assessment / problem is generally correct? If so, I can take a shot at fixing it later this week. This would be a pretty important missing functionality (bug or limitation, depending on your perspective).

Thanks,
-Emilio

[FROM DAN, 1/22]
> Here is the time series xml file: https://github.com/ioos/ioossos2kml/blob/master/SWETesting/OM-GetObservation-SWE-MultiStation-TimeSeries-Corrected.xml

> Emilio's notebook is here: https://www.wakari.io/sharing/bundle/emayorga/pyoos1_test_3

> Looking at the code again, I might be barking up the wrong tree since stations is a local variable.

[Dave Foster]

Seems you have indeed captured the problem (which Dan filed as https://github.com/asascience-open/pyoos/issues/20) - def a limitation in the parsing.  I've made a WIP fix approach that I'm pushing to a branch - my solution is currently to "merge" members of Points within the elements collection when those Points are equal in location/time.

This does not address the vertical local frame thing you brought up though - I am certainly not beholden to keeping elements the way it is (I didn't design this) - maybe the breakdown could be another attribute, 'sensors', at the same level as 'elements', which has the heirarchy, and then keep 'elements' with the merge approach, or maybe as an iterator.  I like the merge output so far in playing with it but def needs to be improved from a performance perspective, it's a very naive approach.

https://github.com/asascience-open/pyoos/commit/b52d71952b23624f2005473ef3eb4a2db7b324bf