help developing or running one successful pyoos IoosSweSos test against a live endpoint

[mayorga]

Hi Dave (and Kyle?),

Dan Ramage and I have been banging our heads into walls trying to get successful tests of an IoosSweSos collector, ideally against a real IOOS SOS Milestone 1 end point. Our test endpoint is the Axiom 52North SOS stable:
http://ioossos.axiomalaska.com/52n-sos-ioos-stable/sos/kvp

You can see some of our tests, dead-ends and errors on this IPytho notebook, from Wakari:
https://www.wakari.io/sharing/bundle/emayorga/pyoos_swesos_axiom52north_test_2

The collector itself seems fine. I can probe all the information coming from GetCapabilities. But issuing a collect or raw method on a collector (after setting some filters) always leads to errors. You can some of these errors on the notebook at In [35], In [37], In [38].

To make sure the SOS end point was working fine, I ran a get_observation request on the same offering from the same endpoint, using only OWSLib SOS machinery. You can see results here:
https://www.wakari.io/sharing/bundle/emayorga/AOOS_SOS_owslib_AlaskaNDBCBuoy_StableDemo

Everything looks fine with OWSLib SOS alone.

I tried looking at pyoos tests from the github repo, to see if they'd help. Specifically, this tests seems very useful:
 tests / parsers / test_ioos_get_observation.py

But the tests as installed on wakari directly from the gihub repo are missing the "parsers" and "resources" directories and files. I built this wakari / anaconda environment around Dec 9 using pip install from the github repos for OWSLib, paegan & pyoos:
pip install git+https://github.com/geopython/OWSLib.git
pip install git+https://github.com/asascience-open/paegan.git
pip install git+https://github.com/asascience-open/pyoos.git

To upgrade everything in case the tests stuff was new, I ran just now the same pip installs but with the "-U" upgrade flag. Now paegan and pyoos failed. (I later confirmed that these test files have been around for a while).

So: can you help complete one successful, end-to-end test against the Axiom SOS stable end point?

Thanks,
-Emilio

[mayorga]

Some progress, though not on pyoos proper.

See this notebook that can run on our pyoos Wakari environment (though I didn't bundle the environment with it):
https://www.wakari.io/sharing/bundle/emayorga/AOOS_SOS_owslib_AlaskaNDBCBuoy_StableDemo2

I've developed a couple of convenience functions to parse an XML string response using LXML objectify, and stripping out all namespace prefixes from XML element tags to make object traversing simpler. Doing this could in principle lead to tag/object name ambiguities and collision, but in practice it's probably fine, at least for this example. lxml objectify after namespace simplification makes object navigation very simple.

So, in this example, I use OWSLib SOS to issue a GetObservation request and get back the IOOS SOS M1 response (as a string). Then I explore the response using object traversing, then pandas and matplotlib plotting at the end. In essence, this is doing a lot of what pyoos is supposed to do, with a bunch of hard-wiring for simplicity.

As you can see, the AOOS/Axiom GetObservation response is perfectly fine (except for at least one small glitch).

I'm done with this for the week. Cheers,
-Emilio

[mayorga]

Hi Dave,

Hope you enjoyed the holidays.

Will you be able to help Dan and I with this, soon? We're stuck until we can get some help with the IoosSweSos collector and parser.

Thanks. Cheers,
-Emilio

[Dave Foster]

I'll be able to look at this tomorrow.  Thanks, and sorry for the delay!

[Dave Foster]

I saw that, thanks.  That will help narrow down the issue.

On Mon, Jan 6, 2014 at 3:23 PM, Dan Ramage <d...@inlet.geol.sc.edu> wrote:

Dave,

I've filed an issue report in the GitHub repository that hopefully helps with at least part of the issue.

Dan

[Emilio Mayorga]

Hi Dave,

Have you had a chance to look into the issues with the IoosSweSos collector/parser?

Thanks. Cheers,

-Emilio

--
You received this message because you are subscribed to the Google Groups "pyoos" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyoos+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dave Foster]

Hi Emilio,

Been looking at your wakari notebooks now.  Plenty of rough edges when using Pyoos, and one obvious bug (https://www.wakari.io/sharing/bundle/emayorga/pyoos_swesos_axiom52north_test_2, input line 38) that I'll address.

Was not able to reproduce Dan's bug (asked for more information there).

Will see what I can do to help streamline this for you!

[dan]

For convenient reference, here is a link to the Git issue I entered: https://github.com/asascience-open/pyoos/issues/19

[dan]

As Dave pointed out, the error is actually in the OWSLib code, not the PyOOS.

[Kyle Wilcox]

ah, ha!  I've figured out how to join a Google Group.  Points for me.

I'm a little confused about what you are after... do you just want the raw SWE responses from an SOS request so you can parse it yourself?

[Dave Foster]

Kyle and I have talked outside of this thread and have both found some rough points to your Pyoos notebook - not to speak for him but I think he understands what you are after.

I’ll be updating pyoos codebase soon with some fixes I have.

--

[Emilio Mayorga]

Thanks, Dave and Kyle.

Ultimately, we want to be able to use the pyoos SWE SOS parser. I tried out a raw SWE response b/c Dan and I were not succeeding at achieving a collect() action on an IoosSweSos collector after defining some filters for the request. See input boxes 25-38 in this notebook:

https://www.wakari.io/sharing/bundle/emayorga/pyoos_swesos_axiom52north_test_2

Having access to the collector's "raw" response will also be very helpful, though.

Cheers,

-Emilio

[Dave Foster]

Hi Emilio/Dan,

Some progress.  Was able to adapt your notebook and fix enough things in pyoos to get it working/streamlined.  I also wanted to hit an ncSOS server (we put experimental builds on this server and should be considered not always up).  Here's a notebook demonstrating simple use:

http://nbviewer.ipython.org/gist/daf/683fadd14b409e5f00ad

Will push some changes to pyoos soon for you folks to try out.

dave

[Dave Foster]

Just pushed some commits to master.  In your wakari env, update pyoos from git and give things a shot!

thanks,

dave

[Emilio Mayorga]

Thanks so much, Dave!!

Yesterday I tried re-building my "IOOS client" Wakari environment with the new pyoos on github master. Unfortunately I had to go through it twice and ran into problems that may be specific to IPython notebooks on Wakari. I also ran into some curious new OWSLib requirements that I think I overcame, but I need to look more closely.

The good news is that I was able to fully reproduce the example you shared on an IPython notebook, involving the Axiom 52North SOS end  point. So that's wonderful! I only did on a bare IPython shell session, not on a notebook. But that shouldn't be a pyoos issue, so don't worry about it.

I'll keep probing today and tomorrow until I have a clean Wakari environment and sample IPython notebook on Wakari (basically, yours, at least as a starting point) that I can share. Then, Dan and I will be able to push forward in exactly the same environment.

Thanks again. Stay tuned,

-Emilio

[Dan Ramage]

I pulled the latest GitHub and am crashing and burning trying to get it installed.

Build was ok, but install is problematic. I seem to need a Fortran compiler now to get SciPy installed. 

I'm running on OS X Lion.

Dan

From: Emilio Mayorga <emilio...@gmail.com>
Date: Tue, 14 Jan 2014 12:35:57 -0800
To: Dave Foster <dave....@gmail.com>
Cc: <py...@googlegroups.com>, Kyle Wilcox <ky...@axiomalaska.com>, Dan Ramage <d...@inlet.geol.sc.edu>, Derrick Snowden <Derrick...@noaa.gov>
Subject: Re: help developing or running one successful pyoos IoosSweSos test against a live endpoint

[Dave Foster]

Haven’t you always required that?

> brew list | grep fort

gfortran

[Dan Ramage]

I managed to build a previous pyoos without that, at least I think I did.

Dan

[Rich Signell]

Dan,
Yipes! This is why Wakari is so useful.

Are you using Enthought Canopy or Continuum Anaconda for your
scientific python environment? Both are free to you (Enthought
because you are .edu, Anaconda because it's free to anyone)

On Tue, Jan 14, 2014 at 3:39 PM, Dan Ramage <d...@inlet.geol.sc.edu> wrote:

Rich Signell
81 Queen St
Falmouth, MA 02540

[Dan Ramage]

Since Emilio is tackling the Wakari setup, I thought I would do local. I'm
gearing up more for what we would do in a production environment.

Dan

[Emilio Mayorga]

FYI, I'm still figuring out issues with my Wakari environment with pyoos. It's bizarre, but my new environment basically freezes up IPython notebooks even before I can do anything at all (not just pyoos stuff). But I don't run into problems on a plain IPython shell.

Must be some obscure interaction between Wakari weirdness and new code weirdness/dependencies.

I'll try to build a bare-bones pyoos env and see what happens. Right now my env has the kitchen sink.

-Emilio

[Rich Signell]

Emilio,
Did you pip install something that upgraded Ipython?
If so, you probably screwed up your notebook environment on Wakari.
Try
pip uninstall ipython
conda install ipython
to recover

[Rich Signell]

BTW, I got the latest pyoos working in Wakari, but then I couldn't
share my environment because Continuum disabled that feature
yesterday. I screamed bloody murder and they said they would
re-enable by the end of the week. Apparently people were clicking
"include my environment" indiscrimimantly and then wondering why
things took so long. I told Continuum this was *essential* for us,
and that it indicated a UI problem, so just change to "advanced
features" or something so it takes two clicks instead of one.

[Emilio Mayorga]

Hey Rich,

Thanks for the tip. I'm not aware of any of my pip installs upgrading IPython. I install IPython in the initial conda create statement.

Still, I went ahead and tried your suggestion, then logged out and back in for good measure. It made no difference.

Anyway, this thread is getting off topic from pyoos. Can you send me offline the statements you used to create your env with the latest pyoos (pip install git ..., I assume)?

Otherwise, I'll try building a lean, pyoos-focused env, and watch for IPython overwrites.

Thanks for complaining to Continuum about removing ability to bundle and share envs! Good catch, and good suggestion to them.

Cheers,

-Emilio

[Rich Signell]

https://github.com/ioos/system-test/blob/master/wakari/pyoos.md

[Emilio Mayorga]

Thanks, Rich! I was able to install a pyoos Wakari env per your instructions. Then I was able to run Dave's test for the Axiom SOS stable demo endpoint, in a IPython notebook, w/o any hiccups.

BTW, kind of circular/twisted that you first built a pyoos-amenable conda env, then exported the package list to a file, then used that file to rebuild a pyoos environment with the latest version of pyoos ... But it worked. And no need to log-out then log back in any more!? Strange (but great) also that I was getting new warnings and issues when explicitly building OWSLib again, and now I had no issues when I let paegan/pyoos install OWSLib as a dependency.

BTW, your use of git clone, then pip install, to install paegan and pyoos seemed excessive. So I tried my luck with the usual pip install git I've been using with paegan and pyoos:

pip install git+https://github.com/asascience-open/paegan.git
pip install git+https://github.com/asascience-open/pyoos.git

Life is good for now. I'll try to make some progress with parsing a GetObservation ("collect") pyoos request by Friday, then report back.

Cheers,

-Emilio