SciPy discussions on Python water & met data access (including IOOS SOS) client libraries

[Emilio Mayorga]

A small group got together at the recent SciPy 2013 conference [1] to
talk about Python-based data encoding and access libraries in
oceanography and hydrology, and possibilities for leveraging and
convergence; I set up a Birds of a Feather session [2] for this, but
ultimately it was just an informal get together that built on
presentations and informal discussions at the conference. It was
attended by Rich Signell (USGS), Alex Crosby (ASA), Andy Wilson &
Dharhas Pothinha (Texas Water Development Board), and Philip Elson (UK
Met Office), with Chris Barker (NOAA Emergency Response Division) and
Massimo Di Stefano (RPI/WHOI?) off to the side semi paying attention
while they hacked on irregular grid handling as part of the overlapping
"Unstructured Grid Data Model" sprint [3].

We quickly focused on client libraries. Specifically:
- ASA's Pyoos [4] and the related Paegan [5] libraries
- Texas Water Development Board's ulmo [6]
- UK Met Office's Cartopy [7] (part of their Scitools/Iris packages; see
Phil Elson's SciPy presentation [8])

I think there was a lot of synergy and enthusiasm to leverage one
another's work. We identified the following two areas as ripe for
convergence and further discussions (with my apologies if I mangled any
of these outcomes!):

1. CF conventions and Common Data Model (CDM) internal mapping.
Iris/cartopy & Paegan both have this functionality to various extents.
Iris/cartopy targeted CF 1.5, while Paegan targeted CF 1.6. One
consequence is that while Paegan was built from the ground up to support
the CDM discrete feature types (points, time series, profiles, etc),
Iris/cartopy doesn't support these yet and would need a considerable
overhaul to do so. Phil expressed a lot of interest in moving in that
direction and in leveraging Paegan to do it, while Alex and others
commented that the CF core data model in Iris seemed to be really clean
and well built, and could benefit Paegan.

2. Pyoos and ulmo have overlapping scope (data access across a range of
data providers, and an extensible architecture to add more data
handlers), including some overlap in data handlers already included
(USGS NWIS WaterML data access). But they also reflect different,
complementary philosophies: Pyoos ultimately interacts with Paegan to
map diverse data sources to the CDM feature types (although it can also
output "raw" data, in its original form), including profiles and
trajectories; while ulmo aims to be fairly light-weight and data-model
agnostic, and only introduces common convenience functionality (eg,
datetime handling, returning data as either Python dicts or pandas
DataFrames, local data caching via HDF5/pytables with subsequent
updating) and similar data request functions. Both are fairly new
libraries, though ulmo has a longer legacy via its previous incarnation
as the now deprecated pyhis. Without some communication, it's likely
that data parsers for the same data sources/formats could be developed
independently by the two groups in the future. We thought it's
worthwhile to follow up and explore the possibility to move towards some
convergence and division of labor, where the lower-level ulmo is used
directly by pyoos to access diverse data sources, and pyoos focuses on
mapping to CF/CDM and providing access via a consistent data model.

I volunteered to help advance #2. Phil and ASA/Alex (and others?) were
very interested in following up on #1. I've already started to make
minor contributions to ulmo via github, and at SciPy Alex helped me get
started installing and playing with Pyoos. Dan Ramage (SECOORA) and
myself plan to start using Pyoos soon for a new joint project.

There were many other related, terrific presentations and discussions at
SciPy, specially at the "GIS - Geospatial Data Analysis" and
"Meteorology, Climatology, Atmospheric and Oceanic Science"
mini-symposia [9]. My favorite was the idea first floated and tossed
around at the GIS mini-symposium on the 26th, that it'd be really cool
to add a geospatial data type to pandas ("geopandas"); only to be
followed by a first live demo of GeoPandas [10] 24 hours later at the
other mini-symposium, hacked by Enthought's Kelsey Jordahl! Kelsey, Alex
and others have continued to advance GeoPandas on github, with lots of
functionality from Sean Gillies' tools (shapely, descartes, etc)

Chime in if you're interested in any of this!

Massimo Di Stefano also has a nice write-up on some of these topics
[11], with lots of links.

Cheers,
-Emilio

Emilio Mayorga
NANOOS / University of Washington

PS. Can someone (Rich or Alex?) forward this message to Phil Elson?

[1] http://conference.scipy.org/scipy2013/
Videos for all presentations and mini-symposia are available at
http://pyvideo.org/category/37/scipy-2013
[2] Data access and munging tools for oceanographic and hydrological
applications (Jun 28)
http://conference.scipy.org/scipy2013/bof_detail.php?id=50
[3] http://conference.scipy.org/scipy2013/sprint_detail.php?id=28
[4] https://github.com/asascience-open/pyoos
[5] https://github.com/asascience-open/paegan
[6] https://ulmo.readthedocs.org
[7] http://scitools.org.uk/cartopy/index.html
[8] http://conference.scipy.org/scipy2013/presentation_detail.php?id=132
[9] http://conference.scipy.org/scipy2013/mini_symposia.php
[10] https://github.com/kjordahl/geopandas,
http://kjordahl.github.io/geopandas/
[11] http://tw.rpi.edu/web/event/SciPy/2013/report

[Signell, Richard]

Emilio,
Great summary. Thanks so much for doing this!

It's exciting to see this convergence on common data models and open
source tools for both scientists and developers to make use of them.
It was also great to be able to use the code sprint days at SciPy to
work together.

I'll pass this along to the British Met Office guys (and cc you).

-Rich

--
Dr. Richard P. Signell (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598

[Jeremy Cothran]

Thanks Emilio for the summary.  Interested in python tools and development and good to learn about various existing or potential usage contexts, features and various workflow combinations of related tools.

On Thu, Jul 25, 2013 at 10:56 PM, Emilio Mayorga <emilio...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "ioos_tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ioos_tech+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.