New ERDDAP python library

[fav...@gmail.com]

Hello all,

I wanted to share in this forum a work I developed for my personal use for projects in academic institutions in order to create analysis and visualization workflows using python and remote data from ERDDAP servers.

It's a python library to interface with the ERDDAP server, tabledap, and griddap datasets thru the restful API.  I have included demonstration notebooks on the usage of this library in the following repository:

https://github.com/hmedrano/erddap-python

I would really appreciate your feedback and be glad if somebody finds it useful for their projects.

Cheers,

-Favio

[Roy Mendelssohn - NOAA Federal]

I just did a quick scan and it looks very interesting and useful. In the R world we have found packages like 'rerddap' are very popular because it just makes it so much easier to get the data within R, and for many users that is what they want - they just want the data. This seems to fulfill a similar function. Nice work!

If you don't mind, I am going to forward this to the IOOS-tech mail list.

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/87d47dd0-5ad0-4e4d-a485-a1e7fef06772n%40googlegroups.com.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Men...@noaa.gov www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

[fav...@gmail.com]

Thanks for the quick response Roy, please feel free to share it,

-Favio

[Rich Signell]

Favio,

For those of us Python users who have been using ERDDAPY ( https://github.com/ioos/erddapy ), what would you say the major differences in functionality or design are for erddap-python?

I know that it's often healthy to have different implementations, so I'm genuinely interested in your response!

Thanks,

-Rich

[Roy Mendelssohn - NOAA Federal]

I haven't checked recently but originally ERDDAPY could only do tabledap. The looks like it can do tabledap and griddap.

-Roy

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/75bfa1ec-4de9-49a4-be61-fe69c5889abfn%40googlegroups.com.

[Filipe Pires Alvarenga Fernandes]

It can do griddap the same way erddap-python does, passing the opendap URL to xarray. That is what we call partial support b/c we don't take advantage of the server side features of erddap. For more info see https://github.com/ioos/erddapy/issues/32

-F

To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/65A9C3C0-38A0-4992-8A36-51308D8EA805%40noaa.gov.

[Bob Simons]

Favio, thanks very much for doing this and making it available for others.  It looks well designed. I suspect a lot of people will find it useful.

[fav...@gmail.com]

The thing that I wanted for this library was a way to explore ERDDAP servers, and examine datasets.  At first, I tried erddapy, and worked with it for a while. But what I wanted was individual object instances for each of the datasets discovered thru the search operations to have a mean to compare them and make loop operations. Also, I have a more coherent class space to place their functionality by doing it this way. 

I included this library functionality to parse the datasets dimensions, variables, metadata information for data discovery, also methods to create valid URL data requests for both Tabledap and Griddap datasets, and a way to request the data. 

In our experience, when we need to analyze the data, a pandas dataFrame result out of a Tabledap data request is more beneficial for an analysis of the data.  I included a way to create this dataframe after parsing a data request with a csv filetype.

And in the case of a Griddap dataset, data analysis makes more sense for me to connect to the OPeNDAP endpoint and create a xarray or a netCDF4.Dataset.   But the functionality to generate a data request and get text response (asc, csv, etc) or a binary format (mat, nc, png's) is there. I haven't put more effort into making higher abstractions for the Griddap data requests slicing, mainly because "xarray" and that the server-side operations that one of you points are only available for the Tabledap datasets (From what I tried, I imagine because adding them to grid results could be very demanding on the server).

A simple ERDDAP_Griddap usage  https://github.com/hmedrano/erddap-python/blob/main/notebooks/Griddap-demo.ipynb

I intend to keep on working on this library as the ERDDAP project keeps growing. ERDDAP has been really handy for the projects that I'm involved.

[Filipe Pires Alvarenga Fernandes]

> The thing that I wanted for this library was a way to explore ERDDAP servers, and examine datasets. At first, I tried erddapy, and worked with it for a while. But what I wanted was individual object instances for each of the datasets discovered thru the search operations to have a mean to compare them and make loop operations. Also, I have a more coherent class space to place their functionality by doing it this way.

Interesting use case. We did play with that idea last year [1] but no
user at the time thought it would be helpful so we abandoned it.
BTW, few free to open issues and get in touch. Open Source evolves
faster and better together!

> I included this library functionality to parse the datasets dimensions, variables, metadata information for data discovery, also methods to create valid URL data requests for both Tabledap and Griddap datasets, and a way to request the data.
>
> In our experience, when we need to analyze the data, a pandas dataFrame result out of a Tabledap data request is more beneficial for an analysis of the data. I included a way to create this dataframe after parsing a data request with a csv filetype.

That is aligned with our past experiments. See [1].

[1] https://gist.github.com/ocefpaf/ae0d650af68c0670e5f09d35c887129c

> And in the case of a Griddap dataset, data analysis makes more sense for me to connect to the OPeNDAP endpoint and create a xarray or a netCDF4.Dataset. But the functionality to generate a data request and get text response (asc, csv, etc) or a binary format (mat, nc, png's) is there. I haven't put more effort into making higher abstractions for the Griddap data requests slicing, mainly because "xarray" and that the server-side operations that one of you points are only available for the Tabledap datasets (From what I tried, I imagine because adding them to grid results could be very demanding on the server).

That probably says a lot about erddapy's docs since all of that is
possible. I'll improve on the examples and documentation to avoid such
confusion in the future.

The only main differences I see are:
1. API that resembles Javascript. That can lower the entry barrier for
users coming from JS but erddapy's python primitives as a first class
allow for people to build on top of it easily. See gliderpy, argopy,
collocate, and others. BTW, you could have achieved the same by
relying on erddapy and avoiding a lot of code duplication!

2. Support for ERDDAP's figures. We actually had that and removed b/c
Python users want to get the data and tweak the figures. The default
one is awesome for browsing data on the server but not very useful
when you want to create a figure for a presentation or paper.

> A simple ERDDAP_Griddap usage https://github.com/hmedrano/erddap-python/blob/main/notebooks/Griddap-demo.ipynb
>
> I intend to keep on working on this library as the ERDDAP project keeps growing. ERDDAP has been really handy for the projects that I'm involved.

Cool. Keep us posted on how it goes. I'll add to it erddapy's README
as an alternative implementation.

Best,

-F

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/d1a5b24f-7ac9-478b-b9e7-8b0c6d2152b7n%40googlegroups.com.

[Roy Mendelssohn - NOAA Federal]

Nice discussion. 

-Roy

To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/CAH2VmmD%3DeOPBom7nKYTSoWgkF5_-VJWFOnoWudY9C5Wa%2Bp9Bag%40mail.gmail.com.

[Tylar Murray]

I too have enjoyed this discussion. I think some of this may be worth including in the docs for either (or both) libraries. Particularly the philosophical approach vs ERDDAPpy:

> The thing that I wanted for this library was a way to explore ERDDAP servers, and examine datasets.  At first, I tried erddapy, and worked with it for a while. But what I wanted was individual object instances for each of the datasets discovered thru the search operations to have a mean to compare them and make loop operations. Also, I have a more coherent class space to place their functionality by doing it this way.

and the comparisons:

> 1. API that resembles Javascript. That can lower the entry barrier for
> users coming from JS but erddapy's python primitives as a first class
> allow for people to build on top of it easily. See gliderpy, argopy,
> collocate, and others. BTW, you could have achieved the same by
> relying on erddapy and avoiding a lot of code duplication!
>
> 2. Support for ERDDAP's figures. We actually had that and removed b/c
> Python users want to get the data and tweak the figures. The default
> one is awesome for browsing data on the server but not very useful
> when you want to create a figure for a presentation or paper.

Is there a page somewhere that lists ERDDAP resources for python users? I can whip up a .md but I wouldn't know where to put it.

[Bob Simons]

There is a list of Python resources related to ERDDAP at

https://coastwatch.pfeg.noaa.gov/erddap/griddap/documentation.html#Python

and https://coastwatch.pfeg.noaa.gov/erddap/tabledap/documentation.html#Python

I've asked Favio if I may add his tool to this list but haven't heard back yet.

It may be that I should make a Tools webpage to highlight the various tools for working with ERDDAP, or add the info to the web page which describes ERDDAP's RESTful services.  I'm open to requests/suggestions.

[Favio Medrano]

Thanks for all your comments,

On what Filipe points thanks for the invitation, I'll be glad to be more participative in projects.  Regarding the observations, yeah, I guess this design was more my choice, and  I have been using it mainly to do backend operation on tools for reporting and visualization.

And about the erddapy documentation, it's ok, I knew we could do request and url building with it, but I wanted a more fluent interface that chained methods to create url's, and requests, in this case the chain can continue with the pandas DataFrame.    So in this style of constructing url querys this libraries look very different.

I agree with the differences you point out, it looks kinda JS, my idea is to leave space to continue connecting to all the functionality of the ERDDAP Restful API, and the erddap-python design with classes for server,dataset,tabledap and griddap objetcs made more sense for me.  

I would point that adding support to get erddap plots can be very useful, for example to make pdf programmatically (our use case),  I agree that most users would prefer to use any of the plot libraries available for python. 

Keep us posted on how it goes. I'll add to it erddapy's README
as an alternative implementation.

I'll sure do, thanks for doing this. I guess I'll do the same as this work is inspired by what you have built.

Cheers,

-Favio 

--

- Favio

[Mathew Biddle - NOAA Affiliate]

@Tylar There is also: https://github.com/IrishMarineInstitute/awesome-erddap

To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/CAK6O7pOpCquvGdCE9xGjnHCWnSNh7quVuBbhBkUNKom5goskRw%40mail.gmail.com.

--

Mathew Biddle, Data Management Analyst
NOAA/NOS
US Integrated Ocean Observing System Office
1315 East-West Highway
Silver Spring MD 20910

ORCiD: 0000-0003-4897-1669
Contractor, Integrated Systems Solutions

http://www.ioos.noaa.gov/