List of ERDDAPs? Search across multiple ERDDAPs?

[Tylar Murray]

I find myself often wanting to search for data available in *any* ERDDAP.

I mostly use google for this currently.

Does anyone maintain a running list of ERDDAP servers?

Or are there plans to support searching across multiple ERDDAP servers?

[Dr. Conor Delaney]

Hi Tylar,

The good people at the Irish Marine Institute put this list together:

https://github.com/IrishMarineInstitute/awesome-erddap

Regards

Conor

--
You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/3f21c64e-4101-41e8-9361-8e85a574328b%40googlegroups.com.

[Bob Simons]

Currently, a Google (or Duck Duck Go, or...) search for "ERDDAP" plus other terms is the only way I know of.

There is a partial and somewhat out-of -date list of ERDDAPs at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#organizations

1) There aren't plans for distributed search. I never really liked that approach because the response time is the response time of the slowest response. If you have 100 remote ERDDAPs, one of them will likely be really slow. But if someone wanted to do this, it would be pretty easy. And perhaps the solution to the slow response is just to show the responses as them come in (e.g., here's the response from ERD, and here's the response from PacIOOS, and from Marine Institute, and from ...).   I see that it would be nice to reject EDD...FromErddap datasets, so that just the original source ERDDAP dataset makes it into the results.

2) One could grab all the information from a group of ERDDAPs (e.g., monthly) and build a search system on top of that (e.g. with Lucene). That is equivalent to what NOAAs catalog does (grab all the ISO 19115's from all servers every month). That's okay, but it isn't very efficient or very up-to-date.

3) One could set up a huge ERDDAP which just pointed to all of the datasets in all of the other ERDDAPs. Because of ERDDAP's subscription system, that would be pretty efficient and quite up-to-date. But it seems like that steals the attention that should go to each of the individual ERDDAPs (e.g., PacIOOS should get full credit for PacIOOS-sourced datasets). So I am reluctant to do that.  (Although there is UAF ERDDAP, which is sort of a NOAA equivalent of that, but it isn't complete or up-to-date for a variety of reasons: https://upwell.pfeg.noaa.gov/erddap/status.html ).

I guess I vote for #1, since it is quick, easy-to-build, reasonably fast, and perfectly up-to-date. 

Does some clever programmer want to build that and host it (or find someone else to host it)? (I bet Rob, Kyle, or Roland could do that in a day.)

[Rob Fuller]

Yes #1 could be implemented quite easily in HTML+javascript just running in the web browser, and could hosted for free e.g. in github.io. The browser can simply send the requests to each of the backend erddaps.

I think it would only work with ERDDAP servers which have secure connections (https) and the CORS header would need to be enabled?

The process for adding new servers might be to add the url into the config file, and make a pull request on github, or just open a github issue, etc.

Sounds like a fun and useful challenge?

--

You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/de47bd82-7824-40f5-a827-4b67fa3d5bc9%40googlegroups.com.

[Kevin O'Brien - NOAA Affiliate]

It seems like leveraging the json-ld support that has been part of ERDDAP since v1.82 (as Rob knows, since he was involved!) is key to searching across ERDDAPs.  If I do a google dataset search on ERDDAP I get many hits (not even sure how many).  It would be great if there was an API into google dataset search which would allow harvesting of erddap endpoints....

kevin

To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/CAHi5mSLEJ32uKSo7yr%2BnWsK7mrQ%3DGuSnS5fHKJc5YePWmEK-kA%40mail.gmail.com.

--

Kevin O'Brien
Software Engineer
UW/JISAO
NOAA/PMEL/SDIG
206-526-6751

"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the  National Oceanic and Atmospheric Administration."

[Tylar Murray]

Thank you all for the great suggestions. This is my first time seeing "google dataset search" and I have bookmarked these lists for my future reference.

Rather than building something new from scratch is there an easy way to tie into data.gov?
A search for "ERDDAP" there does return a lot of datasets and "OPeNDAP" is even a "format" which can be used to filter search results.

Although, I guess this is an implementation of Bob's solution #2 not #1.

[Roy Mendelssohn - NOAA Federal]

My memory is that there have been issues integrating NOAA data catalogs into data.gov, and a better place might be data.noaa.gov. I think Kevin may be more up on this than I.

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/f140186e-b5b3-401c-b05b-318c73644b9a%40googlegroups.com.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Men...@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

[Bob Simons]

To Tylar: Problems with data.gov and data.noaa.gov include:

* They are not very up-to-date (harvested once per month at best).

* They are not very complete (just the info in the ISO 19115 document). 

* They just include US government or NOAA sources. There are lots of great ERDDAP installations outside the U.S. 

* They won't be able to do some types of searches that ERDDAP search can do (e.g., Advanced Search'es or allDatasets searches).

But you can use those sites now with no extra programming. As you say: I think if you include "ERDDAP" in the search terms, you get just ERDDAP sources.

The advantage to distributed ERDDAP searches is that they are perfectly up-to-date and can leverage ERDDAP search methods (which have access to all the information about each dataset).  You could build something on top of ERDDAP's simple search, or Advanced Search (including lat lon time bounding box), and/or allDatasets dataset (for more structured search).

I really like Rob's suggestion of just making it a web page that uses JavaScript and Html. (I told you he was clever.) Advantages:

* There is then no need for a server running an application. 

  You just load a local .html document into your browser and start the search.

  Or, any web site could host this html document. Then user would load it and run it (requests/processing done in user's browser).

* It would be faster, since there is no intermediary server. 

* The code is probably even simpler.

* The user could choose which ERDDAPs to search.

* On the remote ERDDAP's, the requests would correctly identify the IP address of the user. That way, if the user becomes abusive, the ERDDAP admin can blacklist that user.

 

Very interesting. 

[Bob Simons]

Okay. You can now go to

https://coastwatch.pfeg.noaa.gov/erddap/download/SearchMultipleERDDAPs.html

to see a web page that can search ~25 ERDDAPs.

• This is EXPERIMENTAL! I'm not promising this web page will be at this URL forever.
• It's done with JavaScript in the web page. That imposes a few limitations (at least for my meager JavaScript skills).
• It uses .jsonp requests to avoid CORS restrictions.
• It isn't perfect, but I think it's pretty good. I think it's useful.
• If you download the page, you can modify the list of ERDDAPs that it searches. I included almost all of the ERDDAPs listed at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html
• If you want your organization's ERDDAP to be added or removed from the list of ERDDAP's that are searched, let me know.

I hope you like it.

Please let me know if you think this is sufficient, whether it needs changes (which may not be possible, given the JavaScript limitations), or whether it should be rewritten as a web service (which would make it easier to add other features).

Tylar, thanks for the suggestion/request.

[Tylar Murray]

Looks great, thank you!
I will share and ask my group for feedback.
Will this get bundled with the erddap source code on github or posted elsewhere?

[Bob Simons]

For now, it's just going to sit where it is unofficially.

In the future, we can decide if it is sufficient, and how and where it should be included with ERDDAP.

[Cara Wilson - NOAA Federal]

That is very cool Bob! Thanks for taking the time to do that.  

Cara 

--
You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/b210d1bc-91b5-4f7d-ba22-773e85ab97fd%40googlegroups.com.

[Rob Fuller]

Nice one, Bob. It has the perfect look/feel of the ERDDAP interface.

I had begun an implementation but was frustrated by CORS and hadn't thought to use jsonp; Brilliant!

Thanks again, Bob. You're a practical kind of genius.

--

[Melanie Abecassis - NOAA Affiliate]

Thanks Bob, This is really cool and useful!!

Could you please add my ERDDAP to your list:

https://oceanwatch.pifsc.noaa.gov/erddap/index.html

Thanks!

Melanie

[Bob Simons]

Yes. Excellent. I've added your ERDDAP to SearchMultipleERDDAPs.html and the list of ERDDAP installations at setup.html. 

If anyone else wants to include their ERDDAP in either or both of those places, please email me. 

Hopefully, it is good for your ERDDAP and your organization (more people find and use your data) and the ERDDAP project (since prospective installers are impressed by seeing a long list of current installations).

Best wishes.