Request for Comment by September 5th: HSDA Search protocol

[Greg Bloom]

Hi folks - once again with the leadership of API Evangelist Kin Lane, we're going to conduct a quick specification development sprint – specifically around the Human Service Data API specifications' Search protocol. If the finer points of search queries and API protocols are of interest to you, read on.

Some background context: when we published the Human Services Data API protocols last year, we recognized at the time that the Search protocol was relatively under-specified. We've had a series of discussions in our Github queue about how it might be fleshed out. While a whole range of other issues have been tabled until we begin version 2.0, Kin has proposed (and quite a few others have also asked me) that we conduct an upgrade of Search in the meantime.

Kin has finally found some time during the first week of September to work on this. I can be available to co-facilitate at least a half-day real-time virtual conversation on either September 6th or 7th. Between now and then, we're eager to get your input on what should be included. 

To be clear: our objective is NOT to address every possible search use case, but rather articulate a core, high-priority set of actions that are common across the board. If you have technical opinions about this, at a minimum we expect you can offer up some narrative description of your use case; ideally, you would be able to offer a representation of your proposal in the OpenAPI format (i.e. Swagger). (Kin can offer guidance on this front.)

At the end of this Request for Comment period on September 5th, we'll hash out a proposal to bring back to the group for further reflection and discussion and testing. Assuming everything goes well, we'll approve a new version of HSDA with this new Search API protocol in October. (IF we run into conflicts around which we reach no rough consensus, in October we'll identify a more formal testing and deliberation process to move forward.)

Eager to hear your thoughts, either here or in our Slack or in the Github issues. Thanks again to all who got us this far!

~greg

-- 

Leading up the Open Referral Initiative.

Book a time to talk on my calendar.

Reach me directly at 305.962.2859

Web: OpenReferral.org

Community: Open Referral Forum

[Greg Bloom]

Hi folks, I've had a couple of groups ask for more time on the Search API Request for Comment. So i'm thinking we might push this back a month. 

I'll be traveling/conferencing the first week of October, so we might look to the 2nd week of October. I will loop back once we've sorted out a specific target date.

Feel free to reach out to discuss on-thread and/or offline to me. 

Thanks, all!

greg

[Greg Bloom]

Hi folks - thanks to those who offered feedback for our Search API protocol upgrade cycle so far. 

Kin Lane has just posted a slew of comments, recommendations, and questions. If this is your kind of thing, see the roundup of issues here: https://github.com/openreferral/api-specification/issues/84

We'll take time to review these items next week, to be partially coordinated in our Slack. (If you're interested in joining a real-time chat, please let me know and i'll include you in the scheduling process.)

And then the week of 10/22, we'll field one last round of suggestions before Kin delivers a final proposal.

Thanks again, and please reach out if you have any questions or suggestions.

~greg

[Greg Bloom]

Hi folks - I just walked through the proposed set of upgrades with Kin via video chat. Here are links and some very rough notes, also pasted below.

Video chat: https://www.youtube.com/watch?v=6lSFt3CYbAk

Github issues: https://github.com/openreferral/api-specification/labels/hsda-search

5:02 — design objectives for Search API

Live services - Why do we need a separate service? Why can’t we conduct search queryies at the level of the Organization / location / service API protocols? Why is this either or? Well it’s not — you cna conduct searches through the core API specification on any of those entities, and get a basic response. We want to keep that especially simple in order to reach the widest possibile audience, keeping it simple — enabling relatively nontechnical peopel to get a simple CSV, for example — while still meeting the needs of developers who want to be able to do more. So Search API is designed to do advanced level searches across various entity types, to target specific fields with particular behaviors, in a way that enables programming complexity while keeping the core spec simple. Taxonomy management is similar: we have a particular microservice that keeps the taxonomy in a place designed specifically for it, but working well with the others.

8:41 — what’s on the agenda for this cycle?

With Search, where we left off: we got to the point of establishing the API protocol for search but didn’t really answer a lot of questions about how we conduct search. Should queries be included in the URL path, or just in the body? Developers have different opinions about this. HAving well-designed paths can give people guidance.

One of the best conversations i saw happening was around how the search should occur - is it query parameters, as a single path, or is it about posting to the bodies… Many developers might not know about how to do that, but on the other hand, Neil observed that it’s more secure to post into the body. You can’t encrypt query parameters; but if you send it in the body, that can be encrypted. So the goal is to establish a separate path that allows the submission of the body.

So now we’re developing a search protocol that allows for SSL search, encryption and security. The technical bar is high, but we should provide the option for people to be able to do it, while separating it from the core service.

15:17 — where’s the balance between the decisions we make at the spec vs the work that should be left to each particular developer to implement themselves?

You can be HSDA compliant without adopting the entire spec; you don’t have to ahve all the paths, all the services, these are just a set of common patterns coordinated at a high level view that keeps them working together in a coherent way. You can extend it, etc. People have observed that there’s a lot of paths, which is overwhelming. The core documentation doesn’t need to show them all; it can just show the core endpoints up front. (it can be set aside in an Advanced Usage section).

19:25 — Scotland’s implementation

With ALISS, we were able to assess their API design, and found an example of a suggestion that we worked back into the spec just based on their adaptation.

23:00 — so how should we direct people’s attention?

24:55 — overview of issues from current cycle

Rounded up here: https://github.com/openreferral/api-specification/issues/84

26:05: Secure search added -

so that someone can pass queries in POST. For orgs, locations, and services. This doesn’t burden down the basic core search, but can offer compliant secure searching. It can be audited to ensure compliance, and as its own path it’s easier to enforce.  

32:25: searching Services at Locations -

This was missing from HSDA 1.1. can now return results for services that are specifically offered at particular locations. An oversight from HSDA 1.1 that we’re adding back in.

Hierarchy -

Search by taxonomy / categories

35:25: why is Contacts its own object?

Contacts (?) - sometimes you might want to just have a rolodex. This is the CRM layer here. If you want to search for a phone number, you can figure out specifically who that person is. If you want to be able to pull all of the contacts in a given zipcode, it still may filter in the location… but it will give you a full set of contacts. Is this use case important? Or should we remove this segment?

39:49: Subresource fields: location, longitude / latitude, radius and proximity.

Specific fields that are specifically typed.

HSDA Proximity Search

HSDA Search Proximity -

Likewise search by postal codes.

^ two options for doing things. Either in the HSDA API core objects, or in the Search protocol. Can we get folks to weigh in? Should any of these be incorporated for any of the core objects, or just at the Search protocol?

Should secure search be just on the Search API or at the core too?

48:40 - HSDA taxonomies -

can manage multiple taxonomies, each service can be part of one or multiple taxonomies. We should check whether this structure matches with others’ understanding. Do we need this at all of the levels?

56:30 — summing up the theme of these issues

For these various issues, two questions: 1) does the structure look right? 2) does this belong in the core API (is it fundamental?) or should it be included only in the Search API (is it advanced?)

1:03:42 — do we need an established hierarchy for Orgs <> Services <> Locations?

HSDS has a set schema - Organizations, locations and services. HSDA is about how to provide consistent and comprehensive paths to access that information.

1:06:33 — final thoughts

More feedback needed! Chime in on the Github issues, ideally with OpenAPI examples.