Best Practices for resource search syntax

[Sia]

Hi,

Are there best practices for a Boolean like search syntax for RESTful resources? (if "search" and RESTful are even compatible :-)

The idea is to support "and", "or", "not" ... possibly grouping of terms? I'd be interested to know what folks think about searching as a way to reduce the amount of data returned by the service.

Also what do folks think about "sort-by" capability.

Thanks.

[Greg Brail]

How about OData? It specifies a standard syntax for doing just that.

--
Gregory Brail  |  Technology  |  Apigee

[Ed Anuff]

I'd like OData a lot more if it had made use of matrix parameters in its syntax as TBL intended rather than using parenthesis all over the place.
Hoever, they do address the inclusion of query syntax and deal with a lot of the use cases around that.

[Arlo Belshee]

As one of the current designers of the OData protocol, I completely agree with you. Matrix parameters would have been a better approach (it’s the approach I used with my own custom formats before I started contributing to OData).

 

I’ll respond to the OP in another email. But I had to shout out a “hell yes” to your comment first.

 

Several of the architects have ideas of ways to allow changing the key syntax in a way that won’t break all the client implementations out there. But any approach takes a long time. We have to replace convention with server-driven URI templating (or another approach), then get the ecosystem of clients to upgrade. Updating the protocol is relatively easy; getting everyone to update the clients will take years.

 

This is, of course, one of the reasons that we are steadily working to replace all of our conventions with server-control (not just the links).

 

Arlo

[Arlo Belshee]

OData may or may not work well for you. It is a mature and complete protocol that works well for its target cases. But it doesn’t attempt to target all REST API use cases.

 

OData works best for structured or semi-structured data. It allows filtering, navigation, sorting, and projection (select) on the fields of data entities. However, it doesn’t provide facilities to search through unstructured data.

 

There are a number of other considerations that might make it more or less well suited to your particular situation. I’d be happy to provide guidance if you want to start a conversation.

 

I’d love more people to use OData (I do get paid to work on it after all). But I don’t want anyone to use it when it’s going to be a poor match for their situation. I’d far rather have 2 happy users that tell success stories than 50 users who tell tales of woe.

 

Arlo Belshee

Sr. Program Manager, OData, Microsoft

[Ed Anuff]

Thanks Arlo, very much appreciate your response on this.  As I said, I really like OData but feel that the syntax was perhaps decided prior to current thinking about best practices for REST URI syntax.  My hope would be that with efforts within OASIS consider the potential opportunity for wider adoption of OData outside of enterprise usage if some of these syntax issues were revisited.  I understand that the desire not to break backward compatibility and that should be addressed, but consider, the purpose of going to OASIS was likely not to further adoption of OData by IBM, SAP, and Microsoft since you're all presumably committed, but to take the steps to make it appeal to a wider audience.

[Arlo Belshee]

There are actually already efforts to improve the URI syntax. We’ve been working on it in open design within the community for a year now (http://www.odata.org/blog/2012/1/19/format-efficiency-take-2-really-clean-json is an example from the middle of the process; http://www.odata.org/blog/2012/4/17/json-light-sample-payloads has our current thinking). I assume those efforts will continue as design responsibility shifts from odata.org to the OASIS TC.

 

So far, this work has been happening as part of our work on the “JSON Light” format. The goal of this format is to remove the inline metadata from an OData JSON response. Much of the redundant data is URLs. But we don’t want to lose the hypermedia constraint.

 

So we’re adding URL templating to the $metadata resource. The server exposes its templates. And this means that the server no longer has to follow existing OData URI conventions. Or, rather, that it won’t have to once all the clients are smart enough to look for this new information in the existing $metadata resource.

 

We believe (and hope) that the advantages and cool factor of the new format will speed up client adoption of the new feature. And we hope that the TC will see the same value in these two features that we do, and so incorporate them into the new standard.

 

Arlo

 

From: api-...@googlegroups.com [mailto:api-...@googlegroups.com] On Behalf Of Ed Anuff
Sent: Friday, May 25, 2012 11:55 AM
To: api-...@googlegroups.com
Subject: Re: Best Practices for resource search syntax

 

Thanks Arlo, very much appreciate your response on this.  As I said, I really like OData but feel that the syntax was perhaps decided prior to current thinking about best practices for REST URI syntax.  My hope would be that with efforts within OASIS consider the potential opportunity for wider adoption of OData outside of enterprise usage if some of these syntax issues were revisited.  I understand that the desire not to break backward compatibility and that should be addressed, but consider, the purpose of going to OASIS was likely not to further adoption of OData by IBM, SAP, and Microsoft since you're all presumably committed, but to take the steps to make it appeal to a wider audience.

[Mike Schinkel]

On May 25, 2012, at 3:19 PM, Arlo Belshee wrote:

So we’re adding URL templating to the $metadata resource. The server exposes its templates. And this means that the server no longer has to follow existing OData URI conventions. Or, rather, that it won’t have to once all the clients are smart enough to look for this new information in the existing $metadata resource.

Personally, the idea of having a $metadata resource that provides URL Templates for the entire "service" is one I find very compelling because it could allow hypermedia to be overlaid on top of an existing otherwise RESTful API (especially if that $metadata resource itself is really lightweight.)  That way someone could evolve their API to hypermedia without having to rewrite and/or break old clients that are doing URL construction.

I'd like to study OData Light? I assume this is the best documentation for it:

https://skydrive.live.com/?cid=0d23ed2816deea7b&resid=D23ED2816DEEA7B!966&id=D23ED2816DEEA7B!966

If not, do you have a better link?

-Mike

[Arlo Belshee]

That’s the best document right now. Please follow up with any questions.

 

Arlo

 

From: api-...@googlegroups.com [mailto:api-...@googlegroups.com] On Behalf Of Mike Schinkel
Sent: Friday, May 25, 2012 12:29 PM
To: api-...@googlegroups.com
Subject: Re: Best Practices for resource search syntax

 

On May 25, 2012, at 3:19 PM, Arlo Belshee wrote:

[Brandon H]

Before I found this thread I started another (I was using the term "filter" as opposed to "search" when looking for other opinions)

 

https://groups.google.com/forum/?fromgroups#!topic/api-craft/DJCumwXlYB4

 

I'd like to find out more about OData and how it might be used but I'm also of the impression it's not a standard yet and so may have some implementation challenges on different systems. We are a .Net shop here so I am fairly certain it's supported (isn't MS championing the effort to get it included in some protocol?) but just a concern to throw out there if it's valid.

 

B

[Arlo Belshee]

OData is currently not a normative standard, but is also not a proprietary protocol. It’s halfway in-between: it’s been designed by a multi-company community that implements the various libraries. Microsoft has had a very strong influence in that community, but so have SAP and John Spurlock (the core maintainer for the open-source OData4J library).

 

Additionally, the design is always presented on blogs and discussed in the mailing list as soon as we develop a first draft. The discussions significantly alter the design by the time it gets incorporated into the core protocol (and sometimes convince us that a feature shouldn’t be added at all).

 

This is, BTW, exactly the same level of standardization as JSON has right now. There is no standards body, but there is a community that has defined a standard and there are many different people all implementing to that same standard and talking regularly with each other. JSON has more implementations than OData and a simpler problem domain. So this approach works well for them long-term, but isn’t enough for us.

 

The OData design process did have more of a smoke-filled room feel than I would like, but it was still far from proprietary. The smoke-filled room contained people from multiple companies – basically anyone who was making a significant investment in OData libraries or services. And it tried to be as transparent as possible, talking about stuff before it was decided and incorporating feedback.

 

However, we wanted to eliminate the smoke filled room.

 

The OASIS standardization effort is our attempt to eliminate the smoke filled room. Anyone can join the TC (well, in as much as anyone can join any standards body. There are no entrance requirements, but you do have to pay a licensing fee to OASIS). We hope that anyone making any significant investment in OData will either join or contact someone who is already in the TC to act as their representative.

 

If you’re from a big enterprise, join yourself or contact Microsoft, IBM, or SAP (I can give contact info for any of these) to act as your representative. If you’re from a smaller company, you might find Citrix or Progress Software better represent your needs. And if you’re an open-source person, I recommend contacting WSO2.

 

To summarize:

 

·         OData v1 was proprietary. There was only one implementation.

·         OData v2 and v3 were created by a community design process. We all agreed on the standard together, and everyone implemented that standard (well, most implementations have only implemented part of the standard so far, but the implemented parts are all done the same way).

·         OData vNext will be defined by OASIS and will be a normatively-reference-able, official standard.

·         OData v1 & v2 had dramatic protocol changes. V3 and vNext will change significantly less. And libraries are gaining developers and going faster. So most implementations fell behind the protocol standard early on, but the gap is now closing instead of growing.

 

I’m happy to answer any further questions you have about OData. I don’t think it’s the right solution for all APIs, and I don’t want to push it on you. But if you fall into the area where OData shines, I want to make sure you have good information to base your choice on.

 

Arlo Belshee

Sr. Program Manager, OData, Microsoft

 

From: api-...@googlegroups.com [mailto:api-...@googlegroups.com] On Behalf Of Brandon H
Sent: Friday, June 15, 2012 7:29 AM
To: api-...@googlegroups.com
Subject: Re: Best Practices for resource search syntax

 

Before I found this thread I started another (I was using the term "filter" as opposed to "search" when looking for other opinions)