OData goes standard
Arlo Belshee
Given the recent discussion about desire to standardize a Level 3 hypermedia protocol, I thought I’d mention the efforts to standardize a “Level 2.5”(*) one.
I work at Microsoft on the OData protocol. I contribute code to a couple of implementations (mostly libraries) and help architect the protocol (along with many others, both inside and outside Microsoft).
Yesterday we (Microsoft, IBM, SAP, Citrix, Progress Software, and WSO2) requested that OASIS form a TC to produce an official OData standard. We’ve contributed the current protocol spec as a starting point. IBM and SAP have also contributed several potential extensions for the TC to consider.
One of many sites that have picked up our press release: http://www.techweekeurope.co.uk/news/open-data-to-oasis-79671
We would love people like the members of this list to get involved. Here’s some of the stuff going on.
· OASIS has opened the TC charter for comment. See http://lists.oasis-open.org/archives/tc-announce/201205/msg00008.html. This will be open through June 4, whereupon OASIS will decide whether to create the TC.
· If your organization would like to be listed as a Proposer of the TC, contact jo...@oasis-open.org before June 3.
· Assuming OASIS accepts the proposal, the TC will form not long after. Then we’ll start meeting to form the first standardized version of the OData protocol. TC membership is open. If you are interested, follow OASIS’s usual process to join our TC. We’d love to have you.
The TC’s explicit goal is to get a solid v1 out the door quickly. We also want to make it easy for the many existing applications and libraries to move to the new standard. Therefore, OASIS OData v1 is likely to be very similar to current OData v3. But this will not be a rubber stamp. Several of the proposers (including Microsoft) are bringing in proposals for improvements.
We are looking for people who have a passion and ability to:
· Improve the future: Guide a protocol to make the best possible general-purpose solution for sharing data in a REST API.
· Safeguard the past: Treasure the existing ecosystem of implementations and help them continue to flourish.
· Quickly do what works: Get something out the door. Not get stuck in endless in-TC bickering about what is “more correct,” but instead deliver a standard that works. And do it quickly enough that it is relevant and useful.
Balancing those 3 takes talent. If you think that your addition to the TC will help us better achieve our goals, please join.
Arlo
(*) I’m not dumb enough to claim Level 3. We believe strongly in the Level 3 values and try to design everything to meet those constraints – until something requires us to do otherwise. We (try to) ensure that the server is in control of all URIs, but sometimes we need improved regularity to make it easier for people to build automated tools. Also, some clients simply choose to ignore the server’s metadata and go by convention. As a public protocol we can’t control all of the clients, so they can choose to ignore the hypermedia constraint.
Also there’s a bunch of legacy holding over from v1.0 of the protocol. Can’t just eliminate it since backwards compatibility is the most important feature of a general protocol. So we have to find smart ways to add to the protocol in a way that old conventions become server-driven URI schemes.
Mike Schinkel
I work at Microsoft on the OData protocol. I contribute code to a couple of implementations (mostly libraries) and help architect the protocol (along with many others, both inside and outside Microsoft).Yesterday we (Microsoft, IBM, SAP, Citrix, Progress Software, and WSO2) requested that OASIS form a TC to produce an official OData standard. We’ve contributed the current protocol spec as a starting point. IBM and SAP have also contributed several potential extensions for the TC to consider.
Congrats to you and the other OData stakeholders. This is great news for the OData community and for all those who need its power.
-MikeHowever, I think OData is far too heavy to become a general purpose base for hypermedia. I'd envision something that a reasonably capable web developer should be able to grok in 15 minutes or less. If it takes more than that people will just continue to reinvent the wheel and go with URL construction. IMO and FWIW.
Arlo Belshee
I agree. OData is not a universal solution. I think it’s a great solution for some problems.
The complexity in OData is partly due to its attempt to be a complete solution (inherent complexity). However, it is predominantly due to our poor communications around it (incidental complexity).
I’ve seen people implement a legitimate OData service in 10 minutes, using only static files. It only supported a small subset of the full OData operations. But it was fully standard-compliant. And it was enough that some of the simpler OData clients (such as Excel’s PowerPivot plugin) could use it.
One of the things we’re starting to work on is OData capabilities. This lets a server declare what parts of OData it does or does not support (is filter too expensive on your backend? Say so!). As part of this, I hope that we do a good job of clarifying levels of compliance and making it a lot easier to grok the Level 1 OData. And to help people get a simple, compliant OData service up and running and only learn about additional complexity as their needs demand it.
I would, of course, welcome any help in this area.
Arlo Belshee
Sr. Program Manager, OData, Microsoft
Mike Schinkel
Agreed. For some problems you'd have to have a significant level of complexity to address the use-case. Absolutely.That said, I'd be curious if you have a concrete set of criteria that would help someone determine when the complexity of OData is effectively required?
I’ve seen people implement a legitimate OData service in 10 minutes, using only static files. It only supported a small subset of the full OData operations. But it was fully standard-compliant. And it was enough that some of the simpler OData clients (such as Excel’s PowerPivot plugin) could use it.
-MikeOf course. When people master something they can do it quickly. I'm now actually able to write completely correct working regular expressions ~90% of the time. But it took me many years thinking I'd never be able to and thus avoiding them. I didn't get to a point a reasonable RegEx mastery until I had a period of time where I was using them daily for many weeks.While OData probably isn't as hard as RegEx master (but what is? :) it would definitely would take a conscious investment of time and effort to get comfortable with it, no?Also by their very nature we would expect people to invest a lot of time into building a web service so learning OData is not that big of a concern, but it's the building of the client where we really need to be concerned with making as easy and as lightweight as possible, at least for public APIs. Again, IMO and FWIW.
Arlo Belshee
That said, I'd be curious if you have a concrete set of criteria that would help someone determine when the complexity of OData is effectively required?
I’d start by asking myself the following questions. The more Yes answers, the more OData is a good match for your needs.
1. Do I want to expose a data model? (As opposed to exposing a set of unrelated resources, an action-oriented service, or something else. By data model I mean something that is well described by sets of entities. Entities have properties, identity and links to other entities. It will be interesting to my users to filter sets of entities, navigate relationships, and the like.)
2. Do I want REST? (this includes being willing to accept its downsides – such as the latencies involved in request / response protocols.)
3. Do I want to reach part of the existing OData client ecosystem? (not just the libraries and the apps that will be written using them, but existing applications. Several OData clients, such as the Drupal plugin, the Excel plugin, and Tableau, are general and can talk to any OData service. So you get an app ecosystem the instant you launch.)
4. Do you want to enable public apps made by people you don’t know directly? (Many people are making services to support just their own apps. Which is great, but means OData is less critical for them. These may be public services. But they are intended to support only a few custom apps, typically written by the same company as wrote the service.)
OData’s complexity comes from attempting to do all 3 of these things.
#1 means that the service needs to support a useful expression language. It needn’t necessarily be ours, but it will need to support a similar set of functionality. It also means you need a media type. And it pretty much requires that you define a set of URI conventions. The point of a data model is that it has an infinite number of entry points. The server needs to be in charge of defining the allowed patterns – at request time. But the client is going to want to craft requests by combining those patterns.
#2 means that you have to support all the HTTP constraints. And the hypermedia constraints. And it comes into conflict from time to time with #1, so you have to decide how to resolve that conflict.
#3 means that you have to define a protocol that is independent of the particular data model. And it has to have a way for the server to announce its data model to a machine. You need something like our $metadata resource (which in hindsight we wish we’d named $model). It needs to be a well-known entry point that tells a machine everything it needs in order to manipulate your service.
#4 means that you have to worry a ton about back-compat and versioning. So once a mistake makes it into the protocol, it takes years to address it. And every feature is permanent too, which increases complexity. I’ve worked on a service where the majority of our users came through an app that we didn’t create. The company making that app then went out of business. No one had source for the app, but it still met user needs well. So we had to ensure that it kept running against our service. That adds all sorts of complexity, and OData assumes that somewhere in the world there is an OData service with this problem.
Then, of course, there is also incidental complexity. OData has some of that (most protocols & services will). It’s pure waste and we want to (and work to) get rid of it. But it’s something that any service author has to take into account. If they want #3, then they’ll need to join some protocol. And so they’ll take on the baggage that that protocol has not yet managed to eliminate.
When people master something they can do it quickly. … it would definitely would take a conscious investment of time and effort to get comfortable with it, no?
This was less a case of mastery. I honestly think that anyone could learn enough OData to get a working service going from scratch in an hour – if our documentation were up to the task. People can’t today, and that’s because our docs are really poor at getting people started.
A minimal OData service need only:
· speak one format (pick one, but Atom is usually easier due to higher-level tool & lib support),
· expose a minimal $metadata that describes the properties on the entities,
· expose a minimal service doc that lists the entity sets,
· expose its entities in the format of choice.
This is enough that some OData clients (such as PowerPivot in Excel) can use your service. It is enough to become a useful member of the OData ecosystem.
All the other stuff is useful but not required. You can add query options ($filter, etc) as needed. You can add relationships between entities as needed. You can add support for operations beyond GET as needed.
… it's the building of the client where we really need to be concerned with making as easy and as lightweight as possible, at least for public APIs.
Absolutely.
Our solution here is libraries. Microsoft produces client libraries for C# & VB (duh), and also for PHP, JS, and objective-c (that last one is open source, but we’re the primary contributor so far). There is a vibrant open source library for Java (run by John Spurlock, with frequent contributions from SAP), and two just starting out in Drupal (allowing you to bind a Drupal view to an OData service).
The static-language libraries mostly include client-side proxy generation. The dynamic-language ones provide a set of dynamic proxies. Drupal’s module integrates directly into the awesome Views infrastructure, making the OData service appear just like a database would. In each case, these libraries lift the client from speaking OData to speaking something native to the platform.
This doesn’t handle everything – there isn’t a good library for Ruby or Python, for example. But it does make consumption a ton easier for a wide range of clients.
Also, Microsoft shares most (all? I think all, but will only claim most) of its client libraries as shared source. We don’t necessarily take contributions to all of them, but we do release source along with binaries. Well, we’ve automated the binary releases but are actively working on the source release automation. So sometimes source releases lag. But it’ll get there.
We’ve also started releasing a low-level OData client library. This contains a fully valid reader and writer for every format and an experimental URI builder and request parser (produces an AST). This can be used directly on .Net. We also hope that the source can make it easier for others to implement the tricky bits of client libraries in other languages.
All that said, we do care a lot about the simplicity of the OData protocol for clients. We value browser demoability and discoverability. It isn’t our #1 concern, but it is something that we consider strongly. That was, for example, one of the primary driving forces behind the “JSON without inline metadata” movement that led to the upcoming JSON Light format.
Arlo
From: api-...@googlegroups.com [mailto:api-...@googlegroups.com]
On Behalf Of Mike Schinkel
Sent: Friday, May 25, 2012 12:23 PM
To: api-...@googlegroups.com
Subject: Re: OData goes standard
On May 25, 2012, at 3:07 PM, Arlo Belshee wrote:
Mike Schinkel
I’d start by asking myself the following questions. The more Yes answers, the more OData is a good match for your needs....
Thanks, that's a really good analysis; very helpful. Definitely value for when there are a lot of "yes" answers.
This was less a case of mastery. I honestly think that anyone could learn enough OData to get a working service going from scratch in an hour – if our documentation were up to the task. People can’t today, and that’s because our docs are really poor at getting people started.
A minimal OData service need only:· speak one format (pick one, but Atom is usually easier due to higher-level tool & lib support),
· expose a minimal $metadata that describes the properties on the entities,· expose a minimal service doc that lists the entity sets,· expose its entities in the format of choice.This is enough that some OData clients (such as PowerPivot in Excel) can use your service. It is enough to become a useful member of the OData ecosystem.
The big concern is not about how easy is it for one person or team to build a *service* but how easy is it for everyone who may potentially want to interact with the API to build a *client.*Your above comments assume a competent programmer, right? Many people building websites are not. They may be great designers but stumble along with when it comes to code. Or they may be assembling websites from other the shelf themes and plugins and are just starting to dabble in code.I'll contend that it's much easier for someone who really struggles with jQuery.get() to call Twitter's search API using URL construction then it would be to properly navigate an OData feed via hypermedia:
The upshot is if you publish an API that is harder to consume, fewer people will successfully build apps that can consume it. Web consultants w/o the skills to process OData will give their client's expensive quotes for integration because they know they'll have to outsource the work, and so many of those clients will decide to forego the integration. It's that simple and that's just one scenario.
Our solution here is libraries...
Libraries do not limit inherent complexity. Would be client developers still have to inspect the response representations, and the more apparent complexity the more people will be scared away.If the complexity is required by the nature of the use-case, no problem. But if the use-case is simplistic then the web service should be too. At least IMO.
Microsoft produces client libraries ... for Javascript, PHP
The OData Javascript library is 58K minimized (all of jQuery is only 95Kb.) That's a pretty heavy requirement if I want to do something small:
For PHP, evidently the libraries don't work on Mac OS X or Linux and it hasn't been updated since 2012 April:
Shouldn't OData libraries be solid of all major languages and platforms, not C# and .NET and not just Windows? 25 months since last update is a long time. Just saying...Honestly I don't see OData as a solution for most of our clients who are trying to create simple APIs for interacting with their SaaS, especially since our solutions must by their very nature be in PHP and Javascript. And that should be okay, right?That said, I can definitely envision some use-cases for OData. Just not all use-cases.