GET a Resource via Composite Key?

[umbrae]

Hi List,

I've been struggling with mapping resources with composite keys to a RESTful interface. Maybe some of you have some ideas?

Here's a scenario:

You have a huge collection of widgets, and by necessity you've sharded the relational database that stores them into multiple databases. Let's say it's sharded by widget type. Each widget also has its own integer ID. So you essentially have a widget that is identified by a composite key of type and id.

In the API, in a GET of a single widget, you need to be able to specify both the widget type and the widget ID in order to find it quickly.

Here are some sample resources that could do that:

1. /widget-types/{widget_type}/widgets/{widget_id} - like /widget-types/gears/widgets/2321

2. /widgets/{widget_type}:{widget_id} - like /widgets/gears:2321

3. /widgets/{key of base64-encoded-string of widget_type:widget_id} - like /widgets/aGVsbG8gdGhlcmUu

Option 1 is non-ideal because it's pretty terrible design.

Option 2 is OK, but a bit odd and exposes implementation details pretty heavily.

Option 3 works fairly well because it's hiding implementation a bit. You've got a string key, but now you need to base64 decode in order to find your object.

I feel like there may be something here that I'm missing. Anyone else see any other possibilities? Consider a composite key a constraint: Lookup tables and the like are a no-go.

[mca]

umbraes:

<snip>

You have a huge collection of widgets, and by necessity you've sharded the relational database that stores them into multiple databases. Let's say it's sharded by widget type. Each widget also has its own integer ID. So you essentially have a widget that is identified by a composite key of type and id.

In the API, in a GET of a single widget, you need to be able to specify both the widget type and the widget ID in order to find it quickly.

</snip>
It's a bad idea to leak this type of storage detail information into the public interface (the Web API). When you pass out URIs, use a single value that will remain relatively constant and let the server (or middleware) sort out any shards or other details.

If you insist on exposing this data to the public, use a single value that is the combo of the two keys you need:
/widgets/{type}-{id} and let the server/middleware sort it out.

Base64 is unnecessary, unless you fear some exposure of info that b64 sufficiently hides for you.

Some general rules that might help you in your API design:
- your data model is not your resource model
- your object model is not your resource model
- you component model is not your resource model

See what i am sayin' here<g>?

Design your resource interface to closely match the needs of the API consumer, not the server technology. You'll end up with a much more usable and intuitive API and one that is independent of any technical implementation details inside the firewall.

Hope this helps.


mca

+1.859.757.1449
skype: mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://www.linkedin.com/in/mikeamundsen

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group, send email to api-craft+...@googlegroups.com.
Visit this group at http://groups.google.com/group/api-craft?hl=en.
 
 

[Bill Walton]

Hi umbrae,

On Wed, Aug 29, 2012 at 4:46 PM, umbrae <umb...@gmail.com> wrote:

Hi List,

I've been struggling with mapping resources with composite keys to a RESTful interface. Maybe some of you have some ideas?

Here's a scenario:

You have a huge collection of widgets, and by necessity you've sharded the relational database that stores them into multiple databases. Let's say it's sharded by widget type. Each widget also has its own integer ID. So you essentially have a widget that is identified by a composite key of type and id.

My position would be that you no longer have widget resources.  You have widget_type resources.  Lose widget in your url and the problem goes away.

Just my $.02

Best regards,
Bill

[umbrae]

Thanks mca. It's funny; I think when writing this question this was precisely the type of answer I wanted to avoid, hence naming composite as a constraint. I realize now that I was just fighting my instincts that this is bad structure by even posing the question so strictly. This helped me see that. Thanks.

[umbrae]

An interesting thought, Bill. That's certainly one way to handle it, unfortunately for my real use case it's not plausible. Thanks though!

[Andrew Eddie]

Notwithstanding the advice in previous replies, you could try the following if you are locked into the composite key (for whatever reason):

/widgets/:widgetType/:widgetId

Eg /widgets/gears/123

That gives you scope to list all widgets (/widgets/), just all gears (/widgets/gears/) or a single gear.

Regards
Andrew Eddie

[Rakesh Ravuri]

umbrae,Mike,

we have gone with a similar option as mike suggested, in our case when there are multiple id's for a resource like for a retail product you have a EAN, GTIN , ISBN etc.,

we have our url's as /v1/products/ean-123456 or /v1/products/gtin-456789 etc.,

rakesh ravuri
http://twitter.com/rravuri

[Neil Brewster]

I'm facing a similar problem and initially went with /widgets/{type}-{id} as well.

I had planned to urlencode the values of the keys (type and id) then join them by dashes to form the URI. However, I faced the problem that my {type} or {id} values may include dashes ("-").

The simple solution that came to mind was: in addition to urlencoding the value for each key, also replace "-" with "%2D".

However, after doing some reading, it seems it would be more appropriate to use one of the URI reserved delimiter characters (e.g. ":") here - referencing the following excerpts:

https://tools.ietf.org/html/rfc3986#section-2.2

These characters are called
"reserved" because they may (or may not) be defined as delimiters by
the generic syntax, by each scheme-specific syntax, or by the
implementation-specific syntax of a URI's dereferencing algorithm.

...

The purpose of reserved characters is to provide a set of delimiting
characters that are distinguishable from other data within a URI.

https://tools.ietf.org/html/rfc3986#section-3.3

Aside from dot-segments in hierarchical paths, a path segment is
considered opaque by the generic syntax.  URI producing applications
often use the reserved characters allowed in a segment to delimit
scheme-specific or dereference-handler-specific subcomponents.

Am I misinterpreting the RFC? I've found that ":" is tolerated by some tools and not others; apiary.io understands it if you use the reserved expansion URI Template form (e.g. /widgets/{+key}) of , but for example restify won't route requests to handlers that use it's URI-template-like ":id" syntax if the requests have a colon in them.

neil. 

On Wednesday, August 29, 2012 5:55:19 PM UTC-4, Mike Amundsen wrote:

[Repenning, Jack]

On Jun 19, 2013, at 11:02 AM, Neil Brewster <nbre...@leonidsystems.com> wrote:

https://tools.ietf.org/html/rfc3986#section-3.3

Aside from dot-segments in hierarchical paths, a path segment is
considered opaque by the generic syntax.  URI producing applications
often use the reserved characters allowed in a segment to delimit
scheme-specific or dereference-handler-specific subcomponents.

Am I misinterpreting the RFC? I've found that ":" is tolerated by some tools and not others; apiary.io understands it if you use the reserved expansion URI Template form (e.g. /widgets/{+key}) of , but for example restify won't route requests to handlers that use it's URI-template-like ":id" syntax if the requests have a colon in them.

I think you're reading the RFC aright, but that's not much consolation if your tools behave differently. Sounds like Restify is already using ':' for its own purposes.

I would have used "_" in such a case.

[PhilZen]

Hi all, 

Despite the age of this thread i just have to comment on this, hoping it will help somebody our there (or i may kindly be corrected)

On Wednesday, August 29, 2012 11:46:53 PM UTC+2, umbrae wrote:

3. /widgets/{key of base64-encoded-string of widget_type:widget_id} - like /widgets/[aGVsbG8gdGhlcmUu 

 

 [...]

Option 3 works fairly well because it's hiding implementation a bit. You've got a string key, but now you need to base64 decode in order to find your object.

Reg. Hiding Implementation:

From all i've read thus far this is exactly what you are after :) Making the URIs hackable in any way will only encourage clients to hard code stuff or just try to hack with your API endpoints (=URIs) in ways it's not supposed to be used. Better REST is afaik a more opaque URI. 

Reg. Base64:

i'm sure there may be better fits than base* for your specific api, however (from http://code.tutsplus.com/tutorials/base-what-a-practical-introduction-to-base-encoding--net-27590):

If you Base64 encode the number 959, the result is O/. Of course, this isn't a url-safe value because of the "/", so a url pointing to O/ would not be decoded as O/, but as O (which is the decimal value 14)

Just a quick addition - Alternative 4 ;)

Comparing the Stormpath API (http://docs.stormpath.com/rest/product-guide/), which has a great consistence and url "flatness". In a nutshell:

• (almost) no URI with more than 3 path fragments, and if there are (which is the minority) - those semantic relationships are consistently expressed:
• objects/:objectId/:object 
  if :object matches a top-level resource plural, this is a 1:n relationship, giving a collection of related groups (i.e. /applications/:id/groups)
  if :object is singular and doesn't resemble any known top-level entity, it will contain a flat object (= no nested objects)
  If :object is a top-level resource singular, it is a 1:1 relationship (and not found at the url, but at the top-level collection)
  
• All N:M relationship are materialized as a first-class resource themselves (becoming a top-level collection) the name ending *Membership

Looking at their maximum pragmatic approach at hyperlinking it feels like a semantically closed little box that can grow (i'm using saying `expand` on purpose ;) without losing any of it's structural integrity.