Resource Aggregation and Continual Fight to Keep your Conventions

[backwardsCoding Backwards]

Lets say you have this structure for a person for your REST API, this schema:

{
   "id": 12,
   "name":{
          "first":"Angie",
          "last": "Smith",
          "middle": "joy",
          "maiden": "crowly",
    },
    "address": {
          "street": "1122 Something St.",
          ..and so on...
    },
... and so on
}

and lets say you have a client of your API (lets say a very inexperienced set of web developers) come to you and say "Hey I need a list of all unique last names in the system".

Would you:

a) Tell that person "OK, sure, well names are already part of our people resource endpoint. We can return just a list of people that will give you just the name.last field back, and you'd only get one person for every different name.last that exists". Basically just give them an aggregated grouping of people in order to satisfy their need for basically every unique last name in the database for people.

so for hypothetical url example they'd just call /people?fields=name.last?aggregate

b) Say ok, I do everything you tell me to do because you apparently feel you still dictate the API after trying to talk to you the web team tactfully, and say Oh ok, we'll just create a special endpoint for you, we don't know what that resource or endpoint url would look like yet, but we'll give you just this:

{
     "lastNames": {
            "name": "Anderson",
            "name": "Alvertson",
         ....etc.
      }
}

And who knows what resource this even is, there is no name you can even give this.

And who knows how many "special endpoints" we'll need to create for this application therefore coupling our API to their implementation and disregarding the entire purpose of having and reusing resources that I have in place already.

You forgo all the benefits of REST because your client refused to understand that you have an Architecture and convention to maintain and you offer them what they need through resources you setup but they think it's odd that they have to adhere to any sort of resource template whatsoever and expect you to build infinitely/endless custom resources which there is no consistency or no way to even name these resources because they're coupling your API to their application and they expect you to adhere to whatever contract they feel you should design...meaning they feel they can literally tell you how to design your API and that you have no say!

And you know if you give in just this one time and bypass your current REST resource here which is "Person" that they'll most likely expect you to forgo any design decisions you've made or want to make with your API since you gave into them saying they don't care about your design yet you're the platform team creating this API for consumption company-wide!

or

c) just say f this and quit all together due to stress of dealing with these morons, and find a better place and team to work with, because after constructively and tactfully trying to tell them time and time again that we have conventions we're sticking with on the API, the web team still feels they need to dictate your REST API design 100% which they don't have a clue about REST and so they don't care what you say...they think they can just own the API even though it's you and the platform team building it out for the company for not only their consumption but possibly for other apps, services, or even exposing this API to public consumers in the future.

[Jørn Wildt]

I don't have an exact solution for you - but it is my belief that you should think of an API in the same sense as you do UI/UX - the customer has a set of user stories for you, you can question these and improve on them, but you cannot change the basic fact that "business" have needs that change over time. It is your job, as the owner/architect/senior-dev of the API to take these requests and make them possible. Business grows - so must your API do to - asking for new features and indicating new needs does not make people morons, that is quite the wrong place to start.

But reporting *is* a beast to handle (so is naming), but that doesn't relate to API only - it would be exactly the same for those dealing with the UI of the application.

Take a look at URLs for OData for inspiration: http://www.odata.org/documentation/odata-version-3-0/url-conventions/ ... or give the customer an Excel sheet to work with. Sometimes that is actually a feasible solution for custom reports :-)

/Jørn

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.
Visit this group at http://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/d/optout.

[Adrian Lynch]

Welcome to the world of development!

I've had similar battles. Conversations are had where I push requirements back into the client of an API. Maybe teaching them about the principles of good API design might help.

As for a solution...

You have a user end point /users/:id? Do you also have a users end point /users?

How about augmenting it, much like you said, with something like /users?fields=lastname&groupOn=lastName? Or some other query param that makes sense, aggregate, distinct.

There are pros and cons to this, but at least you've nailed their requirement and you've kept your side in a consistent state.

--

[Jørn Wildt]

>  lets say you have a client of your API (lets say a very inexperienced set of web developers) come to you and say "Hey I need a list of all unique last names in the system".

Okay, this is probably only an example you have made up, but lets walk with it anyway. Instead of running away screaming you should take it as a start of a discussion. "Do you *really* need *all* names? That is an awful lot of data - is it wise to get that in one request?". "Why are all the names relevant for you - maybe you need to batch update them afterwards, perhaps we can find a combined solution?". "Could there be other cases where you only need a slice of the data, grouped or otherwise aggregated? Maybe there is a common pattern here we can solve." - and so on.

Never take a solution from a customer as granted - always question it, figure out what the customer is really asking for, their problem and needs, then propose a solution that both solves their problem and works with your product.

/Jørn

[Kijana Woodard]

+1

The most "dangerous" features are the ones where we agree with the other stakeholders at the outset. We tend not to push back / explore those models sufficiently leading to waste/bloat/bugs.

[backwardsCoding Backwards]

"you can question these and improve on them, but you cannot change the basic fact that "business" have needs that change over time"

Right, and REST can be flexible, I'm not saying that we can't be.  But we don't let one application or service consumer dictate the API design 100% therefore coupling your design to ONE application and ONE set of use cases when this is an API to be consumed by many.  Even if it wasn't an API to be consumed by many, I'd probably design it REST but flexible so that it CAN be reused.  Nobody is saying we aren't able to be flexible for the business.  But when you have certain conventions that make your service REST, you don't just dump them and say we're not gonna do it.

[backwardsCoding Backwards]

Yep I proposed something like /people?fields=name.lastname(groupBy=last.name)

now they're rolling with /people/search?fields...blah blah blah treating it as if this is some kinda search now when it's not.  We know the values that we expect to come in.  We know what resource the data they want is tied to, a PERSON.  My original intent which satisfied both sides was going to work right from the get go.

[backwardsCoding Backwards]

no it's really not an example I made up :).  I tried to take it and tactfully explain it to our web team, trust me I did.

No I'm not saying you need all names.  Did you not read, I said I wanted to add the ability to provide partial resources...that is, less fields for a resource.  All in their control!  for god sakes, what is not flexible there.

I do not take what customers want for granted.  I aim to come up with a solution that fits both sides but doesn't impact our API in a way that would hurt it meaning it's gotta be extensible and maintainable.  Nobody here said they can't do what they want, they absolutely can...just not in the way they expect the respone to be, which is barely any different then they proposed in the first place.  They're bitching about conventions.  I'm trying to KEEP my conventions.  REST gives that flexibility.  I'm building a REST service, not some hodgepodge.

That's the reality here.  I want my API to be simple for the end user too.  If we have a hodgepode of random  ways to get at the same data, nothing is reusable and the API becomes a mess to the end users as well because there's nothing familiar or no path to take when it's a fee-for-all and other teams feel they can design your API.

Again if you give them ways to get what they want, you shouldn't have to be fighting to keep your architecture sane.

[backwardsCoding Backwards]

I think that just about sums it up.

[backwardsCoding Backwards]

custom reporting and searches are a totally different ballgame I agree.  This is not that we're talking about here.

[backwardsCoding Backwards]

oh and Why are all the names relevant for you - maybe you need to batch update them afterwards, perhaps we can find a combined solution

because I'm building a REST API and I and the users gain both a lot of advantages to this consistency! that's the enitre point of a REST API.  Simplicity.  Along with flexibility.  Not chaos or lets mold the API with 1000 different structures none of which can be reused internally via the code that takes to get that data for the 1000 different conventions we've decided to go with because we allow all stakeholders to design our API and let along the fact that if we have no consistency, we can't reuse the service anywhere else, it's 100% coupled to that application.

it's pretty obvious to me why.  I've built REST APIs before, it's just I'm at a new place, and working with all new devs as well that just joined 6 mo ago.

If you don't get why I'm trying to be consistent, please, please do me a favor and watch this 1 hour excellent overview of it.  I'm taking a lot of stormpath conventions as well because they did such a great job deigning their public API in fact.  I'm also looking at GitHub's API and a few others.

http://www.slideshare.net/stormpath/rest-jsonapis

and here is an excellent link about partial resources as well, which would address the client's concerns and any client for that matter in the future for a REST API or at least the conventions I'm going with https://developers.google.com/drive/web/performance

[Darrel Miller]

BackwardsCoding Backwards,

We understand you are frustrated, we have all been there at some point.  But that is no excuse for lashing out at people who are trying to help with comments like "Did you not read..."

I do have a whole lot to say about the issues you bring up, but I'm a little reluctant to invest too much time responding to what looks like a throw away email account that just feels like venting.  Venting can be healthy, but I don't think here is the right venue.

I will say that there are lots of people on this list with plenty of experience in building HTTP APIs and who have a range of opinions on what REST APIs should look like.  Personally, I do build REST APIs that are designed with a specific client in mind.  I don't try a build re-usable resources.  That's not where I see re-use working the most effectively on the web.  I think "partial resources" are a mistake.  I know many others disagree with me, but that's ok, context is everything.

I agree with you that Stormpath have built a very nice API.  It is one of the few that have a root discovery document and they have some really nice touches where they have created specialized resources for avoiding having to take multiple steps to do the same thing.  Effectively creating redundant resources!

I think you could get a lot of value from the members of this list, but if you are looking for a somewhere that's just going to say, aren't those newb client devs stupid, you probably won't get that validation here.

regards,

Darrel

[backwardsCoding Backwards]

The what you call lashing out was in response to a poster's reply who clearly did not read my original post and assumed I'm not trying to provide value to the business or be flexible when I clearly stated that that's on my mind all the time in my original post.

Fine, don't respond, my comment wasn't that bad.  You have to realize when venting, you have to be a little more understanding as a replyer as well and again what I said wasn't lashing out, it was true, I asked if he read it because it appeared he did not.

[backwardsCoding Backwards]

Oh and I bet Darrel, next you're gonna say  "wow is this what it's like to work with you".  No, I'm actually tactful in the beginning with people.  It's when they insist in not working both ways and compromising together do I start to get angry :).  And I'm not alone saying this as a developer who is in charge of building this API.  I have other clients to ensure this API works for, not one web-platform guy who refuses to listen and I offered tactfully the first time to explain how we are providing his data and that I was explaining REST to him and why we have the convention to provide that data in a certain way for one of our endpoints.  He clearly did not want to work as a team and say "hmm sure fill me in, I want to understand why you are building and exposing your data this way".  

Did he ask that, nope.  Just go and bitches to his boss and his boss and yada yada.

On Wednesday, September 30, 2015 at 12:23:31 PM UTC-5, Darrel Miller wrote:

[backwardsCoding Backwards]

I think this guy has stated it best.  Just wanted to be sure I wasn't being "ridiculous".

On Wednesday, September 30, 2015 at 8:29:20 AM UTC-5, Kijana Woodard wrote: