GSoC - Generic REST API module for Django
bkonrad
Moss about the concept of a generic REST API module for Django. We
spoke about how this has been attempted before and some of the
remaining issues that still require attention. After our chat, I
compiled a problem statement that I believe roughly accommodates the
needs of the Django community. My objective here is to hone these
points and then create a solution proposal for review by the core
committers. Feedback greatly appreciated.
Problem Statement
Django needs a generic REST API module. This fact has been recognized
for some time by the community, and efforts have been made to build a
viable solution. To date, however, no module has shipped with a
Django release. Using Jacob Kaplan-Moss’s document “REST Worst
Practices” as well as industry resources such as Joe Gregorio
(Google), RESTful Web Services by Richardson and Ruby, and the
original paper on REST by Roy Fielding as guides, a list of Django
REST features has been compiled. By addressing each requirement in
the context of existing work, a specification for a functional
solution can be generated.
1. Predictable URI Naming
A logical, clean URI structure is at the core of any REST API.
Individual items and collections should have standard naming schemes
across all resources. Additionally, URIs should be static to the
greatest extent possible.
2. Graceful Handling of Complex Resources
Jacob’s example is a GET method to “/heros/superman”. Such a resource
needs to return a complete set of data on Superman, and this means
being able to aggregate data from multiple resources.
3. Generic / Pluggable Authentication
Resource data can be sensitive in nature and needs to be distributed
only to those that have a right to receive it. This permissions
system should be generic in the sense that a resource does not
personally allow / deny access, but instead delegates permission
requests to a generic permissions handler.
4. Flexible Serialization
Django API consumers will have disparate data format needs, and these
need to be addressed. Formats such as XML, YAML, JSON need to be
considered during implementation.
5. Clean Request Handling / Mapping
Jacob notes that browser limitations make one-to-one mapping of HTTP
methods (GET/POST/PUT/DELETE) to CRUD operations only a piece of the
puzzle. Flexibility is need here to accommodate various consumer
classes.
6. Proper Links / Foreign Key Resources
All resources which contain foreign key fields should be serialized to
contain links to said resources in the form of a URI. One potential
expansion on this concept is adding “deep” or “full” serialization
support, whereby a consumer can request that foreign key references be
“folded” in to the core object. This is open for debate.
7. Scalable Deployment / Low Coupling.
Jacob makes the good point in his “Worst Practices” document that the
resource usage profile of a heavily used API is nothing like that of
the web app component of a service. Therefore, the API module must be
capable of being run on its own server(s).
Thanks!
Bill Konrad
Ivan Sagalaev
> 1. Predictable URI Naming
Apart from my serious doubt that this can be done at all I'd like to
point out that the very notion of *predicting* URLs is not RESTful. URLs
are generated by server (by whatever logic it chooses) and client treats
them as opaque identifiers. The more you document or specify rules for
constructing URLs for clients the less RESTful your service is.
Kalyan Lanka
6. Proper Links / Foreign Key Resources
If the request for primary key starts our sending out "foldable" foreign key related objects, it can cause some security issues for the application. For example I would like to send about a person information but not all their addresses. I think it would be good if the spec. provides hooks so that the developer can customize based on the role of the consumer on how deep the response can go.
Bill Konrad
Thanks for the quick feedback. What I meant by predictable (and maybe
it's the wrong word in this case) is that when assigning a URI to a
resource, a convention is followed, not that the user can "predict"
the URI itself.
Bill Konrad
Good point. That would have to be a part of the specification 100%.
Any foreign key table entries that are "folded in" would have to check
out with the permissions component.
Alex Gaynor
Alex
--
"I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire
"The people's good is the highest law."--Cicero
Ariel Mauricio Nunez Gomez
http://github.com/ingenieroariel/dapi/tree/master is an intersting project working on the same thing, it's probably a good idea to look at it(and any other attempts) for reference.
Alex
Malcolm Tredinnick
> I was fortunate enough today to have a quick chat with Jacob Kaplan-
> Moss about the concept of a generic REST API module for Django. We
> spoke about how this has been attempted before and some of the
> remaining issues that still require attention. After our chat, I
> compiled a problem statement that I believe roughly accommodates the
> needs of the Django community. My objective here is to hone these
> points and then create a solution proposal for review by the core
> committers. Feedback greatly appreciated.
>
> Problem Statement
>
> Django needs a generic REST API module.
I actually question that very premise. Django needs to support generic
REST precisely by *not* trying to wrap it up as a "module". And I say
that as somebody who write REST-based applications almost all the time,
because it's simply a nice maintainable architecture style.
[...]
>
> 1. Predictable URI Naming
>
> A logical, clean URI structure is at the core of any REST API.
> Individual items and collections should have standard naming schemes
> across all resources. Additionally, URIs should be static to the
> greatest extent possible.
However, URI's vary infinitely across the complete set of all
applications. They need to be consistent within an application, but what
you do for a particular application is domain specific.
What's the existing problem in Django here? Clean, predictable URI's are
already possible. Lots of people already use them.
>
> 2. Graceful Handling of Complex Resources
>
> Jacob’s example is a GET method to “/heros/superman”. Such a resource
> needs to return a complete set of data on Superman, and this means
> being able to aggregate data from multiple resources.
Again, already possible with views. What isn't possible here?
>
> 3. Generic / Pluggable Authentication
>
> Resource data can be sensitive in nature and needs to be distributed
> only to those that have a right to receive it. This permissions
> system should be generic in the sense that a resource does not
> personally allow / deny access, but instead delegates permission
> requests to a generic permissions handler.
which permission system is used is up to the application. Django already
has the contrib.auth module, which can be used even without cookies
(although it requires some different middleware). There's a small gap
here in providing documentation and probably some HTTP authentication
header handling middleware.
>
> 4. Flexible Serialization
>
> Django API consumers will have disparate data format needs, and these
> need to be addressed. Formats such as XML, YAML, JSON need to be
> considered during implementation.
So we already have serialisers that handle all of those for querysets.
If you're serialising other things, there are a number of approaches
discussed in the various threads here already about GSOC for
serialisation. Subclassing the existing serialisers isn't particularly
difficult. "Solving" serialisation in general, whatever that means, is a
project all on its own (as witnessed by the broad scopes of the
proposals already discussed herre).
>
> 5. Clean Request Handling / Mapping
>
> Jacob notes that browser limitations make one-to-one mapping of HTTP
> methods (GET/POST/PUT/DELETE) to CRUD operations only a piece of the
> puzzle. Flexibility is need here to accommodate various consumer
> classes.
That's not too hard to do already. Again, you're describing a bullet
point here, rather than identifying some particular problem in Django or
proposing a solution (which is a consistent problem throughout this
document), so I'm not quite sure where you're coming from.
I have done a fair bit of thinking in this area, trying to work out the
*minimum* changes to Django to allow as much flexibility as possible.
Particularly based on ideas such as JAX-RS (JSR-311 -- the Java REST API
framework) and similar things. I think most of these problems --
dispatching to different handlers for different methods can be solved
fairly easily with two small modifications to how URL resolvers (and
reversers) are picked up and a couple of decorators. If nothing else
happens in the interim, that's something I'll do early in the 1.2
timeframe, as I ran out of time for 1.1.
>
> 6. Proper Links / Foreign Key Resources
>
> All resources which contain foreign key fields should be serialized to
> contain links to said resources in the form of a URI. One potential
> expansion on this concept is adding “deep” or “full” serialization
> support, whereby a consumer can request that foreign key references be
> “folded” in to the core object. This is open for debate.
Indeed it is. General REST design (and, remember, you're claiming to be
discussing a generic REST API, not something that provides a way to
display models) completely separates the API from the persistent
storage. They are orthogonal areas. *Sometimes* there will be a nice
mapping, but that's not the general case.
>
> 7. Scalable Deployment / Low Coupling.
>
> Jacob makes the good point in his “Worst Practices” document that the
> resource usage profile of a heavily used API is nothing like that of
> the web app component of a service. Therefore, the API module must be
> capable of being run on its own server(s).
I can't think of anything in Django prevents this now. Jacob's document
points out that the developers shouldn't be stupid and should understand
sensible computer software engineering practices, which will remain true
pretty much forever.
I realise the above is going to read very negatively, but I'm highly
skeptical of this proposal, particularly at such a late date when you've
already (hopefully!) read all the feedback on other people's proposals
and seen what we are after. Primarily, there's no proposal in this
proposal. What problems are you trying to fix? How are you proposing to
fix them?
Secondly, I'm not sure there's a huge problem that is solved by
implementing a lot of code in Django here. Most of the bullet points are
already possible and fairly easy. There's certainly some ground for
extra documentation, probably quite a lot of it, around this area.
Something I've been personally addressing in an, as yet incomplete,
series of blog posts, for example. However, documentation isn't a Summer
of Code project. The areas I think are missing aren't Summer of Code
length, either (they're more like Two Weekends of Code stuff.).
Don't get me wrong; there are definitely some small pieces of code
support needed so that developers can be as generic as they want to be
when designing their APIs. However, the whole space isn't conducive to
saying "this is the best practice" and forcing something on the users,
as REST is an architectural *style*, not an architecture.
So, if you're really enthused about this, please provide a lot more
details, starting with problem description. You've hopefully carefully
studied existing code such as the django-restapi project to understand
where the difficulties lie in the generic approach. I genuinely doubt
there is a generic set of helpers is possible, since "generic" covers
such a broad area. We need to loosen up a few things, such as allowing
custom URL resolving and reversing so that it can be done a bit more
easily for collections of views. I have an idea of the precise amount of
difficulty needed there, since it's something I've looked at a fair bit.
It's not a summer's worth.
I'd want to see a lot more specifics before I could evaluate a proposal
along these lines.
Regards,
Malcolm
Malcolm Tredinnick
> Ivan,
>
> Thanks for the quick feedback. What I meant by predictable (and maybe
> it's the wrong word in this case) is that when assigning a URI to a
> resource, a convention is followed, not that the user can "predict"
> the URI itself.
It's entirely up to the developer as to which pattern to follow. It's
not something the software can help with (although it's already
encouraged, since we use reg-exps to parse the URIs).
For example, there are very valid use-cases, when including a locale
specifier in a URL for putting it either at the front or the end of the
URL (e.g. /de/foo/blah vs /foo/blah/de). Both are valid and good
practice in different cases. Even within the same website, both styles
might be appropriate (it's not going to be common, but I can think of
some cases). There's no way for a set of libraries like Django to
provide more than a way to parse out useful bits of URLs for situations
like that (and locale parsing, particularly in the leading-component
case, is one of the use-cases for some of the URL resolver modifications
we'll eventually need).
Regards,
Malcolm
Bill Konrad
much feedback.
Please let me clarify one thing. If you read the above as a proposal
than it wouldn't have seemed much like a proposal. I was only
outlining what a REST API module (I know you don't like that naming)
would need to provide. The proposal goes much further and contains
more detail.
The reason that I refer to it as a module and think there is more to
this than just an architectural style is that by providing an API
"service" that models can be registered with (think django-rest-
interface from 2007) a lot of the details can be ignored by someone
who is not interested in writing a full API from scratch. The
objective would be to let the developer define which data can be
accessed through the API, who can access it and have most of the
internal details handled for him.
For the record I 100% agree with you that a REST API can be written in
Django by hand, from scratch and be perfectly adherent to REST
principles. At the same time, however, I see room for a utility like
this. I think that the result would be a higher % of Django
applications exposing public facing APIs and creating an environment
of greater interoperability between applications.
Does that help clarify what I wrote a bit?
On Mar 31, 7:08 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
Ariel Mauricio Nunez Gomez
Malcolm Tredinnick
> First off, thanks for taking the time to read through it and give so
> much feedback.
>
> Please let me clarify one thing. If you read the above as a proposal
> than it wouldn't have seemed much like a proposal. I was only
> outlining what a REST API module (I know you don't like that naming)
> would need to provide. The proposal goes much further and contains
> more detail.
Well, you said you had created a problem statement and I assumed that
mail was it. Apparently not. It's a bit confusing as this whole proposal
to do something for Summer of Code hinges on there actually being a
problem to solve and a solution that solves it, not the fact that REST
exists.
> The reason that I refer to it as a module and think there is more to
> this than just an architectural style is that by providing an API
> "service" that models can be registered with (think django-rest-
> interface from 2007) a lot of the details can be ignored by someone
> who is not interested in writing a full API from scratch.
Then please stop calling it a generic REST API. Because it's not. It's
an interface to expose models. A generic REST API requires the full
power of Python because, as I've already noted, it's an orthogonal
concept to what is stored persistently. We provide a way to retrieve
arbitrary data -- view functions and the ORM. You can't hope to make it
any simpler than that for the general case.
> The
> objective would be to let the developer define which data can be
> accessed through the API, who can access it and have most of the
> internal details handled for him.
So is this another pass at the 2007 project again? Why not contribute to
django-restapi instead and keep that going along? I know why that
project got as far as it did and what the limitations and strengths of
it are (since I was the mentor for that student). For a
model-presentation API, it's going in the right direction and people are
using it.
The devil is in the details here and the worthiness of a proposal along
these lines very much on what you are thinking requires changes in
Django's core. I think the answer to that is "very little". Which then
leads me to wonder about what you're proposing to do.
[...]
> Does that help clarify what I wrote a bit?
Not really, because I still don't know what you're proposing to do and
the specifics are important for a Summer of Code project.
Regards,
Malcolm
Bill Konrad
I took a look at django-restapi and it seems like a great start.
Having been the mentor for the first run at it, what features would
you like to see added? If you have a minute, I think that would
definitely serve as a great guide.
Thanks,
Bill Konrad
On Mar 31, 9:15 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
Bill Konrad
I'm just following up on my last post. I have gone over the django-
restapi library and would appreciate more guidance on the intended
direction of the project. As you suggested, following up on the
existing work there is a great way to get this feature to the public
this summer.
Thanks,
Bill Konrad
Malcolm Tredinnick
> Malcolm,
>
> I'm just following up on my last post.
A whole 12 hours after the previous one. I had a day away from mail.
That happens sometimes.
> I have gone over the django-
> restapi library and would appreciate more guidance on the intended
> direction of the project. As you suggested, following up on the
> existing work there is a great way to get this feature to the public
> this summer.
It's already "to the public" -- it's been publicly available and
maintained (brought up to date to run with Django 1.0) on Google code
for a few years. There's even been a maintainer hand-over last year,
indicating ongoing life and maybe one day, the people maintaining it
will propose it for addition to django.contrib.
This is one of my big problems with your proposal so far. I'm not seeing
what you're bringing to the table here. I don't think working further on
django-restapi is a Summer of Code project. I don't think it's a
candidate to be merged into django.contrib (yet) either. I also don't
have particularly strong feelings about things that are needed, since I
don't really use it. I follow the code changes, but don't use it in my
applications. My REST apis don't map directly to models and there's lots
of intermediate processing going on, so I use normal Django views to do
the work. There's an issues list at Google code and a mailing list with
archives (django-rest on Google Groups) that is low enough volume you
could look through there for ideas.
I like the broad structure used in django-restapi -- Andreas did a good
job of trying out a bunch of possibilities to come up with something
that is actually an aid, rather than an obstruction, for models.
However, this is really a third-party project, not something that is
improving Django's core, so it's out of scope for Summer of Code.
Working on duplicated work just so that it's a Summer of Code project
isn't a valid technical reason to do something, either. This keeps
coming back to my questions about what modifications are needed to solve
actual problems in core that makes this a worthwhile project?
I haven't heard anything from Jacob about what he thought he discussed
with you on this issue. Your version of it is so far a bit shy on
specifics and it looks like you're still hunting around for "do
something cool with REST" project. Unfortunately, asking me for ideas
isn't going to help you, since I've already indicated that apart from a
couple of small resolver changes and short utility functions to mark up
which resolver to use for a function (the former being appropriate for
core in 1.2, the latter maybe or maybe not), I don't see much extra that
is needed. We already have a fairly viable third-party application that
people can contribute to if they want model -> front-end RESTful-style
API (django-restapi). It's not something I've found myself needing,
since writing an appropriate API for the code I have in front of me
isn't that hard.
So, once again, I'm forced to ask -- and this is something you really
need to answer if you're going to come up with a proposal: What problems
are you trying to solve that aren't already possible? Why do these
problems need core changes to Django?
Regards,
Malcolm