Proposal: better support for generating rest apis in django core

[Karol Sikora]

Hi,

We was talked with Russell on djangocon eu about integrating more rich support for working django as rest api provider, focused on dealing with one-page web applications.

The motivations that currently without third party modules like django-rest-framework or tastypie its quite impossible to create rich web applications working as one-page.

As django aims to follow "batteries included" philosophy, so maybe it will be a good idea to integrate support for dealing with modern web applications.

Better rest apis support conists from two parts: models(and other data sources) serialization and handlers.

Django has currently limited support for simple serialization in django.core.serializers, but its quite useless in creating rest apis.

Ideally serialization support should have:

* possibility to serialize data to any format by plugabble serializers(in core i thing xml and json should be included),

* great integration with django models through eg, adding mixin class to model class definition

* possibility to integrate with other data sources(approach with appending mixin to class with data fields seems to be good for dealing with this case)

* possibility to provide data mutators as method in api class

* permissions mangement using user object

I propose approach to create class which can work either as mixin to model classes or as standalone base for other data sources. In this object we can configure serializer class, set up permissions and mutator methods.

This class should have endpoint method like rest_get(self, user, *args, **kwargs), which will be called from view/other to get serialized data, rest_get, rest_post, rest_put and so on(getting request as first parameter i think).

Validations should be integrated through forms I think, its quite mature and stable soultion, I dont see any argument to build custom solution.

Serializers should take care on privileges checking, as its coupled with user models. As we provide deep integrations with models it seems to be a good solution that follows "fat models" approach.

Next thing is dealing with http request. We propose to use CBV and based on them create light wrappers to define handled http verbs, provide serializer class and errors handling.

The question is, what You're thinking about whole concpet(maybe we won't integrate rest api supports in django core and redirect peoples to use django-rest-framerwork and similar projects), maybe my ideas are good but You find some holes on it?

We're read to implement this solution and maintain them in the future.

Waiting for Your opinion!

Karol

[Shai Berger]

Hi Karol,

On Saturday 18 May 2013 13:26:53 Karol Sikora wrote:
> Hi,
>
> We was talked with Russell on djangocon eu about integrating more rich
> support for working django as rest api provider, focused on dealing with
> one-page web applications.
> The motivations that currently without third party modules like
> django-rest-framework or tastypie its quite impossible to create rich web
> applications working as one-page.
> As django aims to follow "batteries included" philosophy, so maybe it will
> be a good idea to integrate support for dealing with modern web
> applications.
>

My understanding of Django's philosophy is quite different; in contrast to what
you imply, I think that the rich eco-system of 3rd-party packages, including
mini-frameworks, is one Django's important strengths.

In other words, I don't see the requirement to use a third-party module as a
problem in itself. If this requirement causes you some other problems, please
explain what they are.

Other than that, I read your suggestion as "let's make a new REST-API mini-
framework, but make it in core". That's not the way Django has done such
things in the past -- usually, a third-party package that gained wide
acceptance, to the point that it was a de-facto standard, is then accepted
into core. A classic example for this is staticfiles. Currently, as far as I
can see, there is no clear "winner" in this space -- django-rest-framework and
tastypie are both strong contenders; so we're not in a good position to take
one of them in, and certainly not in position to throw them both out.

And further -- unless I missed something, there is nothing in your current
suggestion that cannot be implemented as a third-party package. You are more
than welcome to do just that -- perhaps you will be able to do something that
convinces core to include it; but you should do this with code, not plans. I
am pretty certain that you will not get any guarantee that your code will be
included, until core developers can see it.

If I misunderstood your suggestion, please clarify it; if there is anything
missing in core for you to implement your suggestion, please ask for it --
Django has been pretty receptive for changes that make it more extensible and
adaptable.

Hope this helps,
Shai.

[Russell Keith-Magee]

On Sat, May 25, 2013 at 1:00 AM, Shai Berger <sh...@platonix.com> wrote:

Hi Karol,

On Saturday 18 May 2013 13:26:53 Karol Sikora wrote:
> Hi,
>
> We was talked with Russell on djangocon eu about integrating more rich
> support for working django as rest api provider, focused on dealing with
> one-page web applications.

Apologies, but I don't recall this specific conversation (I had a few over the DjangoCon week :-), but it sounds like either I misunderstood the intention of your question, or you misunderstood my answer.

I know I advocated for the Django ecosystem to pay more attention to the ways  that the web is changing (and one-page apps are part of those changes); however, I don't recall suggesting that this necessarily meant Django's *core* needed to gain a ReST framework implementation.

> The motivations that currently without third party modules like
> django-rest-framework or tastypie its quite impossible to create rich web
> applications working as one-page.

Saying "It's impossible to build a 1 page app without using external apps" implies that it's difficult to install external apps, or that you should be able to build a one-page app without external dependencies. Neither of these things are true.

 

> As django aims to follow "batteries included" philosophy, so maybe it will
> be a good idea to integrate support for dealing with modern web
> applications.

No - Django *doesn't* aim to follow a batteries included philosophy. We aim to include the parts that are necessary to solve the core parts of the web problem - the parts necessary to turn an RFC-compliant HTTP request into a HTTP response, and those supporting tools that are so common or difficult to implement that there is value in having an agreed common implementation. 

We actively encourage the development of an ecosystem of third party applications for everything else. We specifically *don't* wan't Django to become a library of *everything* you need to do stuff on the web.

 

My understanding of Django's philosophy is quite different; in contrast to what
you imply, I think that the rich eco-system of 3rd-party packages, including
mini-frameworks, is one Django's important strengths.

In other words, I don't see the requirement to use a third-party module as a
problem in itself. If this requirement causes you some other problems, please
explain what they are.

Other than that, I read your suggestion as "let's make a new REST-API mini-
framework, but make it in core". That's not the way Django has done such
things in the past -- usually, a third-party package that gained wide
acceptance, to the point that it was a de-facto standard, is then accepted
into core. A classic example for this is staticfiles.

True for static files, but not universally true. contrib.messages, for example, was born out of an attempt to merge the best parts out of half a dozen user-messages frameworks that were in use at the time. If it appears that the ecosystem is developing a large range of substantially similar approaches to the same problem, there's an argument to be made to solve the problem once and for all, and get it in core.

However, from my knowledge of TastyPie and ReST Framework, that isn't what has happened. I'm not convinced that the differences are purely cosmetic -- there are a couple of significant architectural differences. 

 

Yours,

Russ Magee %-)

[Tom Christie]

Hi Karol,

  Thanks for bringing that up.  I'll start by saying that I would love to see richer Web API support in core Django - in my opinion there are absolutely some fundamentals that could be addressed.   However, I personally think that the benefit that we'd get probably isn't (yet?) worth the very large amount of work that would be required.  I'll get onto that in a moment, but I'm first going to go through some of the core components that could be addressed...

Request parsing is an obvious one.  Right now it's actually quite awkward to deal with parsing form data in anything other than POST requests.  There's subtleties to get right for supporting both url-encoded and multipart forms, handling charset encodings, handling corrupt data, and dealing with the various possible states of the underlying stream.  That gets more complicated again if you want to support both form data and json or other content types.  That sort of thing has to be tackled each time by every new API framework, and it'd be a good candidate for better support in core.  REST framework's approach is to introduce a `request.DATA` that supports parsing arbitrary content types depending on which parser are registered (either globally or per-view).  Something similar in Django would be valuable, because it's a minimal amount of API, but addresses a fundamental issue.

Serialization is another fundamental that's currently lacking.  This is a deceptively difficult area, particularly the deserialization and validation aspects.  Django's forms aren't sufficient for many of the use-cases you need to support with APIs.  A mature, complete serialization framework needs to be able to support nested objects, various types of representation for relationships, support for both ORM and non-ORM data, serializing and deserializing from composites of models (eg both User data and UserProfile data in a single representation), and various other bits & pieces.  Ideally a serialization framework should be flexible enough that it's equally capable of rendering a model instance into an HTML form representation, as it is of rendering it into JSON.  Furthermore, if someone wanted to get a serialization framework into Django it'd make sense if it could also be used to replace the existing fixture loading/saving.  Getting a really decent serialization framework into Django would obviously be a valuable thing to do, as it'd give us all a common, well-supported way of doing for Web APIs what forms currently allow us to do in Web apps.  However, it'd evidently require a very large amount of work to successfully land in Django.

Content negotiated responses are another possible candidate.  Content negotiation is a core component of Web APIs, but we've currently no support in Django for dealing with it.  There's not any great value in each new API framework solving content negotiation again, because there's a reasonably consistent set of rules about how it should be handled.  Being able to return a response and have it render into various formats depending on the client request is something that could be dealt with in core Django once, and then reused by any API framework.

Another area worth mentioning is authentication - in particular CSRF protection.  The right way to deal with CSRF protection in Web APIs is non-obvious.  Session authentication is valid for AJAX clients, and should require CSRF protection.  Other authentication types such as OAuth are also valid but should not require CSRF protection.  Handling that in a correct, secure way is fiddly, and something that I'm personally a little uncomfortable with API frameworks having to individually deal with.

I think it's feasible that we could address some of these core ares, and still leave plenty of space for API frameworks on top of Django to experiment and thrive, while improving the baseline support for folks using just Django.

Back to the point, though.  The amount of work that'd be required to comprehensively address this in core would be pretty huge.  We're in decent situation at the moment, with at least a couple of great, really well-supported API frameworks as third-party packages.  Having those packages independent to Django is disadvantageous in that it fragments the already limited resources of our community, but it's also advantageous in that they can iterate more quickly than core Django, as well as being able to take opinionated design decisions that suit their domain.

Django's ecosystem is pretty much it's biggest strength.  In my opinion the cheapest, most beneficial option would be to look at ways to better promote third party packages from within the documentation.  Obviously there's some discussion to be had about how to do that in a fair and open way, while still allowing an element of curation and ensuring that packages meet a required quality.  Calling out quality packages in some way from the docs would help new users find their way around the ecosystem, help build communities around successful packages, and be a good incentive to maintainers.  It's hard to know exactly what the most appropriate way to do this would be, but it does feel a little odd at the moment that third-party packages bring so much to Django, yet are rarely explicitly pointed to in the documentation.

If there was a wide consensus from the core dev team that improving Web API support within core Django was something that needs addressing, and a drive to make it happen, then I guess the priorities would change.  For now, continuing to build the communities behind the various third party packages seems like a pretty decent, low-cost win.

Thanks,

  Tom

[Vincent]

Huge +1 on everything Tom said, except the point about the docs:

Calling out quality packages in some way from the docs

The nice thing about developer communities is that they're fairly emergent. Consensus is usually formed all the way from the very bottom, on up. This works well.

Especially to a newcomer, an innocuous claim like, "X is a decent library for Y" from a well-known framework like Django may as well be, "X is the library you need to use for Y". This has the unfortunate effect of creating a monopoly, which invariably results in stagnation.