Ticket 20147 - deprecate and replace request.META

[Luke Plant]

Hi all,

This is already the subject of a ticket, but I didn't get a response
yet. Basically, the idea is replace things like:

request.META['HTTP_ACCEPT']

with

request.HEADERS['Accept']

request.META should be deprecated and replaced with request._META,
because it is just an implementation detail, and a really bizarre one at
that, full of cruft from a previous generation of web applications (CGI)
that should not be exposed in our API.

Anything else needed from META should also be replaced with a sensible API.

This might seem to be a big change simply for the sake of a clean API,
but I'm more and more motivated by these thoughts:

* Web development is hard enough as it is. "Explain this to a newbie
without getting embarrassed" is a good test.

* A general philosophy of pro-actively keeping things clean and sensible
is necessary to avoid being overrun with madness long term.

There is also the advantage of a *much* cleaner repr(request),
especially in a development environment, because you don't have all the
other junk from os.environ.

The biggest problem is what to do with our test Client and
RequestFactory, which allow specifying headers as keyword arguments
using the CGI style of naming e.g.:

self.client.get('/path', HTTP_X_REQUESTED_WITH='XMLHttpRequest')

Since keyword arguments can't have "-" in them, this is harder to
replace. We could either leave it as it is, or add a new 'headers'
keyword argument to these methods, deprecating **extra.

The silliness has infected other places, like SECURE_PROXY_SSL_HEADER
which follows the same CGI convention (and in each case the docs have to
note how to do it correctly!). In this case we can detect people using
the old style 'HTTP_' and raise a deprecation warning, and allow the
sensible way.

We would probably also need to add a few methods/attributes to
HttpRequest to proxy the few things you need from request.META that are
not headers, like REMOTE_ADDRESS and REMOTE_USER

Is anyone strongly opposed to this? If not, in Aymeric's spirit of
decisiveness, I'll push it forward.

https://code.djangoproject.com/ticket/20147

Regards,

Luke


--
Be careful of reading health books, you might die of a misprint.
-- Mark Twain

Luke Plant || http://lukeplant.me.uk/

[Michael Manfre]

+1 to a saner repr(request) and not having to type "HTTP_". What sort of deprecation cycle are you thinking for this?

Regards,

Michael Manfre

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Carl Meyer]

Hi Luke,

On 04/08/2013 02:02 PM, Luke Plant wrote:
> This is already the subject of a ticket, but I didn't get a response
> yet. Basically, the idea is replace things like:
>
> request.META['HTTP_ACCEPT']
>
> with
>
> request.HEADERS['Accept']
>
> request.META should be deprecated and replaced with request._META,
> because it is just an implementation detail, and a really bizarre one at
> that, full of cruft from a previous generation of web applications (CGI)
> that should not be exposed in our API.

I have no problem with providing a nicer API for getting at request
headers that allows asking for un-mangled header names, but I don't
think we should deprecate request.META (or turn it into a private
implementation detail).

Although the concept of a unified "request environ" that includes HTTP
headers mashed together with various other environment and web server
info may date back to CGI, it is not outdated; in fact it is a part of
the WSGI specification too:
http://www.python.org/dev/peps/pep-3333/#environ-variables

I think the WSGI environ should always remain accessible through public
API on a Django request object.

> Anything else needed from META should also be replaced with a sensible API.

I'm not sure how you envision this happening, since there is no fixed
set of things that might be "needed from META" that we can provide
purpose-specific API for. It is a legitimate use of WSGI for people to
set arbitrary environ variables from their WSGI server and expect to be
able to read those variables from Django, and they shouldn't have to use
private API to do this.

> This might seem to be a big change simply for the sake of a clean API,
> but I'm more and more motivated by these thoughts:
>
> * Web development is hard enough as it is. "Explain this to a newbie
> without getting embarrassed" is a good test.

Sure; if we introduce a new API for getting at HTTP headers sans
name-mangling, I think it'd be fine to consider request.META "advanced
API" and adjust the documentation accordingly to make it much less
prominent.

> There is also the advantage of a *much* cleaner repr(request),
> especially in a development environment, because you don't have all the
> other junk from os.environ.

If we are trying to make repr(request) really be a full reproduction of
all relevant request data (such that the request instance could be fully
reconstructed from it) then I don't think this goal is achievable; it is
not an option (IMO) to remove WSGI environ data from the request
entirely, because this would make data that is part of the WSGI spec
inaccessible to Django users.

(I haven't checked whether the current repr(request) meets the
full-reconstruction criteria; if it already doesn't then I don't really
care what we show in it, we could trim it down further with or without
your proposed change.)

> The biggest problem is what to do with our test Client and
> RequestFactory, which allow specifying headers as keyword arguments
> using the CGI style of naming e.g.:
>
> self.client.get('/path', HTTP_X_REQUESTED_WITH='XMLHttpRequest')
>
> Since keyword arguments can't have "-" in them, this is harder to
> replace. We could either leave it as it is, or add a new 'headers'
> keyword argument to these methods, deprecating **extra.

Similar to above, I think we could add a new "headers" arg for
friendlier specification of HTTP headers, but I don't think we should
deprecate **extra. (This of course would mean we have to decide and
document which takes precedence if they conflict.)

> The silliness has infected other places, like SECURE_PROXY_SSL_HEADER
> which follows the same CGI convention (and in each case the docs have to
> note how to do it correctly!). In this case we can detect people using
> the old style 'HTTP_' and raise a deprecation warning, and allow the
> sensible way.

There might be cases where people are using an environ var rather than
an HTTP header to indicate proxied SSL, and currently
SECURE_PROXY_SSL_HEADER works fine this way (despite the HEADER in the
name). That said, I can't immediately find any current prominent
documents recommending this or situations where using a HTTP header
instead wouldn't be workable. Unless someone comes up with such a
situation, and given that both the name of the setting and its
documentation only discuss its use with headers, I think I'd be ok
deprecating its use with non-headers and moving to friendlier header names.

> We would probably also need to add a few methods/attributes to
> HttpRequest to proxy the few things you need from request.META that are
> not headers, like REMOTE_ADDRESS and REMOTE_USER

As I mentioned above, I don't think we can dictate the "few things you
need from request.META" and make the rest inaccessible.

> Is anyone strongly opposed to this? If not, in Aymeric's spirit of
> decisiveness, I'll push it forward.

I'm fine with new request.headers API; I'm opposed to deprecation of
request.META.

Carl

[Luke Plant]

On 08/04/13 21:17, Michael Manfre wrote:
> +1 to a saner repr(request) and not having to type "HTTP_". What sort of
> deprecation cycle are you thinking for this?

I'm thinking the normal cycle:

1.6 - request.HEADERS added, probably a few friends too.
- request._META added
- accessing request.META gives PendingDeprecationWarning

1.7 - request.META gives DeprecationWarning

1.8 - request.META is removed.

If we find that there are some other attributes/method that need to be
added to HttpRequest in Django 1.7, and direct access to request._META
was still necessary, then we could extend the deprecation cycle - 1.7
would give PendingDeprecationWarning.

[Russell Keith-Magee]

I agree with Carl. I like the idea of the new HEADERS that is simpler to access, but I think removing/renaming META isn't desirable. 

HEADERS will be introducing a layer of smarts, and as helpful as those smarts will be under 95% of circumstances, I'd be willing to bet that *someone* has a use case for getting at the raw headers. 

On top of that, there's plenty of code out there that is currently using META without any problems -- the issue is with new users understanding which META key to use, not with the key working correctly once it's been discovered. We're not closing a security hole, or making new functionality possible -- we're just making some code a little easier to read. Forcing all that code to be updated to use HEADERS strikes me as code churn.

I'd much rather see META continue as a publicly available, but generally discouraged API, rather than formally deprecating it.

Same here. 

Russ %-)

[Luke Plant]

On 08/04/13 22:13, Carl Meyer wrote:

> Although the concept of a unified "request environ" that includes HTTP
> headers mashed together with various other environment and web server
> info may date back to CGI, it is not outdated; in fact it is a part of
> the WSGI specification too:
> http://www.python.org/dev/peps/pep-3333/#environ-variables

I realise it is part of the WSGI spec, but encoding the CGI
peculiarities into a more modern spec seems pretty misguided to me - you
have to create a WSGI-CGI gateway to run CGI programs anyway, and you
could put the backwards compatibility mapping there, instead of in the
WSGI spec. I actually wish that WSGI didn't do any HTTP parsing at all.

>> There is also the advantage of a *much* cleaner repr(request),
>> especially in a development environment, because you don't have all the
>> other junk from os.environ.
>
> If we are trying to make repr(request) really be a full reproduction of
> all relevant request data (such that the request instance could be fully
> reconstructed from it) then I don't think this goal is achievable; it is
> not an option (IMO) to remove WSGI environ data from the request
> entirely, because this would make data that is part of the WSGI spec
> inaccessible to Django users.
>
> (I haven't checked whether the current repr(request) meets the
> full-reconstruction criteria; if it already doesn't then I don't really
> care what we show in it, we could trim it down further with or without
> your proposed change.)

You can't currently reproduce the request this way - the stream for the
request body contents comes out as something opaque.

You're right we could trim the repr with or without my change - it would
simply have made the set of things to display more obvious.

We should probably consider the places where repr(request) is used. It's
probably just in logging systems. However, Sentry is one of the most
popular and it does it's own representation of the request object, it
seems, so if we left META alone it wouldn't be affected.

It seems to me that repr(request) is most useful for
development-environment debugging, and currently could be improved a lot
for that use case.

>> Is anyone strongly opposed to this? If not, in Aymeric's spirit of
>> decisiveness, I'll push it forward.
>
> I'm fine with new request.headers API; I'm opposed to deprecation of
> request.META.

I would be fine with your modification of my proposal, not least because
it would be a *lot* less work!

The only disadvantage is "two ways to do it". If you want to grep for a
particular header, you've got 2 things to grep for now. However, that
problem already existed due to META - you've got things like
QUERY_STRING and request.GET, HTTP_HOST and request.get_host(),
X_REQUESTED_WITH and request.is_ajax().

Regards,

Luke


--
Clothes make the man. Naked people have little or no influence on
society.

[Val Neekman]

+1 for leaving META alone. 

--

[Marc Tamlyn]

I agree wholeheartedly with the introduction of request.HEADERS. I can see Carl's point regarding deprecating META, and I think it would be fine for it to exist as an alternative implementation in the same way as request.REQUEST does.

[Tom Evans]

On Mon, Apr 8, 2013 at 9:02 PM, Luke Plant <L.Pla...@cantab.net> wrote:
> Hi all,
>
> This is already the subject of a ticket, but I didn't get a response
> yet. Basically, the idea is replace things like:
>
> request.META['HTTP_ACCEPT']
>
> with
>
> request.HEADERS['Accept']
>
> request.META should be deprecated and replaced with request._META,
> because it is just an implementation detail, and a really bizarre one at
> that, full of cruft from a previous generation of web applications (CGI)
> that should not be exposed in our API.
>
> Anything else needed from META should also be replaced with a sensible API.

request.META is not just headers. What happens to the things that are
arbitrarily decided to be "not needed"? Who decides that HTTP_HOST is
important, but SERVER_PROTOCOL is not?

>
> This might seem to be a big change simply for the sake of a clean API,
> but I'm more and more motivated by these thoughts:
>
> * Web development is hard enough as it is. "Explain this to a newbie
> without getting embarrassed" is a good test.
>
> * A general philosophy of pro-actively keeping things clean and sensible
> is necessary to avoid being overrun with madness long term.
>
> There is also the advantage of a *much* cleaner repr(request),
> especially in a development environment, because you don't have all the
> other junk from os.environ.

The basis of web development was CGI. From CGI came everything.
Therefore, almost all webservers - Apache, nginx - almost all
frameworks - RoR, PHP, Django - that deal with the web will have
similar or equivalent environment hashes like this.
The convention for headers (uppercase header name, s/-/_/g, prepend
'HTTP_') is similarly ubiquitous.

Your argument is that this structure is confusing to the absolute
newbie who has never programmed a web application in his life. My
counter argument is that changing it would be confusing to anyone who
has ever programmed a web application in the last 20 years.

>
> The biggest problem is what to do with our test Client and
> RequestFactory, which allow specifying headers as keyword arguments
> using the CGI style of naming e.g.:
>
> self.client.get('/path', HTTP_X_REQUESTED_WITH='XMLHttpRequest')
>
> Since keyword arguments can't have "-" in them, this is harder to
> replace. We could either leave it as it is, or add a new 'headers'
> keyword argument to these methods, deprecating **extra.

This seems a problem with the API to Client and RequestFactory, not a
problem with request.META

>
> The silliness has infected other places, like SECURE_PROXY_SSL_HEADER
> which follows the same CGI convention (and in each case the docs have to
> note how to do it correctly!). In this case we can detect people using
> the old style 'HTTP_' and raise a deprecation warning, and allow the
> sensible way.

Again, to me that seems like the flaw is in the code handling
SECURE_PROXY_SSL_HEADER. Yes, that code does need to look up the
header name in request.META, it does not need to be specified in that
format.

>
> We would probably also need to add a few methods/attributes to
> HttpRequest to proxy the few things you need from request.META that are
> not headers, like REMOTE_ADDRESS and REMOTE_USER
>
> Is anyone strongly opposed to this? If not, in Aymeric's spirit of
> decisiveness, I'll push it forward.

Changing this just in Django seems wrong - in fact it says that the
only thing of relevance in request.META is headers.

One man's cruft is another man's dinner. Everything in request.META is
of interest to someone, somewhere. When I'm debugging a failed
request, anything in request.META may be of interest.

If the intention is to make header handling easier for beginners to
understand, it might make more sense to normalize how one accesses and
sets headers in django at the same time. Having code like this:

response.HEADERS['Wibble'] = request.HEADERS['Wibble']

makes more sense than:

response['Wibble'] = request.HEADERS['Wibble']

I'd be -1 against dropping request.META, -1 on changing anything in
repr(request), +1 on adding re(quest|sponse).HEADERS that reads from
META/writes to _headers.

Cheers

Tom

[Luke Plant]

On 09/04/13 12:34, Tom Evans wrote:

> The basis of web development was CGI. From CGI came everything.
> Therefore, almost all webservers - Apache, nginx - almost all
> frameworks - RoR, PHP, Django - that deal with the web will have
> similar or equivalent environment hashes like this.
> The convention for headers (uppercase header name, s/-/_/g, prepend
> 'HTTP_') is similarly ubiquitous.
>
> Your argument is that this structure is confusing to the absolute
> newbie who has never programmed a web application in his life. My
> counter argument is that changing it would be confusing to anyone who
> has ever programmed a web application in the last 20 years.

Well, that depends on which frameworks they use. From what I can tell,
Java servlets have an API like request.getHeader("Accept-Language")
without the HTTP_ name mangling, and ASP.NET has
Request.Headers["Accept-Language"]. It's entirely possible to be a
competent web developer without knowing about CGI name mangling.

I don't often follow Java/C#, but I'll gladly follow anyone that is
using a sensible API, and I really don't believe that current
familiarity with quirky historical artefacts should hold us back -
Python 3, to take one example, would never have happened with that attitude.

I don't have (too much) of a problem with things like WSGI following
CGI, if only because they had to in order to gain traction, but that
isn't a reason why developers should be forced to do CGI name mangling
to access a header.

>> The biggest problem is what to do with our test Client and
>> RequestFactory, which allow specifying headers as keyword arguments
>> using the CGI style of naming e.g.:
>>
>> self.client.get('/path', HTTP_X_REQUESTED_WITH='XMLHttpRequest')
>>
>> Since keyword arguments can't have "-" in them, this is harder to
>> replace. We could either leave it as it is, or add a new 'headers'
>> keyword argument to these methods, deprecating **extra.
>
> This seems a problem with the API to Client and RequestFactory, not a
> problem with request.META

Now that we've dropped the proposal to deprecate request.META, and just
add request.headers as an alternative for getting HTTP headers, these
classes don't need to change. It was simply for consistency that I
proposed fixing them.

Regards,

Luke

--
Clothes make the man. Naked people have little or no influence on
society.