QueryDict and its .dict() method - why does it do that?

[Gabriel Pugliese]

This gist is self informative - some information from list is lost: https://gist.github.com/gabrielhpugliese/640b69eefc5b7490a07c

Some of my buddies have pasted Rails(Rack) and PHP conversion right below. Is that something I am missing? Does it have to do with laziness?

[Daniel Roseman]

On Thursday, 26 March 2015 17:52:48 UTC, Gabriel Pugliese wrote:

This gist is self informative - some information from list is lost: https://gist.github.com/gabrielhpugliese/640b69eefc5b7490a07c

Some of my buddies have pasted Rails(Rack) and PHP conversion right below. Is that something I am missing? Does it have to do with laziness?

Because Django is not Rails or PHP.

The [] convention is not a standard part of HTTP. I believe it was developed in PHP and DHH adopted it for Rails, but that doesn't make it a standard. Django does not expect data in that format and does not do any specific conversion for it.

The way to provide multiple values for an item in Django is by simply repeating the name:

    ?categories=1&categories=3

which you can then get from the querydict via `request.getlist('categories')

--

Daniel.

[Masklinn]

On 2015-03-26, at 18:48 , Gabriel Pugliese <gabrielh...@gmail.com> wrote:
> This gist is self informative - some information from list is lost: https://gist.github.com/gabrielhpugliese/640b69eefc5b7490a07c
>
> Some of my buddies have pasted Rails(Rack) and PHP conversion right below. Is that something I am missing? Does it have to do with laziness?

Django (alongside just about every system which isn't PHP, from which
Rails lifted this "feature") doesn't give any semantic significance to
querystring key names. Keys are not parsed, and there is no magical
autovivification feature.

There is no information loss you haven't requested, QueryDict.dict() (or using a
QueryDict as a dictionary, there usually is not reason to reify it to a
native dict object) behaves as if you'd passed the querystring's (key,
value) pairs to a dict constructor: the last key wins.

If you expect a key to be present multiple times in a querystring, use
QueryDict.getlist(). If you want a native dict object, call
QueryDict.lists() and pass its result to dict().

Other Python web frameworks generally behave the same way, though their
interface may be different:
http://werkzeug.pocoo.org/docs/0.10/datastructures/#werkzeug.datastructures.MultiDict
http://bottlepy.org/docs/dev/api.html#bottle.MultiDict
http://docs.webob.org/en/latest/reference.html#query-post-variables
http://tornado.readthedocs.org/en/latest/web.html#tornado.web.RequestHandler.get_query_argument

On 2015-03-26, at 20:21 , Daniel Roseman <dan...@roseman.org.uk> wrote:
> The [] convention is not a standard part of HTTP. I believe it was developed in PHP and DHH adopted it for Rails, but that doesn't make it a standard. Django does not expect data in that format and does not do any specific conversion for it.

To be fair, URLs are not defined directly in HTTP's RFCs and urlencoded
querystrings are not part of the URL or URI RFCs either, which define
the query string as an almost arbitrary pile of stuff between ? and #
(though RFC 3986 does give URL examples using that pattern), it's part
of HTML: http://www.w3.org/TR/html5/forms.html#url-encoded-form-data

[Gabriel Pugliese]

@Daniel, oh really that Rails is not Django? Haven't noticed on my years of Python web development. Thanks!

@Masklinn

That input is from jQuery's default serializer. It sends the data with contentType 'application/x-www-form-urlencoded; charset=UTF-8'. I just need to pass a data parameter with my JS Object (JSON) and it does the magic.

I understand I can use .getlist() or any other non-optimal solution. I also understand what's going on about keys here (that's why I asked about laziness nature of QueryDict). But following DRY, KISS and other philosophies, couldn't Django do the same as other frameworks assuming it's widely adopted that way? What is the advantage of doing things so differently about that?

Why should I need to implement a lib that re-iterates over the same string that QueryDict already parses to achieve the result that is expected to everyone that is new to Django?

Anyway, thanks for the clear answer and information. And sorry about any grammar errors, English is not my first language.

[Carl Meyer]

Hi Gabriel,

On 03/27/2015 01:34 PM, Gabriel Pugliese wrote:
> @Masklinn
> That input is from jQuery's default serializer. It sends the data with
> contentType 'application/x-www-form-urlencoded; charset=UTF-8'. I just
> need to pass a data parameter with my JS Object (JSON) and it does the
> magic.

Yes, jQuery has chosen to adopt the same non-standard that Rails and PHP
use for smuggling nested lists and objects through a flat querystring.

> I understand I can use .getlist() or any other non-optimal solution.

request.GET.getlist('key') is the correct solution in Django for getting
a list of values from a querystring.

> I
> also understand what's going on about keys here (that's why I asked
> about laziness nature of QueryDict).

I don't understand what you think is lazy about a QueryDict, or how that
would be related to this issue. QueryDicts are not lazy; they parse the
querystring immediately on instantiation.

> But following DRY, KISS and other
> philosophies, couldn't Django do the same as other frameworks assuming
> it's widely adopted that way? What is the advantage of doing things so
> differently about that?

Django's philosophy here follows KISS. We give you the querystring in a
parsed format that allows you to easily get the data out of it by the
actual key names found in the querystring. We don't assume that you are
using jQuery on the client side, and we don't add any magical
non-standard features to smuggle nested objects through the querystring.
Such behavior might "do the expected thing" for some new users of
Django, at the expense of being very confusing to other new users of
Django, who understand HTTP and HTML but haven't been exposed to the
jQuery/Rails/PHP behavior.

> Why should I need to implement a lib that re-iterates over the same
> string that QueryDict already parses

You don't need to go back to the raw querystring. It would not be hard
to implement a small function that would take a QueryDict, iterate over
it and interpret it in the jQuery/Rails/PHP style, and return a
dictionary. I'd guess that someone may have already done this, though
I'm not sure what terms I'd use to search for it. If nobody has, then
you could be the first! Put it on PyPI, advertise it, and see what kind
of demand there is.

> to achieve the result that is
> expected to everyone that is new to Django?

I am a counter-example to your assertion that the Rails/PHP/jQuery
behavior is "expected to everyone that is new to Django." I had never
heard of the Rails/PHP/jQuery behavior when I was new to Django, and
Django's handling made perfect sense to me. The first time I ran across
jQuery's behavior, I was very confused.

> Anyway, thanks for the clear answer and information. And sorry about any
> grammar errors, English is not my first language.

No problem. I hope my answer was clear and informative.

Carl

[Gabriel Pugliese]

Hi Carl,

I perfectly understand what you are saying. It was very clear and informative, but do not agree with the design chosen here. Below is just an opinion and you do not have to agree with it:

My buddies have given PHP and Rails examples, but there are other frameworks from other languages that do that the same way. I mean, what's the advantage here doing differently from others?

And I don't agree it follows KISS if I need to re-iterate on the result again to get a dict from it (one clear example usage is destructuring as named function parameters).

Thanks again!

On Thursday, March 26, 2015 at 2:52:48 PM UTC-3, Gabriel Pugliese wrote:

[Carl Meyer]

Hi Gabriel,

On 03/27/2015 02:10 PM, Gabriel Pugliese wrote:
> I perfectly understand what you are saying. It was very clear and
> informative, but do not agree with the design chosen here. Below is just
> an opinion and you do not have to agree with it:

That's good :-) I probably won't continue with the design discussion
past this email, because I don't think we will reach a conclusion which
convinces the Django core team to change the design.

> My buddies have given PHP and Rails examples, but there are other
> frameworks from other languages that do that the same way.

I'm not personally aware of them, but I believe you. If this behavior
ever becomes part of an accepted RFC, I'm sure Django will implement it
in core :-)

> I mean,
> what's the advantage here doing differently from others?

I already tried to explain that: simple, predictable behavior that
exposes the underlying querystring data directly to the user, with no
munging of values which might be unexpected by some users.

Django provides the simple, predictable basic behavior. If you want this
extended special behavior, it's easy to implement it on top of what
Django provides.

> And I don't agree it follows KISS if I need to re-iterate on the result
> again to get a dict from it (one clear example usage is destructuring as
> named function parameters).

If you just want a dict, you don't need to iterate, you can just use the
.dict() method (or other options already explained earlier in this thread).

If you want a dict interpretation of the querystring *that handles keys
with brackets in their names in a special way*, then yes, you have to
implement that yourself. It wouldn't be hard to implement, and then that
code could be shared by all who want it.

Carl

[Simon Charette]

Hi Gabriel,

One thing I dislike about how PHP/Rail deal with this is the fact they expose an easy way to shoot yourself in the foot.

e.g. PHP

Your code expects $_GET['foo'] to be an array() but the querystring is missing the trailing "[]" (?foo=bar) and crash. This also open a door for subtle attack vectors, let's not forget that those implementation assumes a parameter to be a collection or not based on user submitted data.

I strongly prefer how Django forces you to explicitly declare you're expecting to retrieve a collection from a specific parameter.

Simon

[Gabriel Pugliese]

Thanks for kind answers guys.

Best Regards

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/m3U7gfhWs2g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/28c33760-d9cf-4966-a249-aa1ab607909d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen J. Butler]

On Fri, Mar 27, 2015 at 2:50 PM, Carl Meyer <ca...@oddbird.net> wrote:
> Hi Gabriel,
>
> On 03/27/2015 01:34 PM, Gabriel Pugliese wrote:
>> @Masklinn
>> That input is from jQuery's default serializer. It sends the data with
>> contentType 'application/x-www-form-urlencoded; charset=UTF-8'. I just
>> need to pass a data parameter with my JS Object (JSON) and it does the
>> magic.
>
> Yes, jQuery has chosen to adopt the same non-standard that Rails and PHP
> use for smuggling nested lists and objects through a flat querystring.

Let's be accurate here: what PHP, Rails, jQuery, et al. do is not
"non-standard". There's nothing wrong with their key-value pairs in
the query string. This is further illustrated by the fact that no
browser I am aware of has problems encoding them, and QueryDict has no
problem parsing them (it just doesn't do it with quite the same
result).

Perhaps it would be better to think about this issue as Plain Standard
and Enhanced Standard. Django does the bare basics of what is
required. PHP, Rails, jQuery, et al. go a little further in the
parsing. But there's absolutely nothing wrong with the choice they
made.

[James Bennett]

On Sat, Mar 28, 2015 at 5:37 AM, Stephen J. Butler <stephen...@gmail.com> wrote:

Let's be accurate here: what PHP, Rails, jQuery, et al. do is not
"non-standard". There's nothing wrong with their key-value pairs in
the query string. This is further illustrated by the fact that no
browser I am aware of has problems encoding them, and QueryDict has no
problem parsing them (it just doesn't do it with quite the same
result).

Well, there is no adopted standard which says that these query strings are to be interpreted identically:

foo=bar&foo=baz

foo[]=bar&foo[]=baz

(RFC 3986 specifies the syntax of a query string, but not how to interpret it)

Historical practice was that each of these would be interpreted as specifying two values for a particular key, but that the former would specify values for the key "foo" and the latter for the key "foo[]". Then a non-standard convention was adopted by some tools to interpret them identically, as specifying multiple values for the key "foo".

The main issue was jQuery adopted this by default, which in turn has a history of breaking things for users of Django.

Perhaps it would be better to think about this issue as Plain Standard
and Enhanced Standard. Django does the bare basics of what is
required. PHP, Rails, jQuery, et al. go a little further in the
parsing. But there's absolutely nothing wrong with the choice they
made.

There is a problem: the choice they made was incompatible with many pre-existing tools, and with anyone who cared about maintaining easy compatibility with pre-existing tools. Last time I raised the issue with jQuery (five years ago), however, breaking Django and other traditional query string parsing tools by default was considered a wontfix issue on grounds that it's possible to manually set a flag to get the traditional behavior.

That was the last time I bothered trying to interact with the jQuery people. Luckily, I already had good technical reasons for preferring other JS toolkits :)

[Masklinn]

On 2015-03-28, at 11:37 , Stephen J. Butler <stephen...@gmail.com> wrote:
>
> Let's be accurate here: what PHP, Rails, jQuery, et al. do is not
> "non-standard". There's nothing wrong with their key-value pairs in
> the query string.

Well there is the problem that it's really not specified (so hardly a
standard), the original behaviour is very strongly linked to PHP's
interpretation of "arrays" as a mix of array and hash table
(foo[]=1&foo[a]=1; does Rails always output Hash objects with a mix of
string and integer keys?) and it actually loses information
(foo=1&foo[]=1) (which is compounded by PHP's other "innovations" in
handling query strings, I don't know if Rails or jQuery have seen fit to
reproduce the one where space and dots in keys are replaced by
underscores)

> Perhaps it would be better to think about this issue as Plain Standard
> and Enhanced Standard.

"Enhanced" would mean it's strictly better which is not really the case.

If you want a "better standard", just use a JSON query string. That'll
work everywhere, and you'll get to pass numbers and booleans through
unmolested.

[Gabriel Pugliese]

Probably we could get most used frameworks or parser libs from most languages to just get statistics of what is being done? I'm not saying that we need to change anything if we see some "standard", it's just for science.

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/m3U7gfhWs2g/unsubscribe.

To unsubscribe from this group and all its topics, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/53DBA3D2-A8D2-4B9E-B3A2-ABD9605B5306%40masklinn.net.

[Stephen J. Butler]

For those interested, here's some code that does this in Django as
request.PARAMS:

https://github.com/sbutler/django-nestedparams

It is extremely alpha. I wouldn't use it in your code. I did it as a
proof of concept, but if other want to hack on it I'd be happy to
accept pull requests.

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.

> To post to this group, send email to django...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/2cf88340-b9c4-4369-8b19-0cb8f731dddc%40googlegroups.com.