How to encode nested structure in URL query string for Tornado?

[mrtn]

In a GET request, I want to send a fairly customized nested dictionary structure as query string to a Tornado server, so that Tornado can decode the parameters into something like the following:

{
 'keyA': {'1': 'value1', '2': 'value2'},
 'keyB': ['blah', 'blah'],
 'keyC': {'1': ['sth', 'sth_else']}
}

I've tried this query string: keyA[1]=value1&keyA[2]=value2&keyB=blah&keyB=blah&keyC[1]=sth&keyC[1]=sth_else

Unfortunately, Tornado decodes the query string into this (self.request.query_arguments):

{
 'keyA[1]': 'value1',
 'keyA[2]': 'value2',
 'keyB': ['blah', 'blah'],
 'keyC[1]': ['sth', 'sth_else']
}

In other words, the nested dict structure is lost in decoding. So how should I encode the query string to make Tornado correctly decode it into a nested structure as above?

P.S. Sending JSON and GET requests are not compatible.

[aliane abdelouahab]

You can try this trick:

from base64 import b64encode, b64decode

a = {

 'keyA': {'1': 'value1', '2': 'value2'}, 

 'keyB': ['blah', 'blah'], 

 'keyC': {'1': ['sth', 'sth_else']}

}

b = b64encode(str(a))

b

Out[7]: 'eydrZXlDJzogeycxJzogWydzdGgnLCAnc3RoX2Vsc2UnXX0sICdrZXlCJzogWydibGFoJywgJ2JsYWgnXSwgJ2tleUEnOiB7JzEnOiAndmFsdWUxJywgJzInOiAndmFsdWUyJ319'

b64decode(b)

Out[8]: "{'keyC': {'1': ['sth', 'sth_else']}, 'keyB': ['blah', 'blah'], 'keyA': {'1': 'value1', '2': 'value2'}}"

[Ben Darnell]

The `application/x-www-form-urlencoded` format (which is used for GET url parameters) is extremely limited and cannot represent complex structures. HTTP does not require that the query portion of URLs be in this format, but since it provides for no way to negotiate the format of this data Tornado (like most web frameworks) unconditionally tries to interpret the query as form-urlencoded.

Some frameworks and libraries (including php, rails, and jquery) support a non-standard extension (or one of several similar extensions; it's unclear how compatible these encodings are across languages) to encode additional structure with square brackets in the query parameters (http://stackoverflow.com/questions/30261513/is-there-a-spec-for-nested-urlencoded-params). Tornado doesn't know anything about this but it should be straightforward to parse and reconstruct yourself given the data available in self.request.query_arguments (once you define what exactly the rules are for the format you want to use).

However, as soon as you use any of these square-bracket encodings, you're no longer compatible with plain HTML forms so there is little reason to stick with the form-encoded format. Personally I would just stuff a json-encoded string into a single form-encoded argument. It's not very human-legible but it's easier for machines to work with than an extension of the form-encoded format: `?q=%7B%22keyC%22%3A%7B%221%22%3A%5B%22sth%22%2C%22sth_else%22%5D%7D%2C%22keyB%22%3A%5B%22blah%22%2C%22blah%22%5D%2C%22keyA%22%3A%7B%221%22%3A%22value1%22%2C%222%22%3A%22value2%22%7D%7D`

Finally, a general piece of advice: Tornado's get_argument() family of methods is for use with HTML forms. If you're doing something that is not an HTML form, then instead of thinking "how do I make get_argument() work with my data", you should think "what do I use instead of get_argument()".

-Ben

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mrtn]

Thanks Ben.

Regarding the last advice, what shall we use for processing query string parameters in the case of a REST API? When you mentioned 'get_argument() family of methods', I suppose they also include get_query_argument(s) since they all call either _get_argument or _get_arguments.

[Ben Darnell]

On Sun, Jul 12, 2015 at 6:39 PM, mrtn <mrtn...@gmail.com> wrote:

Thanks Ben.

Regarding the last advice, what shall we use for processing query string parameters in the case of a REST API?

I would say the first goal should be to try to simplify your data model enough that it can fit within the constraints of a url-encoded data set (i.e. think about how you would define a no-javascript HTML form for this query). If that is impossible but you can make it work with some simple naming conventions for related parameters, then do that. But only do it if it is simple to implement by hand; if you find yourself wishing you had a library to handle more complex structure then I would go straight to JSON. Just put an entire json-encoded string in one query parameter (as I did in my previous message) and read it with `json.loads(self.get_query_argument("q"))`.

 

When you mentioned 'get_argument() family of methods', I suppose they also include get_query_argument(s) since they all call either _get_argument or _get_arguments.

Yes, get_query_argument is included in that comment.