[REST] encoding character in query patterns

[Mamisoa Andriantafika]

Hi,

Using this usual REST code in controller :

def api_users():
    response.view = 'generic.json' # or 'generic.' + request.extension
    def GET(*args,**vars):
        patterns = [
            "/user[auth_user]",
            "/user/{auth_user.id}",
            "/user/{auth_user.id}/:field",
            "/user/{auth_user.first_name.contains}/{auth_user.last_name.contains}/{auth_user.dob_pid7.eq}"
            ]
        db.auth_user.created_by.readable = db.auth_user.modified_by.readable = db.auth_user.created_on.readable = db.auth_user.modified_on.readable = True
        parser = db.parse_as_rest(patterns, args, vars)
        data = parser.response
        if parser.status == 200:
            return dict(content=data)
        else:
            raise HTTP(parser.status, parser.error)

with the last pattern, if I use character with accent e.g:

/user/maït/lié

the query do not find match as it is not interpreted as:

"ma\u00eft" for first vars and
"li\u00e9" for the second vars

which is the encoding I find when I do the request /user/ma/li .

How can I correct the encoding transcription so that the request comply to the format:

/user/maï/lié

?

Mike

[Richard Vézina]

It unicode :

http://stackoverflow.com/questions/15092437/python-encoding-utf-8

What I don't understand is why to get those without the u''

>>> print u"ma\u00eft"
maït

>>> print u"ma\u00eft".encode('utf8')
maït

>>> u"ma\u00eft".encode('utf8')
'ma\xc3\xaft'

Inside web2py it should be : 'ma\xc3\xaft'

So I am not sure where the string that you mention come from these two :

"ma\u00eft" for first vars and
"li\u00e9" for the second vars

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mamisoa Andriantafika]

Just to be clearer, here is a response from the following get:

https://localhost/test/default/api_users/user/ma/li/1978-01-01




{"content": [{"birth_country_pid23": "France", "first_name":
"Ma\u00eft", "last_name": "Li\u00e9", "modified_by": null,
"id": 42, "gender_pid8": 2, "created_by": null, "birth_town_pid23":
"Paris", "ssn_pid19": "", "created_on": "2016-04-23 15:59:49",
"modified_on": "2016-04-23 15:59:49", "maiden_name_pid6":
"Li\u00e9", "idc_num": "", "dob_pid7": "1978-01-01"}]}

but if I add the search with the accent:

https://localhost/test/default/api_users/user/maï/lié/1978-01-01




{"content": []}

??

[Richard Vézina]

I don't understrand... Are you trying to generate this URL or you receive the request? Does it fails for a request you receive?

[Mamisoa Andriantafika]

The goal is to receive the right result from the query in the GET request.

As you saw, in the first query I get the result of the user that where first name, name and dob correspond to the parameters.

But when I had the accented character, I cannot get the expected same result.

From the first response, one can see that the query may have been done in UTF8. So I think that I should feed the GET request with a kind of UTF8 encoded characters, although I'm already in UTF8?

[Richard Vézina]

I think it comes from here :

https://github.com/web2py/pydal/blob/85c530cc791e2aadbbbf6302b2f58d354d9800b5/pydal/helpers/rest.py#L70

As RestParse is used here :

https://github.com/web2py/pydal/blob/85c530cc791e2aadbbbf6302b2f58d354d9800b5/pydal/base.py#L538

By parse_as_rest()

RestParse don't seems to support unicode URL yet...

You may consider open an issue here :

https://github.com/web2py/pydal/issues/new

If someone (Massimo or Giovanni) can confirmed this...

Thanks

Richard

[Richard Vézina]

@Massimo... Look here!!

Richard

[Leonel Câmara]

You need to url encode those characters. Have you tried that?

[Richard Vézina]

@Leonel, he receives unicode characters which are parsed and result with no record in case those special characters are include in the REST query... There is not really any way to url_encode has he not generate unicode URL...

Richard

On Tue, Apr 26, 2016 at 11:05 AM, Leonel Câmara <leonel...@gmail.com> wrote:

You need to url encode those characters. Have you tried that?

--

[Richard Vézina]

Also URL(url_encode=True) is default...

[Mamisoa Andriantafika]

Hi,

I did try but still not working.

[Leonel Câmara]

You cannot have unicode characters in the path of an URL that is not a valid URL. If percent encoded path components do not work then it is a bug/missing feature. If you submit an issue in github I could look at it.

[Mamisoa Andriantafika]

Thank you I will do post an issue.

[Richard Vézina]

Thanks Leonel... I learn something today :)

http://stackoverflow.com/questions/2742852/unicode-characters-in-urls

--