Make python-requests not percent-encode the url

[Thomas Levine]

I'm scraping a surprisingly disgusting website. One problem I've run
into is that the site does not use standard percent-encoding and that
python-requests automatically encodes the url. For example

requests.get('http://example.com/%') #loads 'http://example.com/%25'
urllib2.urlopen('http://example.com/%') #loads 'http://example.com/%'

How do I disable the encoding?

I tried adjusting requests.defaults.defaults['encode_uri'], but that
didn't seem to change it.

Thanks

Tom

[Johannes]

Hey Thomas,

To configure request's default behaviours, you want to use the config parameter, either when making the request, or instantiating the session.

So something like:
requests.get(url, config={'encode_uri': False})

Relevant docs: http://docs.python-requests.org/en/latest/user/advanced/#configuring-requests

Hope this helps!

Johannes

[Nikki McCavee]

This looks obsolete now. Any way to pull this off with a POST request in the current version?

Thanks

[Páll Hilmarsson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

See:
https://github.com/kennethreitz/requests/issues/369

On February 28, 2014 7:40:57 PM GMT, Nikki McCavee <nikki....@gmail.com> wrote:
>This looks obsolete now. Any way to pull this off with a POST request
>in
>the current version?
>
>Thanks
>
>On Tuesday, April 10, 2012 5:19:19 AM UTC+4, Johannes wrote:
>>
>> Hey Thomas,
>>
>> To configure request's default behaviours, you want to use the config
>
>> parameter, either when making the request, or instantiating the
>session.
>>
>> So something like:
>> requests.get(url, config={'encode_uri': False})
>>
>> Relevant docs:
>>
>http://docs.python-requests.org/en/latest/user/advanced/#configuring-requests
>>
>> Hope this helps!
>>
>> Johannes
>> On Apr 7, 2012 9:40 PM, "Thomas Levine"

><tho...@scraperwiki.com<javascript:>>

>> wrote:
>>
>>> I'm scraping a surprisingly disgusting website. One problem I've run
>>> into is that the site does not use standard percent-encoding and
>that
>>> python-requests automatically encodes the url. For example
>>>
>>> requests.get('http://example.com/%') #loads 'http://example.com/%25'
>>> urllib2.urlopen('http://example.com/%') #loads
>'http://example.com/%'
>>>
>>> How do I disable the encoding?
>>>
>>> I tried adjusting requests.defaults.defaults['encode_uri'], but that
>>> didn't seem to change it.
>>>
>>> Thanks
>>>
>>> Tom
>>>
>>
>

>--
>You received this message because you are subscribed to the Google
>Groups "ScraperWiki" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to scraperwiki...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.

- --
pal...@gogn.in | http://gogn.in | http://twitter.com/pallih |
https://github.com/pallih
GPG: C266 603E 9918 A38B F11D 9F9B E721 347C 45B1 04E9
-----BEGIN PGP SIGNATURE-----
Version: APG v1.0.9

iQE+BAEBCAAoBQJTEOqXIRxQ4WxsIEhpbG1hcnNzb24gPHBhbGxpaEBnb2duLmlu
PgAKCRAiGI5W+Le5j9aPCACkUAzcHnspOp77aFNiecEkjyKMeWCh2HGUy5RxdlUO
DxJmF/ysucPk+ThsKQ9PYgbHnUt1SN03KJBAC7EBdoAQ/sOB/wo4aL/3JHlM3MDz
ugQoPh3L4TYlmxav9Y1KUdqUeXH/NrUk25DCQLoVTpctBBVX7VFcBzJoaEiaBOGY
dbVixJFi/SoxWQWBYAyZnehk1gtNhgLphdM/HGpKUhW7E+1zfeJHbarzNWjVQebp
vtMdc7R88mB2/kYeyaLrAHhPnPrtFQdZy/dB2zTKLEYle6Fk0jdaZl9JAUZtfdaT
9YVZxiwh6szVLsE+XY9TxnfCXndVLkhxSyUcHekKjEjB
=z6KB
-----END PGP SIGNATURE-----