Python-requests seems to 404 with Django/Tasty-pie?

[Victor Hooi]

Hi,

I have a Django app that's serving up a RESTful API using tasty-pie.

I'm using Django's development runserver to test.

When I access it via a browser it works fine, and using Curl also works fine:

curl "http://localhost:8000/api/v1/host/?name__regex=&format=json"

On the console with runserver, I see:

[02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=&format=json HTTP/1.1" 200 2845

However, when I try to use the Python requests module (http://docs.python-requests.org/en/latest/), I get a 404:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>

or:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&amp;format=json')
>>> r
<Response [404]>

or: 

>>> payload = { 'format': 'json'}
>>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>>> r
<Response [404]>
>>> r.url
u'http://localhost:8000/api/v1?format=json'

Also, on the Django runserver console, I see:

[02/Oct/2012 17:25:01] "GET http://localhost:8000/api/v1/host/?name__regex=&format=json HTTP/1.1" 404 161072

For some reason, when I use requests, runserver prints out the whole request URL, including localhost - but when I use the browser, or curl, it only prints out the part *after* the hostname:port

I'm assuming this is something to do with the encoding, user-agent or request type it's sending? Why does runserver print out different URLs for requests versus browser/curl?

Cheers, 

Victor

[Cal Leeming [Simplicity Media Ltd]]

Hi Victor,

I've had my fair share of exposure with python requests - so thought I'd chime in.

On first glance, this looks to be an issue with specifying the port number into python-requests, doing so seems to send the entire "http://localhost:8000/api/v1/host/?name__regex=&format=json" as the request. However, further analysis shows that might not be the case.

Looking at the python requests code;

https://github.com/kennethreitz/requests/blob/develop/requests/models.py

>>> urlparse.urlparse("http://localhost:8080/test/url?with=params")

ParseResult(scheme='http', netloc='localhost:8080', path='/test/url', params='', query='with=params', fragment='')

It then sends this directly into urllib3 using connection_from_url();

https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py

This then calls the following;

    scheme, host, port = get_host(url)

    if scheme == 'https':

        return HTTPSConnectionPool(host, port=port, **kw)

    else:

        return HTTPConnectionPool(host, port=port, **kw)

get_host -> parse_url()

https://github.com/shazow/urllib3/blob/master/urllib3/util.py

Tracing through urllib3 finally gets to parse_url();

>>> urllib3.util.parse_url("http://localhost:8080/test/url?with=params")

Url(scheme='http', auth=None, host='localhost', port=8080, path='/test/url', query='with=params', fragment=None)

So, lets look at path_url() instead;

https://github.com/kennethreitz/requests/blob/develop/requests/models.py

>>> lol = requests.get("http://localhost:8000/api/v1/host/?name__regex=&format=json")

>>> lol.request.path_url

'/api/v1/host/?name__regex=&format=json'

Performing a test connection shows;

 fo...@test01.internal [~] > nc -p8000 -l

GET /api/v1/host/?name__regex=&format=json HTTP/1.1

Host: localhost:8000

Accept-Encoding: identity, deflate, compress, gzip

Accept: */*

User-Agent: python-requests/0.11.1

So, from what I can tell, python requests is functioning normally.

Personally, I'd say get wireshark running, or use the nc trick shown above, perform 1 request using curl and 1 using python requests, then compare the request headers.

Can't throw much more time at this, but hope the above helps

Cal

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/ycLjP71ciAEJ.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Victor Hooi]

heya,

Thanks for the tips - you're probably right, I might need to whip out wireshark or something and see what exactly is going on.

However, one thing I did notice - I normally have the http_proxy environment variable set, as we use a HTTP proxy at work.

However, if I unset the http_proxy variable, Python requests suddenly seems to start working again.

I tried to set the no_proxy variable, and put in localhost and 127.0.0.1 in there - however, Python requests doesn't seem to respect that?

Cheers,

Victor

[Cal Leeming [Simplicity Media Ltd]]

Ahh!

By sheer coincidence, it was actually me that integrated the latest SOCKS/proxy support you see in urllib3 lol.

Basically, the current proxy handling is quite badly broken and required a bunch of fixes.

I submitted a stable set of patches for integration here;

https://github.com/shazow/urllib3/pull/68

This was later merged and branched by the core repo maintainer on the following branch;

https://github.com/shazow/urllib3/tree/socks-proxy

Sadly, the project we were working on that required proxy access was finished, and I wasn't able to contribute any more time.

There are a few ways to hack this into working.. most of them involve monkey-patching the hell out of urllib3 ;)

You could maybe try patching up 'path_url', with something really hacky like str.replace("http://localhost:8080", "") (don't shoot me!)

Let me know how you get on

Cal

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/AebVvlYTg5EJ.

[Alex]

I just discovered that the requests library honors HTTP_PROXY, but does not honor NO_PROXY. 

Based on what you say below, I'm betting this is your problem, or at least a part of it.

One solution is for you to explicitly unset HTTP_PROXY when you don't want it.

Another is for you to specify a proxies dict for requests.

Good luck!

Best

Alex