urlfetch problem

[alokiN]

Hi everyone,

I'm having trouble with using urlfetch. I'm trying to fetch a page
that needs authentication. The authentication is done by using an url
parameter (ie http://server/index.php?auth=authstring).

When I write the url in my browser, the page I want is shown. When I
call the fetch function with the same url, the login page is retrieved
instead of the expected page, as if the authentication parameter
wasn't recognised. I'm calling the function only with the url as the
parameter ( urlfetch.fetch(http://server/index.php?auth=authstring) ),
so the request should be a normal GET request.

Could you please tell me what am I doing wrong?

Thanks in advance.

[djidjadji]

Did you use urllib.urlencode() to construct the parameter part

url = 'http://server/index.php'
url_fields = { 'auth': authstring }
url_fields = urllib.urlencode(url_fields)
result = urlfetch.fetch(url+'?'+url_fields)

[admin go2]

You should use POST instead of GET. Since GET request would hit the google cache server.

If you must use GET request, please  try this url

http://server/index.php?auth=authstring&random=a_random_number

 

And make sure authstring is urlencoded.

 

go2.appspot.com

 

http://go2.appspot.com

[alokiN]

Ok, urlencoding solved the problem. Thanks again...

On Feb 8, 3:22 pm, admin go2 <go2apps...@gmail.com> wrote:
> You should use POST instead of GET. Since GET request would hit the google
> cache server.

> If you must use GET request, please  try this urlhttp://server/index.php?auth=authstring&random=a_random_number

>
> And make sure authstring is urlencoded.
>
> go2.appspot.com
>
> http://go2.appspot.com
>
>
>
>
>
>
>
> > Hi everyone,
>
> > I'm having trouble with using urlfetch. I'm trying to fetch a page
> > that needs authentication. The authentication is done by using an url

> > parameter (iehttp://server/index.php?auth=authstring).

[DJ]

I wanted to take this opportunity to mention I just spent a week or so
feverishly trying to diagnose a similar problem, which also came down
to the URL encoding in my requests.

I'm working on prototyping a Facebook application where the middleware
is hosted by GAE, and the backend is a third-party web service. The
flow of information looks something like this:

client PC <--> apps.facebook.com <--> myapp.appspot.com <--
mydatasource.com

Because fancy authentication is needed, I'm using urllib2 to fetch the
data from mydatasource.com. In GAE, urllib and urllib2 are built on
the native urlfetch library.

I've been finding that for many requests, the call to urlfetch throws
DownloadError: ApplicationError 2. This led me to believe, after some
searching around, that the problem was a timeout in urlfetch.
Documentation states the timeout is fixed at 5 seconds. If my services
were slower than that, I'd have very few options for fixing the
problem.

But it turned out the data source was actually quite fast - subsecond
responses to most queries. Assuming it was a timeout issue, this could
only mean that Google's cloud was taking more than four seconds to
find, connect to and download from mydatasource.com, which seemed
absurd. If nothing else, the Cloud is very good at moving information
quickly.

Today I finally discovered that the DownloadError: ApplicationError 2
had *nothing* to do with timing. Not that there was anything in this
sparsely worded and sparsely documented exception, or in its call
stack, to implicate the actual problem, which was a malformed URL. It
dawned on me what was happening when I compared some of the requests
that were and weren't working. I found a pattern:

?param1=foo&param2=bar <-- this query always succeeds.
?what=foo bar&where=bar, baz <-- this query always fails.

What kept me from seeing it was for some reason I thought the latter
was failing probabilistically, say 90% of the time, and I concluded
that the slightly more complex query was taking too long. Turns out it
failed 100% of the time and I assumed otherwise.

In hindsight the problem is very obvious. Some of my parameters were
unchecked for invalid URI encoding, and contained spaces and commas.
There's a very simple way to tell this is the problem, and alokiN and
djidjadji touched on it: ***if a GET request fails with urlfetch, but
is valid when typed in the browser***, it is likely a URL-encoding
bug. This is because browsers will silently correct your URLs, but App
Engine does not. There are other significant possibilities to
consider, but I had ruled them all out. Once I rewrote my code to
assemble requests with urllib.urlencode(), it started working
properly.

None of this is strictly App Engine doing wrong - in my experience
Python exceptions have always been kind of hard to follow. Given that
Google controls this API, though, it would be nice if urlfetch checked
for this, or if the exception made it clear that illegal characters
may be choking the system.

On Feb 8, 9:22 am, admin go2 <go2apps...@gmail.com> wrote:
> You should use POST instead of GET. Since GET request would hit the google
> cache server.

> If you must use GET request, please  try this urlhttp://server/index.php?auth=authstring&random=a_random_number

>
> And make sure authstring is urlencoded.
>
> go2.appspot.com
>
> http://go2.appspot.com
>
>
>
> > Hi everyone,
>
> > I'm having trouble with using urlfetch. I'm trying to fetch a page
> > that needs authentication. The authentication is done by using an url

> > parameter (iehttp://server/index.php?auth=authstring).