improved functionality for fetch()

[Richard]

If I understand correctly, fetch() was added to web2py because at the
time GAE did not support urllib2.
Now GAE does support urllib2 which makes fetch() sort of redundant,
but we need to keep it for backward compatibility.

I found that urllib2 on GAE does not handle cookies as normal, so have
been using the function below. Would it be useful to replace fetch()
with something like this?

import urllib
import urllib2
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPCookieProcessor
()))
import Cookie
cookie = Cookie.SimpleCookie()

def download(url, data=None, user_agent='Mozilla/5.0'):
data = data if data is None else urllib.urlencode(data)
headers = {'User-agent': user_agent}
try:
from google.appengine.api import urlfetch
except ImportError:
request = urllib2.Request(url, data, headers)
html = urllib2.urlopen(request).read()
else:
headers['Cookie'] = ' '.join('%s=%s;' % (c.key, c.value) for c
in cookie.values())
method = urlfetch.GET if data is None else urlfetch.POST
while url is not None:
response = urlfetch.fetch(url=url, payload=data,
method=method, headers=headers, allow_truncated=False,
follow_redirects=False, deadline=10)
data = None # next request will be a get, so no need to
send the data again
method = urlfetch.GET
cookie.load(response.headers.get('set-cookie', '')) # load
cookies from the response
url = response.headers.get('location')
html = response.content
return html

[mdipierro]

Good points. Perhaps we should integrate your function with fetch.