Having a problem pulling data from Wikipedia?

[Trevor]

Ok, so i am trying to pull data from a Wikipedia table. here is the
code below. As you will see when you run it, i am only getting errors
in the html.

The error you will see over and over in the html code returned is "
Error: ERR_ACCESS_DENIED, errno [No Error]" Im not sure why?!

from BeautifulSoup import BeautifulSoup
import urllib


url = 'http://en.wikipedia.org/wiki/
List_of_United_States_mobile_phone_companies'
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)


P.S. i have also tried pulling the html using wget and have had
various errors... Code below.

import urllib, os
from BeautifulSoup import BeautifulSoup

url = 'http://en.wikipedia.org/wiki/
List_of_United_States_mobile_phone_companies'
soup = BeautifulSoup(str(os.system("wget
http://en.wikipedia.org/wiki/List_of_United_States_mobile_phone_companies")))

[Jim Tittsler]

On Fri, Sep 4, 2009 at 12:37, Trevor<trevoro...@gmail.com> wrote:
> Ok, so i am trying to pull data from a Wikipedia table. here is the
> code below. As you will see when you run it, i am only getting errors
> in the html.
>
> The error you will see over and over in the html code returned is "
> Error: ERR_ACCESS_DENIED, errno [No Error]" Im not sure why?!

Pulling from MediaWiki (and the Wikipedia sites in particular) you are
probably better off using the MediaWiki API. If Python is your tool
of choice, I highly recommend mwclient.
http://mwclient.sf.net/

[Jitendra Nair]

Trevor,

Normally you get  "HTTP Error 403: Forbidden" while fetching wikipedia pages using urllib2.urlopen.

Change the User-Agent string  to  say me be  Mozilla  example :

import urllib2

hdrs = { 'User-Agent': "Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11" }

#First create the Request object, using the headers argument to change the UserAgent.

req = urllib2.Request("http://en.wikipedia.org/wiki/List_of_United_States_mobile_phone_companies" , headers = hdrs)

#use the Request Object to fetch the page.

fd = urllib2.urlopen(req) 

This  works for me .

Thanks

Regards

Jitendra Nair.

[Akshay G]

And if there is unreadable response, and you are struggling that whats wrong (of which i'm sure at later stage). Then read the response its in some unreadable format. Its actually in zipped format. Read about its Content-Encoding: its unzipped.

Then unzip it and then send that response to the BS. 

If you struck and need help then let me know. 

And yes for this problem its agent problem. Python urllib2 library default agent is kindda bot and Wikipedia blocks it. Just set the agent.

import urllib2

import gzip

# this method is used by the method below

def unzip_response(self, contents):

compressedstream = StringIO.StringIO(contents)

gzipper = gzip.GzipFile(fileobj=compressedstream)   

return gzipper.read()                               

#just pass the url to this method it will unzip if needed after reading its Content-Encoding :)

def get_url(self, url):

'''get_url accepts a URL string and return the server response code, response headers, and contents of the file'''

req_headers = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13', 'Referer': 'http://python.org'}

request = urllib2.Request(url, headers=req_headers) # create a request object for the URL

opener = urllib2.build_opener() # create an opener object

response = opener.open(request) # open a connection and receive the http response headers + contents

#code = response.code

#headers = response.headers # headers object

contents = response.read() # contents of the URL (HTML, javascript, css, img, etc.)

if response.info().has_key('Content-Encoding') and (response.info()['Content-Encoding'] == 'gzip'):

##print "its zipped unzipping it..."

contents = self.unzip_response(contents)

##print "unzipped :) ..."

return str(contents)

else:

return str(contents)

If any problem, drop me a mail.

Thanks,

Akshay

Sapna Solutions, Pune