user agent is sending with url

11 views
Skip to first unread message

Avnesh Shakya

unread,
May 21, 2013, 2:34:45 AM5/21/13
to beauti...@googlegroups.com
hi,
    I am extracting data from other site and I want to do that my user agent should be change randomly whenever i run my python script for extracting data
my python file--

from bs4 import BeautifulSoup
import urllib2
import simplejson as json
import sys
from random import choice

def ext():
    url = "https://www.example.com/"
    headers = [
               {'User-Agent' : 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)'},
               {'User-Agent' : 'Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.11'},
               {'User-Agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'},
               {'User-Agent' : 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; ko; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2'}
              ]
    headerRandom = choice(headers)
    req = urllib2.Request(url, None, headerRandom)
    page = urllib2.urlopen(req)
    data = BeautifulSoup(page.read())
    -------
    .
    .
    .
It's working fine but but when I am tracking packet and method using wireshark then it's showing default firefox user agent string, That means it's not taking my own user agent.
I want-  It should be change every time user agent randomly.

Please help me, how is it possible?

Thanks

Reply all
Reply to author
Forward
0 new messages