Scrapping multiple pages with beautiful soup

72 views
Skip to first unread message

Hina Imran

unread,
Mar 4, 2015, 12:25:57 PM3/4/15
to beauti...@googlegroups.com
HI,
Is there a possibility to scrape multiple pages of a website with beautiful soup? I have the following code 

import requests
from bs4 import BeautifulSoup
import urllib2,sys
import lxml.html as lh
import time


content = requests.get(url).text
soup = BeautifulSoup(content, 'html.parser')
title = soup.find_all('div', class_ = 'cathead')
print  title
date = soup.find_all('div', class_ = 'cat-meta')
print  date
text = soup.find_all('p')
print text

the url just changes to page/3 and page/4 until page/224. 
Reply all
Reply to author
Forward
0 new messages