how to make BeautifulSoup follow the different pages of a site with tabs?

18 views
Skip to first unread message

Stephie Goune

unread,
Jun 24, 2022, 10:12:44 AM6/24/22
to beautifulsoup
Hello to the whole community of Africa. I'm working on a scraper that will scrape search results from the Ask.com search engineThis scraper works using beautifulsoup and the Django python framework. Django just takes care of sending the scraped data to BeautifulSoup and getting the scraped data back to the site. bs4 meanwhile takes care of scraping, I created a first version of my scraper. But Ask.com being a multi-page search results site like Google, I would like bs4 to be able to go to subsequent pages of results and scrape the data there. Unfortunately, the version of my scraper only allows bs4 to scrape the data of the first page sent, but I would like it to scrape the data of the different pages sent. Here is my code for the current version  if you want the complete code containing django here it is. here is what i tried to resolve the problem with this code but without success because when I run it it works exactly like the previous code i.e. it only returns the first page but does not show any errors . Being a beginner in BeautifulSoup, I hope to be able to count on the help of the members of the community, thank you in advance for the answers and have a nice day everyone.
Reply all
Reply to author
Forward
0 new messages