Python beautifulsoup : not scrapping for different pages of a website

35 views
Skip to first unread message

sujay....@gmail.com

unread,
Oct 14, 2017, 3:24:34 PM10/14/17
to beautifulsoup
Hello,

I am trying to get product names and prices from a website. I have written the code which gets the details from page an URL specified below 


But when i try to scrape data from different page by changing the page number (URL as given below) I still get data from Page 1 only (above URL)


Any thoughts on why this is happening??? 

import requests
import numpy as np
from bs4 import BeautifulSoup

jumbo=[]
jumboP=[]
i=1
y=0
while (i<=10):
    r  = requests.get("https://www.jumbo.ae/mobile-phones/#/?stock_check[]=in_stock&page="+ str(i) +"&per_page=18&_xhr=1")

    soup = BeautifulSoup(r.content,"lxml")
    pdata = soup.find_all("ul", {"class":"grid-view horz-list block-grid md-bl-gd-3 sm-bl-gd-3 xs-bl-gd-2 clearfix"})

    pdata1 = pdata[0].find_all("li", {"class":"clearfix"})

    loop1=len(pdata1)
    j=0
    while(j<loop1):

        pdata2 = pdata1[j].find_all("div",{"class":"variant-desc"})
        pdata3 = pdata2[0].find_all("span",{"class":"variant-title"})
        pdata4 = pdata3[0].find_all("a")

        ppdata3=pdata2[0].find_all("span",{"class":"price"})
        ppdata4=ppdata3[0].find_all("span",{"class":"variant-final-price"})
        ppdata5=ppdata4[0].find_all("span",{"class":"m-w"})
        
        jumbo1 = [pdata4[0].text,ppdata5[0].text]
        jumbo.append (jumbo1)
        j=j+1
    i=i+1
    y=y+1

y=np.array(jumboP)
print(y)
Reply all
Reply to author
Forward
0 new messages