Scrapy Splash not waiting for JS to bring results

395 views
Skip to first unread message

Shafaq Maalik

unread,
Apr 24, 2016, 9:54:49 AM4/24/16
to scrapy-users
I am trying to scrape data from this page:
 
https://angel.co/armando-biondi

When the page loads JS loads the Investment portion of the page. I am using Scrapy-Splash, but it does not seem to wait before the JS has loaded the page with the investment part.
Following is my start request sample. To circumvent the problem I had to use Selenium.

def start_requests(self):

        url
= 'https://angel.co/armando-biondi'
       
yield scrapy.Request(url, self.parse, meta={
           
'splash': {
           
'args': {
           
'wait' : 10.0,
           
},
           
'endpoint': 'render.html',  
         
}
         
})
Enter code here...

Complete code is here: Code-gist

Thanks.


Reply all
Reply to author
Forward
0 new messages