I am trying to scrape data from this page:
https://angel.co/armando-biondiWhen the page loads JS loads the Investment portion of the page. I am using Scrapy-Splash, but it does not seem to wait before the JS has loaded the page with the investment part.
Following is my start request sample. To circumvent the problem I had to use Selenium.
def start_requests(self):
url = 'https://angel.co/armando-biondi'
yield scrapy.Request(url, self.parse, meta={
'splash': {
'args': {
'wait' : 10.0,
},
'endpoint': 'render.html',
}
})
Enter code here...
Complete code is here:
Code-gistThanks.