Hello,
I'm trying to scrape data from a widget. It was working for the first page but there was tons more data below that it wasn't scraping. So, next I added code to scroll to the end of the page so all the data could be scraped. Now, however
when it's finished scrolling to the end of the page, it just waits and never prints. Any idea how to get it to stop waiting and print? Eventually, I'd like to try to bring the data into excel if anyone knows how to do that too. Thanks
from selenium import webdriver
url = 'http://www.tradingview.com/screener'
driver = webdriver.Firefox()
driver.get(url)
SCROLL_PAUSE_TIME = 0.5
# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
# Scroll down to bottom
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# Wait to load page
time.sleep(SCROLL_PAUSE_TIME)
# Calculate new scroll height and compare with last scroll height
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
element = driver.find_element_by_id('js-screener-container')
print (element.text)