Using selenium to scrape my LinkedIn newsfeed

[Stephen Smyth]

hi

Anyone have experience with using Selenium on LinkedIn? i want to write some code that can run twice daily to scrape my news feed for interesting articles and links. 

I can log on to LinkedIn easily enough:

    browser = webdriver.Chrome(executable_path='c:/python/chromedriver.exe')

    browser.get('https://LinkedIn.com/uas/login')

    emailElement = browser.find_element_by_id('session_key-login')

    emailElement.send_keys("AN_EMAIL")

    passElement = browser.find_element_by_id('session_password-login')

    passElement.send_keys("A_PASSWORD")

    passElement.submit()

But am then really struggling to identify anything that i can latch onto to download text and links in the news feed. If you inspect elements in google it refers to various classes of "ember-view" for the section relating to the news feed but i am at a loss to find text or hrefs in there.

Any ideas?

Steve

[David]

Why do you want to do this with Selenium? Might want to look into whether you could scrap it using (unofficial, or not for public use) LinkedIn API calls if you can sniff out the calls the site makes to load/display the feeds. Then simply make the HTTP calls to login to your LinkedIn account and fetch the feeds. Faster and less brittle then Selenium.

But Selenium would be your last resort option when nothing else works.

On your issue, do check if you're in the right frame/iframe as well, maybe it's nested in some frame you have to switch to.