scraping Linkedin profiles with description and country info

88 views
Skip to first unread message

Aleksandr

unread,
Jun 22, 2020, 1:22:27 PM6/22/20
to Common Crawl
Hi there.
Is there any way for scraping Linkedin profiles with URL, title, description and country info using special keywords as site:linkedin.com/in/ AND "intern" AND "student" AND "united kingdom".
May be need to use search engine with a fresh indexing
Безымянный.jpg

Sebastian Nagel

unread,
Jun 22, 2020, 2:06:15 PM6/22/20
to common...@googlegroups.com
Hi Aleksandr,

there's practically nothing from LinkedIn in the Common Crawl archives,
simply because of:

User-agent: *
Disallow: /

in https://www.linkedin.com/robots.txt

Exceptions are
- few subdomains: about.linkedin.com, business.linkedin.com, engineering.linkedin.com
- the robots.txt captures :)

Best,
Sebastian
> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com
> <mailto:common-crawl...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-crawl/439b5caf-33ea-4bfd-825c-aeda70bc3df0o%40googlegroups.com
> <https://groups.google.com/d/msgid/common-crawl/439b5caf-33ea-4bfd-825c-aeda70bc3df0o%40googlegroups.com?utm_medium=email&utm_source=footer>.

Erik Hayton

unread,
Jun 22, 2020, 11:35:16 PM6/22/20
to Common Crawl

I built this scraper a few months ago. Happy to chat about it.

rct 23

unread,
Jul 7, 2021, 10:13:33 AMJul 7
to Common Crawl
Hi Erik, can you tell me more about the scraper you built?
Reply all
Reply to author
Forward
0 new messages