Scraping complex sites with a lot of JS

65 views
Skip to first unread message

Ножкин Андрей

unread,
Apr 7, 2014, 1:19:13 PM4/7/14
to zomb...@googlegroups.com
Sorry if this is already answered somewhere - I found only pretty old stack overflow topics (2011 year).

So, we have some complex websites, with a lot of JS and AJAX events. Just imagine something like Facebook (we're not going to scrape FB, it's just an example). It's not only about opening a single page - it's about complex interactions, such as clicking on some jQuery UIs and other dynamic JS elements, sending text there, etc.

We are currently using PhantomJS and webdriver but they are really heavy - CPU and memory consumption are very high and we're looking for 'light' alternatives.

Will I be able to scrape and parse these sites?

Thanks!
Reply all
Reply to author
Forward
0 new messages