Using javascript / Ajax in a scraper

[Darrell Smith]

Hi

I'm looking for a way of scraping a site that requires a javascript call to call the next page. 

Is it possible to fire off a javascript call within a scraper or is it protected?

cheers

D

[Zarino Zappia]

Standard scraping libraries like lxml, mechanise and nokogiri don't run javascript code.

You'll need to use something like Selenium or PhantomJS.

There have, on occasion, been people trying to use Selenium on ScraperWiki Classic. I'm not sure how they did it though.

Usually scraping javascript sites is enough of a pain that you're best spending time making absolutely sure there's no other way to get the data you're after (eg: guessing IDs that'll be put in URLs, brute-forcing URLs, sniffing AJAX calls to see if there's a pattern in what's requested)

Zarino Zappia

VP of Product, ScraperWiki

@zarino

--
You received this message because you are subscribed to the Google Groups "ScraperWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scraperwiki...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Darrell Smith]

Thanks, I was afraid that was the case. Oh well
The site seems to generate the link for the next page when it loads and so so theres no way to predict the 'next' link and I haven't found ant patterns to it....yet.
There must be a way though

Thanks for the fast reply :)

D

[Thad Guidry]

Darrell,

Sometimes for Ajax and Javascript heavy sites, it's much easier to just use a dedicated tool such as Outwit Hub (free or paid) and use it's Macro functionality or Scripts capability if you need to go that far.  The built-in HELP for Outwit Hub shows you exactly how to just fire off a next page in various scenarios, and the embedded browser engine performs the remaining magic without you having to worry about the Javascript call itself or the wiring.

Download it and git it a try in this scenario.  http://www.outwit.com 

You can always then export what you need from OutWit Hub to a CSV or Excel file and then upload that back to Scraperwiki to continue your scraping adventures, if you desire.

D

--

You received this message because you are subscribed to the Google Groups "ScraperWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scraperwiki...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
-Thad
http://www.freebase.com/view/en/thad_guidry

[Paul Tremberth]

Hi,

there's also this project from the guys behind Scrapy

https://github.com/scrapinghub/scrapyjs

I haven't tried it myself though

It apparently lets you do the downloading and javascript rendering with webkit (using python-(js)webkit)

Paul.

[Darrell Smith]

In the end I opted for the sitemap.xml page and used that as the basis of the scrape.

Theres always a way if you look for it :)