Using scrapy to scrape html code generated by Javascripts

[Scott Chen]

Hi all,

I'm trying to scrape realtime data that is generated by a javascript <noscript> on a webpage. The html file does not contain the info I would like to scrape. All I can see from the source is a <noscript> tag at the place where my data is at when I use inspect element.

Therefore, I was wondering if scrapy is capable of scraping data generate by javascripts.

Thanks in advance!

Scott

[Valdir Stumm Junior]

Hey, Scott!

You can use Splash (https://splash.readthedocs.org/en/stable/), which is a JS rendering service that integrates with Scrapy through Scrapy-splash (https://github.com/scrapinghub/scrapy-splash).

There is this tutorial explaining it in details: http://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-with-splash/

Regards,

Valdir.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

--

Valdir Stumm Junior  Developer Evangelist, Scrapinghub  stummjr

We turn web content into structured data. Lead maintainers of Scrapy.