Web scraping in a chrome extension

[methos o]

I am writing a chrome extension that downloads a page for user, parses contents and shows useful bits to user.

I first tried it using $.get(). But some of the downloaded pages (for e.g. gmail.com), did not send any content back. It showed a message that "your browser does not have javascript enabled". 

Then I tried to download the page using iframe. But even that did not work.

1. If I use iframes in content-scripts, some sites do not allow iframes at all in their pages (e.g. stackoverflow).

2. If I use iframes in background page, the page is not displayed for sites using X-frame-options (e.g. google.com).

So how can I download web pages in the background and parse their contents?

(P.S.: There is a similar question on SO that is unanswered: http://stackoverflow.com/questions/6508393/web-scraping-in-a-google-chrome-extension-javascript-chrome-apis

[Matt Kruse]

Well, there is no simple answer. The results you are seeing is what makes it difficult. Since so many sites are ajax and javascript-driven, simple loading a page via ajax will not give you the content you want, and trying to run it in an iframe will cause other issues like you've seen.

The answer really depends on exactly what kind of content you want to scrape. In some cases, you can hit the mobile version of a site and it will give you the same content but in a much more scrape-friendly format. In other cases you can find a different url that gives you a cleaner version of the data you want, in a stand-alone way. And in still other cases, there are actually web services you can call directly to get the raw data.

It all depends.

Matt Kruse