Crawling angularjs sites - SEO

[Roger Alsing]

We have put together a small service for those who want to serve static snapshots of their dyanmic content to google and other serach bots.

e.g. 

http://crawlr.wombit.se/Crawler/htmlsnapshot?url=http://angularjs.org/

This will return the generated html for angularjs.org

(see the two todo items are present there and those are angularjs generated)

You just have to wire up your backend to respond to googles _escaped_fragment_ requests .

pseudo code:

server code:

     if (querystring.containskey("_escaped_fragment_")

     {

            var page = querystring["_escaped_fragment_"];

            var url = "http://www.mysite.com/%23!/" + page;

            var html = httppost("http://crawlr.wombit.se/Crawler/htmlsnapshot?url=" + url);

            response.write(html);

            response.end();

            return;

     }

Sorry if this comes out as promotion or anything, but I build this service to scrape my own angularjs site for search engine indexing.

Maybe someone else have some use for it..

//Roger

[Jose Luis Rivas]

On 8/6/13 2:44 PM, Roger Alsing wrote:
> We have put together a small service for those who want to serve static
> snapshots of their dyanmic content to google and other serach bots.
>
> e.g.
> http://crawlr.wombit.se/Crawler/htmlsnapshot?url=http://angularjs.org/
>
> This will return the generated html for angularjs.org
> (see the two todo items are present there and those are angularjs generated)
>
> You just have to wire up your backend to respond to googles
> _escaped_fragment_ requests .
>

This is awesome. Did you built it open source? Is the code of the
service available somewhere?


--
Jose Luis Rivas
http://joseluisrivas.net/

[dev mondo]

great, could you please share source code if it is not a problem

[Roger Alsing]

Absolutely.

Its pretty simple actually.

I start up PhantomJS inside wombit crawlr and execute a small javascript that loads the page.

The result is then piped from the phantom process back to my app and returned to the client.

I think it's less than 100loc all in all.

I'll set up a repo for it when Im back at work.

[dev mondo]

cant wait, awesome

[Kirk Bushell]

What an awesome idea.... Well done!!!

[Jeff Lu]

This is great. Is the repo available?

[Karan Jassar]

Here's one I found on Github.

https://github.com/steeve/angular-seo