Creating Google-crawlable pages with Angular

[Chris Smith]

Hello,

One of the things I like most about the Angular framework is that it allows me to create purely client-side templates. My backend just focuses on providing data, and everything else I can do in JavaScript and HTML.

However, it just dawned on me that having a website where all my content is obtained via AJAX means that the site cannot be indexed by Google out of the box. My understanding, is that I need to handle special GoogleBot directives and serve static web pages at a different URL. (See http://code.google.com/web/ajaxcrawling/docs/getting-started.html.)

Since I am new to this concept, is there an easy way to work around this? E.g. automating the generation of an HTML snapshot for the Google crawler? Or, is this something that is platform dependent? (For example, I saw a solution using an HTML unit testing framework when running on AppEngine.)

Any insights on how the Angular community deals with this would be very helpful.

Thanks,

-Chris

[Daniel Nelson]

I think I saw Igor mention this link at some point:
http://code.google.com/web/ajaxcrawling/docs/getting-started.html

[Misko Hevery]

Hi Chris,

Unfortunately how you generate the static snapshot depends on the backend you use, and there you are totally on your own.

This is an issue for all AJAX technologies, not just Angular. But we think it should be possible in theory to execute angular applications on the node.js server with dom.js and so it should allow you to generate the HTML which you could then send back to the crawler. But this is very much theoretical, and to our knowledge no one has tried it yet.

--
You received this message because you are subscribed to the Google Groups "AngularJS" group.
To post to this group, send email to ang...@googlegroups.com.
To unsubscribe from this group, send email to angular+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/angular?hl=en.

[Johan Steenkamp]

There is information on hash bang #! url support in the docs here:

http://docs-next.angularjs.org/guide/dev_guide.services.$location

However as Misko pointed out the issue is really how your server generates the static representation from the #! uri when rewritten by the search engine as a query string using _escaped_fragment_

There are many articles about it, here are a couple from SEOMoz from Smashing: 

http://www.seomoz.org/blog/how-to-allow-google-to-crawl-ajax-content

http://coding.smashingmagazine.com/2011/09/27/searchable-dynamic-content-with-ajax-crawling/

[Ben Kitzelman]

If your running a ruby app and want to adhere to the google ajax crawling scheme - there's a gem that implements the crawler detection and snapshotting for any rack app....

    gem install google_ajax_crawler

writeup of how to use it is at http://thecodeabode.blogspot.com.au/2013/03/backbonejs-and-seo-google-ajax-crawling.html, source code at  https://github.com/benkitzelman/google-ajax-crawler

-Ben

[Michael Natkin]

Ben, I was all excited to use your gem, but then I realized it is hard/impossible to use capybara on heroku. Know of anyone working on any alternatives that can live on heroku? 

[Ben Kitzelman]

hmm its hard as the snapshotting process relies on a headless browser.... You could still use the gem with minimal effort, if you feel like contributing, by writing a driver using Bokor - a wrapper gem for zombiejs - a js implementation of a browser - it would probably render faster too. A driver really should only be 10 - 15 lines of code

[Michael Natkin]

Cool, thanks! I ended up going a different way, writing a rails-side controller to render a super minimal version of our recipe page when it sees _escaped_fragment_. That will get me through for now, but as our site becomes more complex (chefsteps.com),  we might need to move move to the zombiejs solution.

----------------------------------------------------------------------------------------------------------------------------------

Michael Natkin

CTO

http://ChefSteps.com

--
You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/yClOeqR5DGc/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.

To post to this group, send email to ang...@googlegroups.com.

Visit this group at http://groups.google.com/group/angular?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Joshua White]

I just implemented the solution https://prerender.io/ with Rails and Angular. It was relatively painless.

There were a few tricky parts

When you actually try to visit one of your own pages like:
http://mypage.com?_escaped_fragment_=

rails chokes because (in my case) it is single threaded. So it means a sitemap is required to completely index my site using this method.

I testing the output by pasting the test url into the prerender.io 'cached pages' page with the 'add url' button then checking out the 'raw html'. From there I could go to http://mypage.com?_escaped_fragment_= and get the expected result.

I also had to use the prerender.io feature wait for render

https://prerender.io/getting-started#wait-for-render

I added the window.prerenderReady = false to my application.js, but it could probably just go in the link function of the directive I used below.

 

https://gist.github.com/jibwa/10694880

[Antonio Fernández Porrúa]

My approach is make a frontend for google crawler that uses the same rest API, but generates a very simplified HTML versión of my web.

Google does not need any fancy UI but your relevant content, so you do not need to have snapshots nor replicates your web server side.

In few words, give to google a very simple html with the relevant content and links.