GWT & Googlebot/SEO questions

[Daniel]

Hi,

I have 3 questions regarding GWT Apps and search engine crawlers (particularly googlebot):

1. I implemented the AJAX crawling scheme http://code.google.com/web/ajaxcrawling/docs/getting-started.html on some websites with static small HTML pages created on the server for googlebot.

However it seems to me that googlebot indexes a lot less content from websites using this scheme. Does anyone know any GWT websites using AJAX crawling scheme where a "site:" search in google delivers many and fresh results? Or any other advice and experience in this area??

2. On Sites using "#!" in their URL (like google groups here) I ALWAYS see a directory-like structure after the hashtag. Something like #!/value1/value2/... instead of #!key1=value1&key2=value2

Is there a reason for that? Especially concerning SEO and PageRank flowing on the different pages within a site?

3. Google also recommends rendering HTML snapshots of the GWT page on the server and sending it to googlebot (instead of delivering alternative "static" HTML pages). Suppose the HTML output of my GWT Application doesn't have any <a href="...."> links at all because all navigation is done by other "modern" controls etc..

Will googlebot then even be able to crawl the snapshot? Or do I have to fall back to serving static HTML pages with links to googlebot in that case? (or alternatively adding some classic link elements to my GWT code?).

Thanks for your help and ideas.

[Raphael André Bauer]

On Wed, Nov 23, 2011 at 4:59 PM, Daniel <eklip...@googlemail.com> wrote:
> Hi,
> I have 3 questions regarding GWT Apps and search engine crawlers
> (particularly googlebot):
> 1. I implemented the AJAX crawling
> scheme http://code.google.com/web/ajaxcrawling/docs/getting-started.html on
> some websites with static small HTML pages created on the server for
> googlebot.
> However it seems to me that googlebot indexes a lot less content from
> websites using this scheme. Does anyone know any GWT websites using AJAX
> crawling scheme where a "site:" search in google delivers many and fresh
> results? Or any other advice and experience in this area??

You can use Google's Webmastertools and the "Crawl as Googlebot" tool
to check out how your site looks like for Google.

Normally it should index the very same content...

> 2. On Sites using "#!" in their URL (like google groups here) I ALWAYS see a
> directory-like structure after the hashtag. Something like
> #!/value1/value2/... instead of #!key1=value1&key2=value2
> Is there a reason for that? Especially concerning SEO and PageRank flowing
> on the different pages within a site?

Sry...
No idea on that... There must be some kind of influence because the
url is part of the ranking. But I guess it's Google's alchemy.

> 3. Google also recommends rendering HTML snapshots of the GWT page on the
> server and sending it to googlebot (instead of delivering alternative
> "static" HTML pages). Suppose the HTML output of my GWT Application doesn't
> have any <a href="...."> links at all because all navigation is done by
> other "modern" controls etc..
> Will googlebot then even be able to crawl the snapshot? Or do I have to fall
> back to serving static HTML pages with links to googlebot in that case? (or
> alternatively adding some classic link elements to my GWT code?).
> Thanks for your help and ideas.

Yea. You absolutely need <a href tags... If you do something else
(aka clickhandler and such) Google will not find it... That might be
related to your question 1).

Hope that helps a bit...

Apart from that #! crawling should just work fine...

Best,

Raphael

[Daniel]

thanks for your answer.

So I'll "force" some link tags to the GWT HTML output too, even though the static HTML output has links. Just to be sure, in case google checks the javascript-version for links every now and then.

Another related question:

I noticed twitter uses 302 redirects. Any ideas why that is "good" for SEO with Ajax pages?? Or has their 302 redirect other reasons?

Example:

In the search results of google you will find: twitter.com/username

When you open that link twitter will do a 302 redirect to twitter.com/#!/username

[Raphael André Bauer]

Guess they simply changed their internal structure... in case somebody
searches for "username twitter" google will automatically point to
twitter.com/#!/username and not the "old" twitter.com/username. 302's
are read by Google and applied accordingly - but they are simply there
to help Google (and a user's browser ;) ) make sense of a changed
website structure...

Best,

Raphael

>
> --
> You received this message because you are subscribed to the Google Groups
> "Google Web Toolkit" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-web-toolkit/-/nFzEQGDHaqQJ.
> To post to this group, send email to google-we...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-web-tool...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-web-toolkit?hl=en.
>

--
inc: http://ars-machina.raphaelbauer.com
tech: http://ars-codia.raphaelbauer.com
web: http://raphaelbauer.com

[Daniel]

But why don't they do a 301 redirect then? It seems to me they want google crawl twitter.com/username and point to those pages on the index. And yet they don't want their users to see those pages, so they redirect them to twitter.com/#!/username instead.

Should I do such redirects too? I'm just curious what's the best way to make Ajax pages crawlable. Twitter is so far the only Ajax Webseite I've found which has more than a few thousand pages indexed and gets indexed frequently