Crawlable AJAX - jQuery Load

[Dusty]

I've been reading the "Making AJAX Applications Crawlable" specs
(http://code.google.com/intl/sv-SE/web/ajaxcrawling/docs/getting-
started.html) and I'm at a loss at how to apply this to my
application.

I use jQuery to load an HTML page fragment into a DOM element on a
mostly pure HTML page. It takes a few seconds for the 'results' page
fragment to load from my server, this is why I use AJAX to dynamically
load that content into the static page. Meanwhile the static page
shows a 'throbber' and loading message. How exactly do I apply these
specs in this scenario? I don't use the hash mark in my URLs.

The page in question can be seen here: http://friendorfollow.com/dustyreagan/following/

Any pointers or recommendations on how to make the results crawlable?

--
You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group.
To post to this group, send email to google-we...@googlegroups.com.
To unsubscribe from this group, send email to google-web-tool...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-web-toolkit?hl=en.

[Katharina Probst]

You might use <meta name="fragment" content="!"> in your html page, and when the crawler requests the corresponding _escaped_fragment_= URL, you'll have a version without jQuery, where you load the results page fragment into the static HTML and return that.  Depending on what your server looks like, it first has to recognize that it is getting a request for such a URL, and then it will have to produce the results page fragment and put it together with the static HTML - so everything happens on your server before you return a result.

As long as the resulting DOM is the same as what the user would get, it should be ok.

kathrin

[Dusty]

So, if I understand you right, it should work like this?:

1) I put <meta name="fragment" content="!"> in my static page (ie my
example: http://friendorfollow.com/dustyreagan/following/)
2) Google will see it, then request http://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=
3) When my webserver sees _escaped_fragment_ = null, I return my page
with the results HTML fragment inline instead of through jQuery load.
4) Google associates that HTML to http://friendorfollow.com/dustyreagan/following/
in it's index.

Sound correct?

On Apr 19, 11:02 am, Katharina Probst <kpro...@google.com> wrote:
> You might use <meta name="fragment" content="!"> in your html page, and when
> the crawler requests the corresponding _escaped_fragment_= URL, you'll have
> a version without jQuery, where you load the results page fragment into the
> static HTML and return that.  Depending on what your server looks like, it
> first has to recognize that it is getting a request for such a URL, and then
> it will have to produce the results page fragment and put it together with
> the static HTML - so everything happens on your server before you return a
> result.
>
> As long as the resulting DOM is the same as what the user would get, it
> should be ok.
>
> kathrin
>
>
>
>
>
> On Mon, Apr 19, 2010 at 2:15 AM, Dusty <dustyrea...@gmail.com> wrote:
> > I've been reading the "Making AJAX Applications Crawlable" specs
> > (http://code.google.com/intl/sv-SE/web/ajaxcrawling/docs/getting-

> > started.html<http://code.google.com/intl/sv-SE/web/ajaxcrawling/docs/getting-%0Ast...>)

> > and I'm at a loss at how to apply this to my
> > application.
>
> > I use jQuery to load an HTML page fragment into a DOM element on a
> > mostly pure HTML page. It takes a few seconds for the 'results' page
> > fragment to load from my server, this is why I use AJAX to dynamically
> > load that content into the static page. Meanwhile the static page
> > shows a 'throbber' and loading message. How exactly do I apply these
> > specs in this scenario? I don't use the hash mark in my URLs.
>
> > The page in question can be seen here:
> >http://friendorfollow.com/dustyreagan/following/
>
> > Any pointers or recommendations on how to make the results crawlable?
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google Web Toolkit" group.
> > To post to this group, send email to google-we...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-web-tool...@googlegroups.com<google-web-toolkit%2Bunsubs cr...@googlegroups.com>
> > .

> > For more options, visit this group at
> >http://groups.google.com/group/google-web-toolkit?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group.
> To post to this group, send email to google-we...@googlegroups.com.
> To unsubscribe from this group, send email to google-web-tool...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/google-web-toolkit?hl=en.

[Katharina Probst]

Yes, that's right!

kathrin

[Dusty]

Thanks so much for your help Kathrin! :)

However, I've tried these steps and Googlebot is still giving me no
love.

If you go to http://friendorfollow.com/dustyreagan/following/ you can
see I've placed the meta tag in the header.

Then if you go to http://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=
you can see the results fragment is embedded inline in the static
HTML.

However, if I try "Fetch as Googlebot" (http://www.google.com/support/
webmasters/bin/answer.py?hl=en&answer=158587) it shows the original
pages HTML, not the _escaped_fragment_ version.

Any idea what I might have done wrong?

> > > > google-web-tool...@googlegroups.com<google-web-toolkit%2Bunsubs cr...@googlegroups.com><google-web-toolkit%2Bunsubs

[Katharina Probst]

Hi Dusty,

what URL did you enter in "Fetch as Googlebot"?  It should work for http://friendorfollow.com/dustyreagan/following/?_escaped_fragment_= (or for http://friendorfollow.com/dustyreagan/following/#!, but that doesn't really apply to your site, because you're not using hash fragments).  If you as it to fetch http://friendorfollow.com/dustyreagan/following/, it should give you just the HTML, because that's what the crawler will see (and it will then extract the meta tag, construct the _escaped_fragment_ URL and fetch that).

Does that do the trick? I hope so.:)

kathrin

[Dusty]

I asked it to fetch http://friendorfollow.com/dustyreagan/following/

If I ask it to fetch either:
http://friendorfollow.com/dustyreagan/following/#!
http://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=

It works correctly.

Shouldn't it also work for the plain URL? http://friendorfollow.com/dustyreagan/following/

When I try the plain URL it doesn't show the whole indexable HTML.

Thanks again! :)

On Apr 19, 6:40 pm, Katharina Probst <kpro...@google.com> wrote:
> Hi Dusty,
>
> what URL did you enter in "Fetch as Googlebot"?  It should work forhttp://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=(or

> forhttp://friendorfollow.com/dustyreagan/following/<http://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=>#!,

> but that doesn't really apply to your site, because you're not using hash

> fragments).  If you as it to fetchhttp://friendorfollow.com/dustyreagan/following<http://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=>/,

> it should give you just the HTML, because that's what the crawler will see
> (and it will then extract the meta tag, construct the _escaped_fragment_ URL
> and fetch that).
>
> Does that do the trick? I hope so.:)
>
> kathrin
>
>
>
>
>
> On Mon, Apr 19, 2010 at 7:32 PM, Dusty <dustyrea...@gmail.com> wrote:
> > Thanks so much for your help Kathrin! :)
>
> > However, I've tried these steps and Googlebot is still giving me no
> > love.
>

> > If you go tohttp://friendorfollow.com/dustyreagan/following/you can

> > > > google-web-tool...@googlegroups.com<google-web-toolkit%2Bunsubs cr...@googlegroups.com><google-web-toolkit%2Bunsubs

[RPB]

Hi Dusty,

I noticed the same thing when testing the Ajax crawlability for my
website:

Googlebot didn't return me the static page if I run 'fetch as
googlebot' for http://www.bookwhack.com/us/
But it works fine for http://www.bookwhack.com/us/?_escaped_fragment_=

However, I can see in webmaster tools that my pages are now being
crawled. So I suspect that the 'Fetch as Googlebot' tool takes the
request at face value and does not process the <meta name="fragment"
content="!"> but the actual crawler notices the fragment and ensures
the bot also crawls the escaped fragment . So I believe that as long
as you are seeing the correct page when you fetch as googlebot with
the ?_escaped_fragment_= you should be ok.

Just sharing my experience, happy to be corrected if any of the above
is incorrect.

-R
www.bookwhack.com

On Apr 20, 12:57 am, Dusty <dustyrea...@gmail.com> wrote:
> I asked it to fetchhttp://friendorfollow.com/dustyreagan/following/
>
> If I ask it to fetch either:http://friendorfollow.com/dustyreagan/following/#!http://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=
>
> It works correctly.
>
> Shouldn't it also work for the plain URL?http://friendorfollow.com/dustyreagan/following/

[Katharina Probst]

Hi guys,

I see your point - it's a bit confusing that Fetch as Googlebot only "works" with the _escaped_fragment_, but arguably it's also "more correct" this way (plus it lets you see the difference between the two versions).  I will pass along the feedback.  Meanwhile, I'm glad it seems to have worked out for both of you.

kathrin

[Dusty]

Brilliant! :)

I trust it's working. I'll give it some time and check Google's cache
on my pages in a few weeks.

Thanks Kathrin and R for your help! I really appreciate it! :)

On Apr 20, 7:42 am, Katharina Probst <kpro...@google.com> wrote:
> Hi guys,
>
> I see your point - it's a bit confusing that Fetch as Googlebot only "works"
> with the _escaped_fragment_, but arguably it's also "more correct" this way
> (plus it lets you see the difference between the two versions).  I will pass
> along the feedback.  Meanwhile, I'm glad it seems to have worked out for
> both of you.
>
> kathrin
>
>
>
> On Tue, Apr 20, 2010 at 5:58 AM, RPB <robbol...@gmail.com> wrote:
> > Hi Dusty,
>
> > I noticed the same thing when testing the Ajax crawlability for my
> > website:
>
> > Googlebot didn't return me the static page if I run 'fetch as

> > googlebot' forhttp://www.bookwhack.com/us/
> > But it works fine forhttp://www.bookwhack.com/us/?_escaped_fragment_=

>
> > However, I can see in webmaster tools that my pages are now being
> > crawled. So I suspect that the 'Fetch as Googlebot' tool takes the
> > request at face value and does not process the <meta name="fragment"
> > content="!"> but the actual crawler notices the fragment and ensures
> > the bot also crawls the escaped fragment . So I believe that as long
> > as you are seeing the correct page when you fetch as googlebot with
> > the ?_escaped_fragment_= you should be ok.
>
> > Just sharing my experience, happy to be corrected if any of the above
> > is incorrect.
>
> > -R
> >www.bookwhack.com
>
> > On Apr 20, 12:57 am, Dusty <dustyrea...@gmail.com> wrote:
> > > I asked it to fetchhttp://friendorfollow.com/dustyreagan/following/
>
> > > If I ask it to fetch either:

> >http://friendorfollow.com/dustyreagan/following/#!http://friendorfoll...<http://friendorfollow.com/dustyreagan/following/#%21http://friendorfo...>

>
> > > It works correctly.
>
> > > Shouldn't it also work for the plain URL?
> >http://friendorfollow.com/dustyreagan/following/
>
> > > When I try the plain URL it doesn't show the whole indexable HTML.
>
> > > Thanks again! :)
>
> > > On Apr 19, 6:40 pm, Katharina Probst <kpro...@google.com> wrote:
>
> > > > Hi Dusty,
>
> > > > what URL did you enter in "Fetch as Googlebot"?  It should work

> > forhttp://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=(or<http://friendorfollow.com/dustyreagan/following/?_escaped_fragment_=%...>

> > > > > To post to this group,...
>
> read more »