Struggling with Crawling

48 views
Skip to first unread message

objectuser

unread,
Jan 22, 2012, 12:37:41 PM1/22/12
to google-we...@googlegroups.com
I've tried to implement the approach here:


I've done two things:
  1. look for the _escaped_fragment_ query parameter and use it to generate a non-ajax view of the site
  2. added the meta tag in my index.html:
<meta name="fragment" content="!">

If I visit my site with _escaped_fragment_ as a query parameter, a snapshot is correctly delivered to my browser.

However, when I use the Google Webmaster tools to "Fetch as Googlebot", I just get the raw (and content free) index.html text: I never get the snapshot.  Maybe that's what I should expect, I don't know.  In any case, if I do a google search for some of the words on the first page, I don't get any results.

I've just added some counters to detect the Googlebot (by looking for that string in User-Agent) and the _escaped_fragment_ parameter to see if I'm getting either, but the Webmaster tools do say that my app is being crawled.

If I've just implemented it incorrectly somehow, or if there's a major step I'm missing, that would be much appreciated feedback.

Thanks!

Raphael André Bauer

unread,
Jan 22, 2012, 5:06:30 PM1/22/12
to google-we...@googlegroups.com

No - I guess that's simply an inconsistency of "fetch as googlebot".

It's the same result for our site - but our index is indexed correctly
by the real googlebot...


Best,


Raphael


--
inc: http://ars-machina.raphaelbauer.com
tech: http://ars-codia.raphaelbauer.com
web: http://raphaelbauer.com

objectuser

unread,
Jan 23, 2012, 7:54:12 AM1/23/12
to google-we...@googlegroups.com
Thanks, Raphael.  It helps to know that's not the place to look. :)

Interestingly, last night I got two visits from the Googlebot, but neither included the _escaped_fragment_ parameter (I keep the stats separate).  So I must still not have something right.  I thought the meta tag was enough there.

objectuser

unread,
Jan 26, 2012, 9:29:36 PM1/26/12
to google-we...@googlegroups.com
My app has now been crawled with the _escaped_fragment_ parameter, so that's progress (I didn't make any changes, it just took a while it seems).

It doesn't appear the app is properly indexed yet, however.  So progress, but not quite there yet.

Raphael André Bauer

unread,
Jan 30, 2012, 5:08:58 AM1/30/12
to google-we...@googlegroups.com
What do you mean by "doesn't appear the app is properly indexed yet".
What are the symptoms?

Best,

Raphael

objectuser

unread,
Jan 30, 2012, 4:35:53 PM1/30/12
to google-we...@googlegroups.com
If I search for things not on the home page, they don't show up in the search results.  So I think the home page is indexed, but I don't think that any pages linked off the home page are indexed.
Reply all
Reply to author
Forward
0 new messages