Gmail Calendar Documents Reader Web more »
Recently Visited Groups | Help | Sign in
Google Groups Home
Google Webmaster Help
Home
About this group
Join this group
Discussions > Crawling, indexing, and ranking > problem with Google and robots.txt
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   2 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
rch  
View profile  
(1 user)  More options Nov 17 2008, 10:14 am
From: rch
Date: Mon, 17 Nov 2008 07:14:59 -0800 (PST)
Local: Mon, Nov 17 2008 10:14 am
Subject: problem with Google and robots.txt
Forward | Print | Individual message | Report this message | Find messages by this author
I have both a regular web and a mobile site that publishes local
business listings...

I submitted sitemaps...mobile sitemap for the mobile site etc... well
Google was indexing pages from my mobile site and displaying them on
regular web searches... which is fine for the end user but not for
me...it's the same content but when a regular web user goes to a
mobile page they see no (revenue producing) advertising or AdSense
Ads...(this really sucks)

SO...I was advised (supposedly by Google) to add the following to my
robots.txt file:
{
User-agent: Googlebot-Mobile
Allow: /

User-agent: Googlebot
Disallow: /

}

Now Google is telling me there is a problem with my sitemap because of
the robots.txt file...

It really sucks now that you can "contact" Google and get a response
to a specific problem not covered (like so many problems) in their
bloody FAQ etc


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
JohnMu Google employee  
View profile  
 More options Nov 19 2008, 8:15 am
From: JohnMu
Date: Wed, 19 Nov 2008 05:15:25 -0800 (PST)
Local: Wed, Nov 19 2008 8:15 am
Subject: Re: problem with Google and robots.txt
Forward | Print | Individual message | Report this message | Find messages by this author
Hi rch and welcome to the groups!

The problem you're most likely running into is that Google grabs
Sitemap files using the normal Googlebot. According to your robots.txt
file however, you're blocking all accesses by the Googlebot, so we
can't grab your Sitemap file.

A simple solution would be to just allow the Sitemap URL, for
instance:

User-agent: Googlebot
Disallow: /
Allow: /sitemapfile.xml

In this case, the more specific entry is the one we follow, so we
would have no problem accessing http://domain.com/sitemapfile.xml
using a Googlebot.

Hope it helps!

John


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2009 Google