Gmail Calendar Documents Reader Web more »
Recently Visited Groups | Help | Sign in
Google Groups Home
Google Webmaster Help
Home
About this group
Join this group
Discussions > Crawling, indexing, and ranking > User-agent: * and Disallow: /search
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   5 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
anylee  
View profile  
 More options Jul 12 2007, 5:58 am
From: anylee
Date: Thu, 12 Jul 2007 02:58:11 -0700
Local: Thurs, Jul 12 2007 5:58 am
Subject: User-agent: * and Disallow: /search
Forward | Print | Individual message | Report this message | Find messages by this author
Dear all,
I saw this massage at my diagnostic report in about one  week and
today I see it like :
User-agent: *
Disallow: /search
Sitemap: http://anylee-anylee.blogspot.com/feeds/posts/default?orderby=updated

Any body know what is it ? ( I didn't change my robot) and how to
solve this problem ?
Can some one tell me step by step please ...

Thank's a lot


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sebastian  
View profile  
(1 user)  More options Jul 12 2007, 6:17 am
From: Sebastian
Date: Thu, 12 Jul 2007 10:17:05 -0000
Local: Thurs, Jul 12 2007 6:17 am
Subject: Re: User-agent: * and Disallow: /search
Forward | Print | Individual message | Report this message | Find messages by this author
That's Bloggers support for Ms. Googlebot. It auto-submits a feed
containing your recent updates as XML sitemap to search engines. And
no, you can't change a robots.txt at blogspot.com, because that file
does not exist, it's just a script's output. IIRC we nagged Google
since ages WRT to sitemaps and blogger, and that's a really neat
response.
Sebastian

On Jul 12, 11:58 am, anylee wrote:


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
JLH  
View profile  
 More options Jul 12 2007, 6:31 am
From: JLH
Date: Thu, 12 Jul 2007 10:31:08 -0000
Local: Thurs, Jul 12 2007 6:31 am
Subject: Re: User-agent: * and Disallow: /search
Forward | Print | Individual message | Report this message | Find messages by this author
The disallow statement stops search queries on your site from being
included in the index because without it your site could be an
infinitely large site.  For example if I search for a monkey on your
site (using the search bar in the top left) the resulting URL is
http://anylee-anylee.blogspot.com/search?q=monkey  The robots.txt just
stops that from being crawled.

On Jul 12, 5:17 am, Sebastian wrote:


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sebastian  
View profile  
(1 user)  More options Jul 12 2007, 6:32 am
From: Sebastian
Date: Thu, 12 Jul 2007 10:32:25 -0000
Local: Thurs, Jul 12 2007 6:32 am
Subject: Re: User-agent: * and Disallow: /search
Forward | Print | Individual message | Report this message | Find messages by this author
Forgot to add: Disallow: /search tells crawlers not to fetch result
pages from your blog's internal search facility. It makes no sense to
index those.
Sebastian

On Jul 12, 12:17 pm, Sebastian wrote:


    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
anylee  
View profile  
 More options Jul 12 2007, 2:16 pm
From: anylee
Date: Thu, 12 Jul 2007 11:16:28 -0700
Local: Thurs, Jul 12 2007 2:16 pm
Subject: Re: User-agent: * and Disallow: /search
Forward | Print | Individual message | Report this message | Find messages by this author
So...I don't need to do anything ?
Thank's a lot for helping me

    Forward  
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2009 Google