Account Options

  1. Sign in
The old Google Groups will be going away soon.
Switch to the new Google Groups.
Google Groups Home
« Groups Home
Discussions > Crawling, indexing, and ranking > Blogspot Robots.txt: Disallow: /
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  24 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
vld2czech  
View profile  
 More options Jul 22 2007, 9:53 pm
From: vld2czech
Date: Sun, 22 Jul 2007 18:53:28 -0700
Local: Sun, Jul 22 2007 9:53 pm
Subject: Blogspot Robots.txt: Disallow: /
One week ago I found on my blogspot blog (www.stockweb.blogpsot.com)
following formula:

Disallow: /

meaning that nothing is indexed and my number of visitors dropped
sharply. I really don't understand. I hasn't been against Google
guideline and work hard with my blog. I haven't received any message
from Google.

Could you please help me what shall I do or how to contact Google...

Thank you in advance


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 22 2007, 10:14 pm
From: webado
Date: Mon, 23 Jul 2007 02:14:39 -0000
Local: Sun, Jul 22 2007 10:14 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
Hmm, you sure you are on blogSPot?

Your url states  . blogPSot.

Whatever that site is, it's got nothing to to with Blogspot and
Blogger.

On Jul 22, 9:53 pm, vld2czech wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 22 2007, 10:43 pm
From: webado
Date: Mon, 23 Jul 2007 02:43:33 -0000
Local: Sun, Jul 22 2007 10:43 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
Could it be that yuo have abotu 5 other blogs all looking pretty much
the same way and with about the asme content?
I don't know, just throwing idea around.

Have you listed this blog as public or private?

On Jul 22, 10:14 pm, webado wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
djc  
View profile  
 More options Jul 23 2007, 9:25 am
From: djc
Date: Mon, 23 Jul 2007 06:25:25 -0700
Local: Mon, Jul 23 2007 9:25 am
Subject: Re: Blogspot Robots.txt: Disallow: /
http://blogpsot.com/ exists to pickup typos.

The one for the real blogspot has
User-agent: *
Disallow: /search
Disallow: /

I would double check your settings to make sure you didn't make it
private.

On Jul 22, 9:43 pm, webado wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 23 2007, 10:20 am
From: webado
Date: Mon, 23 Jul 2007 14:20:24 -0000
Local: Mon, Jul 23 2007 10:20 am
Subject: Re: Blogspot Robots.txt: Disallow: /
I figured out about the typo crap, eventually ;)

The public vs private blog is the likely candidate.

I have mine private but I publish it to my own website so the point is
moot, I handle my own robots.txt and sitemap and all that.

On Jul 23, 9:25 am, djc wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
vld2czech  
View profile  
 More options Jul 25 2007, 10:49 am
From: vld2czech
Date: Wed, 25 Jul 2007 07:49:19 -0700
Local: Wed, Jul 25 2007 10:49 am
Subject: Re: Blogspot Robots.txt: Disallow: /
I'm sorry I mistyped.

My url is www.stockweb.blogspot.com

Please any help, I like blogger as it is easy to work with. And I
don't want to tranfer to wordpress and custom domain.

Thank you in advance

On 23 Čec, 17:20, webado wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 25 2007, 10:51 am
From: webado
Date: Wed, 25 Jul 2007 07:51:44 -0700
Local: Wed, Jul 25 2007 10:51 am
Subject: Re: Blogspot Robots.txt: Disallow: /
Did you make the blog Public or is it still Private?
Check the dashboard.

On Jul 25, 10:49 am, vld2czech wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
vld2czech  
View profile  
 More options Jul 25 2007, 3:42 pm
From: vld2czech
Date: Wed, 25 Jul 2007 12:42:24 -0700
Local: Wed, Jul 25 2007 3:42 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
It is public blog.

Before it has standard robots.txt but 10 days ago changed into
"Disallow: /"

On 25 Čec, 17:51, webado wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
djc  
View profile  
 More options Jul 25 2007, 4:41 pm
From: djc
Date: Wed, 25 Jul 2007 13:41:52 -0700
Local: Wed, Jul 25 2007 4:41 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
Set it to private, close your browser....not just the tab, the entire
browser
Open your browser, now change it to public.  Check to see if it's
still "Disallow: /"
If so, I suspect you got banned....or someone has figured out a way to
hack blogspot.

On Jul 25, 2:42 pm, vld2czech wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
djc  
View profile  
 More options Jul 25 2007, 4:49 pm
From: djc
Date: Wed, 25 Jul 2007 13:49:38 -0700
Local: Wed, Jul 25 2007 4:49 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
I'm not sure these errors would get you banned, but you need to fix
them

http://www.blogger.com/rsd.g?blogID=6675237082283386719 gives a 405
error.  Your should switch your feed choice to ATOM.
http://archipelago.phrasewise.com/rsd hasn't been used in quite some
time

http://www.blogger.com/comment.g?blogID=6675237082283386719&postID=41...
returns a 400

http://www.blogger.com/comment.g?blogID=6675237082283386719&postID=77...
returns a 400

http://astore.amazon.com/stoceasteurot-20 returns a 405

and those are just a few errors I found

On Jul 25, 2:42 pm, vld2czech wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sebastian  
View profile  
 More options Jul 25 2007, 5:43 pm
From: Sebastian
Date: Wed, 25 Jul 2007 21:43:17 -0000
Local: Wed, Jul 25 2007 5:43 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
The www prefix is useless, but the robots.txt under the canonical URL
blocks spiders too. Weird.
http://sebastianx.blogspot.com/2007/07/blogger-to-rule-search-engine....
Sebastian

On Jul 25, 10:49 pm, djc wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 25 2007, 6:57 pm
From: webado
Date: Wed, 25 Jul 2007 22:57:45 -0000
Local: Wed, Jul 25 2007 6:57 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
It's all OK now, for www and non www:

User-agent: *
Disallow: /search
Sitemap: http://stockweb.blogspot.com/feeds/posts/default?orderby=updated

If you want that to actually be the sitemap, you need to edit the
template to use
Change
<link rel="alternate" type="application/atom+xml" title="StockWeb -
Atom" href="http://stockweb.blogspot.com/feeds/posts/default" />

to
<link rel="alternate" type="application/atom+xml" title="StockWeb -
Atom" href="http://stockweb.blogspot.com/atom.xml" />

This will get also changed in the robtos.txt file I believe (it
better).

You can submit http://stockweb.blogspot.com/atom.xml  as a sitemap in
Webmaster Tools.

Don't try to go for the www version, too many things only point to non
www.

On Jul 25, 5:43 pm, Sebastian wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sebastian  
View profile  
 More options Jul 25 2007, 7:32 pm
From: Sebastian
Date: Wed, 25 Jul 2007 23:32:29 -0000
Local: Wed, Jul 25 2007 7:32 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
Webado, did you do offline consulting or did that happen by itself? Do
you know why we see the standard robots.txt now all of a sudden?
Thanks
Sebastian

On Jul 26, 12:57 am, webado wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 25 2007, 8:19 pm
From: webado
Date: Thu, 26 Jul 2007 00:19:33 -0000
Local: Wed, Jul 25 2007 8:19 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
Nope. There's not an iota I can do online or offline LOL

Maybe he finally made his blog public correctly. Maybe it was cached
for a bit. Maybe the robots.txt take a while to get changed over after
the blog is made public.

Maybe we'll never find out LOL

On Jul 25, 7:32 pm, Sebastian wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 25 2007, 8:21 pm
From: webado
Date: Thu, 26 Jul 2007 00:21:01 -0000
Local: Wed, Jul 25 2007 8:21 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
He's not however changed the template  to capture the atom feed
properly so that can be submitted.

On Jul 25, 7:32 pm, Sebastian wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sebastian  
View profile  
 More options Jul 26 2007, 4:52 am
From: Sebastian
Date: Thu, 26 Jul 2007 08:52:16 -0000
Local: Thurs, Jul 26 2007 4:52 am
Subject: Re: Blogspot Robots.txt: Disallow: /
Thanks, I've updated my blog post.
Sebastian

On Jul 26, 2:19 am, webado wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
vld2czech  
View profile  
 More options Jul 26 2007, 3:20 pm
From: vld2czech
Date: Thu, 26 Jul 2007 12:20:45 -0700
Local: Thurs, Jul 26 2007 3:20 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
Hello to all,

first thank you very much for your help. Sudenly robots.txt is fine.
Nothing was changed by me...Btw. I have few question to you:

1. to webado: If I edit HTML in template tab  I don't see following
code:

<link rel="alternate" type="application/atom+xml" title="StockWeb -
Atom" href="http://stockweb.blogspot.com/feeds/posts/default" />

in order to change it as you proposed. How can I change it?

2. to Sebastian: www.business-house.net is stealing my feed. I want to
have full feed for readers. Any suggestion how to block him?

3. to Sebastian and Webado: Shall I keep these meta tags or delete
them?
<meta content='index,follow' name='GOOGLEBOT'/>
 <meta content='index,follow' name='robots'/>

And how about meta tags keywords and meta descriptions on my blog? Is
it useless ?

Thank you all

On 26 Čec, 11:52, Sebastian wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sebastian  
View profile  
 More options Jul 26 2007, 5:01 pm
From: Sebastian
Date: Thu, 26 Jul 2007 21:01:40 -0000
Local: Thurs, Jul 26 2007 5:01 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
Vlada, I've answered these questions you've posted to my blog:
http://sebastianx.blogspot.com/2007/07/blogger-to-rule-search-engine....
And BTW your case made it on Sphinn's home page.

"index, follow" respectively "all" is the default behavior. If you
don't want to block crawlers, you don't need robots or googlebot meta
tags.

Site wide description and keyword meta tags are totally and utterly
useless. You're following an outdated and unproved SEO theory about AV
theming from 1999 or so. Nowadays that certainly hurts.

Did you ask the RSS scraper to remove your contents?

Thanks
Sebastian

On Jul 26, 9:20 pm, vld2czech wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 26 2007, 5:46 pm
From: webado
Date: Thu, 26 Jul 2007 14:46:50 -0700
Local: Thurs, Jul 26 2007 5:46 pm
Subject: Re: Blogspot Robots.txt: Disallow: /
I'm sure that line is there in the template, in the head section.
See where atom is mentioned. It might not have the full url., it might
be relative.

On Jul 26, 3:20 pm, vld2czech wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Dean @ The Answer May Surprise You  
View profile  
 More options Jul 31 2007, 9:40 am
From: Dean @ The Answer May Surprise You
Date: Tue, 31 Jul 2007 13:40:47 -0000
Local: Tues, Jul 31 2007 9:40 am
Subject: Re: Blogspot Robots.txt: Disallow: /
You won't find that feed line in the template, because the new Blogger
includes that information within a "widget":

<b:include data='blog' name='all-head-content'/>

Blogger controls the information within that tag, so that if you
change your feed to redirect to Feedburner, for instance. you don't
actually have to edit the template yourself. In fact, the stockweb
site is indeed using Feedburner.

My Blogger-powered blog (http://www.surprisinger.com) is set up the
same way, except through Blogger's customized URL feature. The
robots.txt file (which I did not create, and which seems to have
appeared of its own volition recently) looks exactly the same as
stockweb's current robots.txt, and the feed listed is formatted
identically (and the link redirects to Feedburner).

Could you explain why that link won't work as a site map? Did someone
at Blogger create a network-wide robots.txt file using bad code?

On Jul 26, 5:46 pm, webado wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 31 2007, 10:57 am
From: webado
Date: Tue, 31 Jul 2007 07:57:34 -0700
Local: Tues, Jul 31 2007 10:57 am
Subject: Re: Blogspot Robots.txt: Disallow: /
It will not work as a sitemap for the very simple reason that it is
located in a lower subdirectory, but a sitemap has to be no lower than
the homepage of the site.
It's nto that it is bad code in robots.txt it's just that the feed
should be locaated in the root not way down lower.
No idea if you might have other ways to control that from the
dashboard.

In my Blogger blog I am not using any widgets and the template I used
has that line in it, in the head.

On Jul 31, 9:40 am, Dean @ The Answer May Surprise You wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 31 2007, 11:00 am
From: webado
Date: Tue, 31 Jul 2007 08:00:43 -0700
Local: Tues, Jul 31 2007 11:00 am
Subject: Re: Blogspot Robots.txt: Disallow: /
Can you not use atom instead? Check the dashboard.
Under Feeds, I use this setting:

Site Feed Filename     atom.xml

On Jul 31, 9:40 am, Dean @ The Answer May Surprise You wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Dean @ The Answer May Surprise You  
View profile  
 More options Jul 31 2007, 11:07 am
From: Dean @ The Answer May Surprise You
Date: Tue, 31 Jul 2007 15:07:55 -0000
Local: Tues, Jul 31 2007 11:07 am
Subject: Re: Blogspot Robots.txt: Disallow: /

I think that's under the basic setting. I'm using the advanced,
because I like to know how many subscribers I have, and this forces
everyone onto one Feedburner feed.

I don't understand where the robots.txt file is coming from. Isn't
that supposed to be something I control?

On Jul 31, 11:00 am, webado wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
webado  
View profile  
 More options Jul 31 2007, 11:54 am
From: webado
Date: Tue, 31 Jul 2007 08:54:07 -0700
Local: Tues, Jul 31 2007 11:54 am
Subject: Re: Blogspot Robots.txt: Disallow: /
Both basic and advanced setting for feeds allow you to change that
field.

The robots.txt is automatically created and managed by Blogger based
on your dashboard settings. You cannot manage it yourself since you
have no access to the your blogspot account as such except through
Blogger. if you were to publish yoru blog to a website then youd' be
able to do all that.

On Jul 31, 11:07 am, Dean @ The Answer May Surprise You wrote:

...

read more »


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »