Joomla 3.3.3 & robots.txt

[Hils]

I am the end of a new Joomla site, before any SEO work, except for .htaccess. I am not asking for assistance, just observing. I have just received an email from Google Webmaster Tools which concerns me because, as an almost vanilla site, I would expect the same would be the default for all new Joomla site installs.

Googlebot can't access your site

Over the last 24 hours, Googlebot encountered 6 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 60.0%.

This site was installed by the host and it is my first 3 site with them but I was surprised to see two 'robots' files - one named robots.txt dated December 17th 2013 and one named robots.txt.dist dated July 25th 2014.

This is where Google is seeing the problems (they linked me to the robots.txt file in the email):

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Crawl-delay: 10

This is what the .dist file contains:
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

Currently, in my install, images are disallowed which I can see has been corrected in the .dist file as per
 http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEdit&tracker_item_id=32517&start=75

I am posting this here to check that the newer file will become the default rather than a .dist, or have I got a weird installation! Shouldn't the .dist be default? 
This post is not asking for help with this but from concern that this would confuse new inexperienced users. I haven't identified what
the rest of the 60% errors are yet but the old Disallow/images issue stuck out!

Sys info

PHP Built On	Linux serv01.ams17.siteground.eu 2.6.32.59-sg3 #9 SMP Wed Sep 26 03:29:25 CDT 2012 x86_64
Database Version	5.5.35-33.0-log
Database Collation	utf8_general_ci
PHP Version	5.3.29
Web Server	Apache
WebServer to PHP Interface	cgi-fcgi
Joomla! Version	Joomla! 3.3.3 Stable [ Ember ] 25-July-2014 13:00 GMT
Joomla! Platform Version	Joomla Platform 13.1.0 Stable [ Curiosity ] 24-Apr-2013 00:00 GMT

[Hils]

Having just downloaded and extracted Joomla 3.3.3 again, I confirm that the robots.txt file has been removed. There is now only the robots.txt.dst. Is this right or should the robots.txt.dist be renamed as robots.txt? (Maybe I missed something that happens during install?)

Thanks for checking

Hils

[Tom Hutchison]

Hi Hils

The .dist keeps the robots.txt file from being overwritten when updating a site. If someone puts in additional Disallow directories, files or other unique syntax to their site, they won't have to worry about their customized robots.txt file being overwritten.

Having to ask again myself, thanks Michael, the Joomla installer app will rewrite the robots.txt.dist to robots.txt only on the initial install of Joomla so their is a live and readable robots.txt in the installation. Yes, the .dist is normal in a new package for installation.

Take care

Tom

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

--

----

Tom Hutchison

Joomla! Production Leadership Team

[Hils]

Thanks for that clear reply Tom! Let me know if it would help if I added that to docs, if it would help.

Hils :)

[brian teeman]

For reference see http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEdit&tracker_item_id=29677

[Hils]

Thanks for that link Brian. I had both files because I was 'in between' the change and did an upgrade. The .dist file arrived during the upgrade but the robots.txt was already installed! This seems to just have been a co-incidence.

Hils