[Inside AdSense] I, Robots.txt

[Inside AdSense Team]

Often when troubleshooting targeting inquiries, we find that publishers have unknowingly blocked the Googlebots from crawling sections of their site, resulting in PSAs or mistargeted ads. With the newest release of Google Sitemaps, you can get a report that shows you Googlebot's view of your robots.txt file.

In addition to general status updates such as the last time Googlebot downloaded your robots.txt file, this new report allows you to test specific URLs to see if your robots.txt file is blocking them from the MediaPartners Googlebot, the Google content crawler. Plus, you can test changes to your robots.txt file against other Google crawlers (e.g., search or image) or the Robots Standard. This way you can experiment with changes to see how they would impact the crawl of your site, as well as make sure there are no errors in the file, before making changes to the file on your site. Learn more about this recent release on the Google Sitemaps Blog.

Still not sure if Sitemaps can really help you? Check out the newest Sitemaps success story showcasing fellow AdSense publisher ApartmentRatings.com.

Posted by Shaluinn - Sitemaps Product Marketing

--
Posted by Inside AdSense Team to Inside AdSense at 2/07/2006 11:39:00 AM