Sitemaps files using the X-Robots-Tag HTTP header
230 views
Skip to first unread message
Michael Lissner
May 20, 2012, 2:24:50 AM5/20/12
to django-d...@googlegroups.com
I just created ticket 18351, which is to add the X-Robots-Tag HTTP header to our default sitemaps.
The reason for this is somewhat complicated, and I've explained it in the ticket, but basically, the goal is to make it so sitemaps don't appear in search results anymore (though of course, you want their urls to continue appearing). Currently we don't set any special headers on our sitemaps, and as a result, you can find sitemaps within search results.
If you so desire, you can prove this with clever queries like: [ sitemap.xml site:yourdjangosite.com ]
I've put a patch on the bug that adds:
response['X-Robots-Tag'] = 'noindex, noodp, noarchive, noimageindex'
To all sitemap files. I haven't added this to the documentation, since it seems like an implementation detail. I've run the tests, and I hope I'm good to have the patch landed.
Would appreciate any feedback though, since it's my first patch to Django.
Mike
The reason for this is somewhat complicated, and I've explained it in the ticket, but basically, the goal is to make it so sitemaps don't appear in search results anymore (though of course, you want their urls to continue appearing). Currently we don't set any special headers on our sitemaps, and as a result, you can find sitemaps within search results.
If you so desire, you can prove this with clever queries like: [ sitemap.xml site:yourdjangosite.com ]
I've put a patch on the bug that adds:
response['X-Robots-Tag'] = 'noindex, noodp, noarchive, noimageindex'
To all sitemap files. I haven't added this to the documentation, since it seems like an implementation detail. I've run the tests, and I hope I'm good to have the patch landed.
Would appreciate any feedback though, since it's my first patch to Django.
Mike
Reply all
Reply to author
Forward
0 new messages