how to add robots.txt to wagtail?

[Thierry Durand]

Hi,

where to place robots.txt and do make it seen by google?

thanks & regards.

Thierry

[Matthew Westcott]

Hi Thierry,

I usually put robots.txt in my app's static folder, and configure nginx to serve it using a setting like:

location /robots.txt {
root /path/to/static;
}

Cheers
- Matt

[Jordi J.]

Alternatively you can create it under your templates folder and set it on your urls.py file:

url(r'^robots\.txt', TemplateView.as_view(template_name='robots.txt', content_type='text/plain')

[Thierry Durand]

thanks guys, i've added it in the nginx setup, same for sitemap.xml

regards.

[Matt Davies]

We're looking into this issue at the moment, our production and staging sites have a need for different setups in robot.txt.

We're looking into solutions at the moment, using different urls.py files for staging and production, etc etc.

I found this online.

http://django-robots.readthedocs.org/en/latest/

I'd much prefer to hand over the management of the robots.txt to our content editors.

Does anyone know if that app will work with current wagtail?

Notice it's dependent on South.

Any tips, greatly appreciated.