Help with creating the robots.txt file

[Aroldo Rique Júnior]

Hello everybody.

Do you know how I can access my website's robots.txt file?

We are still using Dspace 7.5 (we will be upgrading to 7.6.2 and then 8.0 soon). We found that some Meta crawlers were making our site very slow.

We are using the default robots.txt.ejs from the Dspace 7.5 frontend (src/robots.txt.ejs), but when I try to access the file in production (https://biblioteca.mpf.mp.br/robots.txt), it is not found . What is missing?

Aroldo

[Sean Carte]

Hi Aroldo

When I access that link, I get redirected here:

https://biblioteca.mpf.mp.br/repositorio/robots.txt

I think you need to look at what your proxy is doing.

Maybe you need to exclude /robots.txt from the redirection?

Sean

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dspace-tech/CACKFi%2Bf7Ei7z-qnVwXRvYEkp4vZqw%2BO%3DZDEfQu4e52GaVG9NGw%40mail.gmail.com.