Robots txt files

[Timothy Stowell]

Has anyone here used Robots.txt files to restrict a search engine or enable a search engine to index one's site?

If so, what sort of info did you put in the robots.txt file?

Thanks,

Tim Stowell

MNGenWeb SC

[Shone Brooks]

I can help, Tim. We can discuss it apart from the group, if helpful.

Examples:

https://migenweb.org/iosco/robots.txt

https://migenweb.org/jackson/robots.txt

https://migenweb.org/washtenaw/robots.txt

--
You received this message because you are subscribed to the Google Groups "State Coordinators" group.
To unsubscribe from this group and stop receiving emails from it, send an email to state-coord...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/state-coord/CAF49Dn30oiEHpx1UzZBuXqZZfpXS7jXP98t5UA9fczchvRNpjQ%40mail.gmail.com.

--
Shone Brooks, MIGenWeb Assistant State Coordinator and County Coordinator for Iosco, Jackson, and Washtenaw Counties

[Dale H. Cook]

On 7/5/2026 2:36 PM, Timothy Stowell wrote:

> Has anyone here used Robots.txt files to restrict a search engine or
> enable a search engine to index one's site?

I have, but I got complaints from CCs that their pages were not getting
indexed, so I disabled them.
--
Dale H. Cook, Member, NEHGS, AGS, MA Soc. of Mayflower Descendants;
Massachusetts State Coordinator for the USGenWeb® Project;
Administrator of https://plymouthcolony.net