Robots.txt for google indexing

[Kirill Batyuk]

Hello,

 

Could someone please share their robots.txt that only allows for Google to index their DSpace? We are on DSpace 7.2 and just noticed that none of our content is indexed by Google. We do not want to be overloaded by a large number of bots, but it is important to us to have our content discoverable in Google searches.

 

Thank you,

 

 

Kirill Batyuk

Systems Librarian

MBLWHOI Library

Data Library and Archives

Woods Hole Oceanographic Institution

508-289-2850

kba...@whoi.edu

mblwhoilibrary.org -- whoi.edu

 

[Chapman, Kimberly A - (kimberlychapman)]

Hi Kirill,

 

I can’t answer your question about general Google, but I wonder if the Google Scholar inclusion instructions will be useful?

 

https://scholar.google.com/intl/en/scholar/inclusion.html

 

There have also been several presentations on Google Scholar indexing over the years that you can find on the DSpace Lyrasis Wiki https://wiki.lyrasis.org/display/DSPACE/ - if you search Google Scholar in the upper right corner you’ll start to see the list.

 

Hope this helps.

 

Kimberly

 

Kimberly Chapman

Campus Repository Services

University of Arizona Libraries

kimberl...@arizona.edu

520-626-1910

 

 

From: dspace-c...@googlegroups.com <dspace-c...@googlegroups.com> On Behalf Of Kirill Batyuk
Sent: Friday, December 1, 2023 6:55 AM
To: dspace-c...@googlegroups.com
Subject: [EXT][dspace-community] Robots.txt for google indexing

 

External Email

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-communi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-community/DM6PR16MB35452DA6E3C3C426B25921E6C281A%40DM6PR16MB3545.namprd16.prod.outlook.com.

[DSpace Community]

Hi Kirill,

The default robots.txt should work well for Google / Google Scholar indexing purposes.  That said, it appears there were fixes to the robots.txt in the DSpace 7.5 release: https://wiki.lyrasis.org/display/DSDOC7x/Release+Notes#ReleaseNotes-7.5ReleaseNotes  

If possible, I'd highly recommend upgrading to the latest version of DSpace 7.  Some releases do fix search engine optimization (SEO) issues, and those are not always easy to backport to older releases of DSpace 7. There are also documented SEO guidelines at https://wiki.lyrasis.org/display/DSDOC7x/Search+Engine+Optimization

Tim