DVN records and Google search queries

[Thu-Mai Christian]

Our clients often wonder how well Google queries return DVN records in their results.  Does the DVN have any special provisions for search engine optimization?

[Condon, Kevin]

Hi Thu-Mai,

We do open our site for crawling by Google but do not yet implement any optimization strategies. We are looking at this area more closely as we have also received inquiries, specifically around author name.

Is there something in particular you are wondering about or would like to see?

Regards, 

Kevin

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/5a9a718c-cce0-48cf-b4df-8e76a501902d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Amber Leahey]

Hello,

I'd like to revive this thread if possible, but if there is another place for it let me know. 

We've done some tests and generally findability in Google is an issue. One of our members has dug into the google indexing of DV data and it seems very poor. We would like to improve this if possible, is it something we can customize on our end, or a code fix on yours?

Harpinder on our systems team found the possible source of the poor findability:

"We have following in robots.txt:

 

User-agent: *

Disallow: /

User-agent: Googlebot

Disallow: /

 

Not very Google friendly, I would say.

May be that was done for a reason or this might be default option because Harvard has similar robots file.

Any thoughts to help us out?

Thanks in advance, 

Amber 

[Philip Durbin]

Huh. I'm afraid I'm not up to date on why the Harvard Dataverse is blocking the Googlebot at the moment. Seems strange. I left a note to myself to bring this up at our next meeting.

The most recent discussion around indexing by Google is probably https://github.com/IQSS/dataverse/issues/2717 but it has a slant toward Google *Scholar*. It does mention sitemaps, however, as something that could help: http://www.sitemaps.org/protocol.html

I'd say if there's something specific in the code or docs you'd like to see, a new GitHub issue would be appropriate to capture the idea. We can always close it as a duplicate if need be.

Phil

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/489cf5e5-b39a-4789-877d-da943ce6478d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Philip Durbin]

Ok, I just brought this up with the Dataverse development team. Here's my take:

We stopped letting Google crawl the Harvard Dataverse back when we were having some stability issues. Resolving these stability issues was a focus of 4.2.2 and here's where we posted some benchmark numbers of the improvement over 4.2.1: https://github.com/IQSS/dataverse/issues/2777#issuecomment-162050545

We're waiting to let the Google bot back in until we're pretty sure we're out of the woods, stability-wise. Also, we would like to research the sort of special provisions for search engine optimization that Thu-Mai mentioned at the start of this thread. I guess for now I'd keep an eye on https://github.com/IQSS/dataverse/issues/2717 for any updates in this area, even though that issue is more about Google Scholar specifically.

I hope this helps!

Phil