Searching across Dataverse Repositories

63 views
Skip to first unread message

Leif Longva

unread,
Sep 1, 2015, 2:59:08 AM9/1/15
to Dataverse Users Community
At the The Dataverse Project site (http://dataverse.org/ ) there is a Search box with the lead text: Search data across Dataverse Repositories through Harvard Dataverse.

This may indicate that the search returns data sets from Dataverses anywhere, harvested by Harvard Dataverse. However, the search results obviously are from the Harvard Dataverse only. So my question is: Is there a service harvesting and indexing Dataverses (and possibly other research data repositories) from around the world? Will The Dataverse Project develop such a harvester, and create a global indexing and search service?

Leif

Jonathan Crabtree

unread,
Sep 1, 2015, 10:59:39 AM9/1/15
to dataverse...@googlegroups.com
Leif,

I do think this is a great need. It is on the work plan for upcoming releases. I know that here at Odum it is one of the things we dearly love about our Dataverse.

Jon


Jonathan Crabtree
jonc...@gmail.com


--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/72df470a-9070-4bf1-8986-2c8bbe55c3f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elizabeth Quigley

unread,
Sep 1, 2015, 11:56:49 AM9/1/15
to Dataverse Users Community

Hi,


The Harvard Dataverse does have harvested and indexed results. However, these are metadata records that were harvested in v3.6 and migrated into v4.0 as harvested items. We have not implemented OAI-PMH harvesting sets in v4.0 yet as Jon pointed out.


You can see the harvested metadata records currently in the Harvard Dataverse here:


https://dataverse.harvard.edu/dataverse/harvested


And this includes items not only from installations but also from other repositories such as ICPSR. An item that is harvested has an icon in the upper-right corner of the results card, as well the info message at the bottom, that informs the user that clicking these harvested results will direct them to the "archival source of the data".



Cheers,


Elizabeth


Elizabeth Quigley

User Experience Lead

Data Science @ IQSS

Harvard University

http://datascience.iq.harvard.edu/

equi...@iq.harvard.edu


On Tuesday, September 1, 2015 at 10:59:39 AM UTC-4, Jonathan Crabtree wrote:

Leif,

I do think this is a great need. It is on the work plan for upcoming releases. I know that here at Odum it is one of the things we dearly love about our Dataverse.

Jon


Jonathan Crabtree
jonc...@gmail.com


On Sep 1, 2015, at 2:59 AM, Leif Longva <leif....@uit.no> wrote:

At the The Dataverse Project site (http://dataverse.org/ ) there is a Search box with the lead text: Search data across Dataverse Repositories through Harvard Dataverse.

This may indicate that the search returns data sets from Dataverses anywhere, harvested by Harvard Dataverse. However, the search results obviously are from the Harvard Dataverse only. So my question is: Is there a service harvesting and indexing Dataverses (and possibly other research data repositories) from around the world? Will The Dataverse Project develop such a harvester, and create a global indexing and search service?

Leif

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Eugene Barsky

unread,
Sep 1, 2015, 12:00:28 PM9/1/15
to Dataverse Users Community
Leif:

You can use Dataverse's OAI to expose your records to other discovery systems. We run our Dataverse for four institutions here in British Columbia - http://dvn.library.ubc.ca/dvn/, and we have used OAI to send the appropriate granular records to each institution's Summon discovery engine.

Moreover, I think that I did put a ticket in the recent past for indexing our Abacus Dataverse with almost 30K records in Harvard's

Eugene



On Tuesday, 1 September 2015 07:59:39 UTC-7, Jonathan Crabtree wrote:
Leif,

I do think this is a great need. It is on the work plan for upcoming releases. I know that here at Odum it is one of the things we dearly love about our Dataverse.

Jon


Jonathan Crabtree
jonc...@gmail.com


On Sep 1, 2015, at 2:59 AM, Leif Longva <leif....@uit.no> wrote:

At the The Dataverse Project site (http://dataverse.org/ ) there is a Search box with the lead text: Search data across Dataverse Repositories through Harvard Dataverse.

This may indicate that the search returns data sets from Dataverses anywhere, harvested by Harvard Dataverse. However, the search results obviously are from the Harvard Dataverse only. So my question is: Is there a service harvesting and indexing Dataverses (and possibly other research data repositories) from around the world? Will The Dataverse Project develop such a harvester, and create a global indexing and search service?

Leif

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Philip Durbin

unread,
Sep 3, 2015, 12:56:53 PM9/3/15
to dataverse...@googlegroups.com
Interesting. I hadn't heard of Summon but I'm took a quick peek at http://www.proquest.com/products-services/The-Summon-Service.html . Yes, I see you mentioned this, Eugene, in https://help.hmdc.harvard.edu/Ticket/Display.html?id=196381 (an internal ticket). It's pretty cool that you can get OAI data into Summon.

I understand that this thread is a lot about harvesting and Liz did a great job of summarizing the current status in a different branch of this thread, but I'd like to address Leif's original question about searching from the Dataverse project site: http://dataverse.org

It would be fantastic to have a global indexing and search service (as Leif puts it) from which you could search all Dataverse installations. I'm not sure how many people here have been following the SHARE project but I find it fascinating that you can search both a DVN 3.6 installation ( http://dataverse.scholarsportal.info ) and a Dataverse 4.0 installation ( https://dataverse.harvard.edu ) from https://osf.io/share

To search a Dataverse 4.0 installation, SHARE uses the Dataverse 4 Search API: https://github.com/fabianvf/scrapi/blob/0.9.2/scrapi/harvesters/harvarddataverse.py (I helped them a bit with using the API).

I'm not exactly sure where I'm going with this but I find it all quite interesting. Should everyone running DVN or Dataverse register as a SHARE provider at https://osf.io/share/registration/ ? Maybe! Should someone build a search service exclusive to Dataverse installations? Maybe! It's fun to be in the Dataverse club! ;)

Anyway, for now, as has been observed, the Dataverse project site ( http://dataverse.org ) only searches https://dataverse.harvard.edu . We do still intend to harvest from every Dataverse installation some day. I just thought I'd touch on search at a high level. :)

Comments welcome!

Phil


On Tue, Sep 1, 2015 at 12:00 PM, Eugene Barsky <eugene...@gmail.com> wrote:
Leif:

You can use Dataverse's OAI to expose your records to other discovery systems. We run our Dataverse for four institutions here in British Columbia - http://dvn.library.ubc.ca/dvn/, and we have used OAI to send the appropriate granular records to each institution's Summon discovery engine.

Moreover, I think that I did put a ticket in the recent past for indexing our Abacus Dataverse with almost 30K records in Harvard's

Eugene



On Tuesday, 1 September 2015 07:59:39 UTC-7, Jonathan Crabtree wrote:
Leif,

I do think this is a great need. It is on the work plan for upcoming releases. I know that here at Odum it is one of the things we dearly love about our Dataverse.

Jon


Jonathan Crabtree
jonc...@gmail.com


On Sep 1, 2015, at 2:59 AM, Leif Longva <leif....@uit.no> wrote:

At the The Dataverse Project site (http://dataverse.org/ ) there is a Search box with the lead text: Search data across Dataverse Repositories through Harvard Dataverse.

This may indicate that the search returns data sets from Dataverses anywhere, harvested by Harvard Dataverse. However, the search results obviously are from the Harvard Dataverse only. So my question is: Is there a service harvesting and indexing Dataverses (and possibly other research data repositories) from around the world? Will The Dataverse Project develop such a harvester, and create a global indexing and search service?

Leif

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/e3837707-a093-4c63-86a4-05b11dcc779f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages