Getting odd behaviour when trying to setup an OAI harvesting set

[Joerg Messer]

Greetings,

I was wondering if anyone could shed some light on why I seem to be getting very odd behavior when trying to setup an OAI harvesting set.  This was driving me crazy last Friday since, according to the docs, it's quite a straight forward procedure.  I defined a simple search (country:"Canada"), tested it in a collection and then setup an OAI harvesting set.  Unfortunately, I could not get the set to return any results via the OAIHandler interface (verb=ListRecords).  The basic OAI functionality appears to work fine when tested with verb=Identify or verb=ListSets.  To compound my confusion, the OAI query using this set started working on the weekend.  This morning I tried duplicating the same set and could not get it to return any results at all!  I've check other sites (e.g. IQSS) and OAI seems to be working fine.

WORKS -> http://dvn.library.ubc.ca/dvn/OAIHandler?verb=Identify

WORKS -> http://dvn.library.ubc.ca/dvn/OAIHandler?verb=ListSets

WORKS -> http://dvn.library.ubc.ca/dvn/OAIHandler?verb=ListRecords&set=canada&metadataPrefix=oai_dc

BROKEN -> http://dvn.library.ubc.ca/dvn/OAIHandler?verb=ListRecords&set=canada2&metadataPrefix=oai_dc

Any thoughts on where we might be going wrong?  Is there some periodic indexing required before the OAI set is active?  Confused!

We're running DVN v3.6.2.

//Joerg

[Joerg Messer]

This morning the second (e.g. 'canada2') query appears to be working fine.  I'm a little surprised by the time delay between yesterday morning when I first defined the second OAI set and this morning when it actually worked.  If this was mentioned in the docs than I completely missed it.  I naively thought that OAI sets would behave identically to the collections that I used to test the queries and have their results available immediately.  Still confused but happy that this feature works. 

[Condon, Kevin]

Hi Joerg,

Sorry I did not respond sooner. OAI sets need two things to become available for harvest: 1. studies need to be exported and 2. the set membership needs to be preserved so later queries for updates to the set can be updated with deletions, updates, etc. When this was first created, we relied on a nightly export process to do this but it can be accomplished on demand using the export utilities.

So, using the utilities you can either click "Run Export" which acts like our nightly export: it exports and updates harvest studies (set membership) or narrow the scope somewhat and export a single dataverse but then you need to run update harvest studies afterwards.

We will be be revisiting this functionality and how it works in a future release.

Regards,

Kevin

---

From: dataverse...@googlegroups.com [dataverse...@googlegroups.com] on behalf of Joerg Messer [joerg....@gmail.com]
Sent: Wednesday, February 11, 2015 1:25 PM
To: dataverse...@googlegroups.com
Subject: [Dataverse-Users] Re: Getting odd behaviour when trying to setup an OAI harvesting set

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/290a9ecf-a78b-4798-80c0-4e47e3bf27c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joerg Messer]

Hi Kevin,

Thanks for your response.  I was just starting to look around for a way to trigger the export.  Works like a charm.  One more question, how would I setup a query that covers all studies in a given Dataverse?  The metadata options are very rich but I'm missing a simple way to create this query.  Having a "dataverseId" similar to "studyId" would be ideal.  In our case, we're trying to track our Research Data dataverse (http://dvn.library.ubc.ca/dvn/dv/UBC_RD). 

//Joerg

On Tuesday, 10 February 2015 12:02:42 UTC-8, Joerg Messer wrote:

[Condon, Kevin]

Use:

dvOwnerId:

and the db id of the dataverse from the vdc table as in dvOwnerId:1 , etc

---

From: dataverse...@googlegroups.com [dataverse...@googlegroups.com] on behalf of Joerg Messer [joerg....@gmail.com]

Sent: Wednesday, February 11, 2015 2:22 PM
To: dataverse...@googlegroups.com
Subject: [Dataverse-Users] Re: Getting odd behaviour when trying to setup an OAI harvesting set

[Leonhard Maylein]

Hi Kevin,

we are highly interested in an automated export functionality (and
therefore availability via OAI) when a study is released.

Leonhard Maylein

Am 11.02.2015 um 19:34 schrieb Condon, Kevin:
>
> Hi Joerg,
>
> Sorry I did not respond sooner. OAI sets need two things to become
> available for harvest: 1. studies need to be exported and 2. the set
> membership needs to be preserved so later queries for updates to the set
> can be updated with deletions, updates, etc. When this was first
> created, we relied on a nightly export process to do this but it can be
> accomplished on demand using the export utilities.
>
> So, using the utilities you can either click "Run Export" which acts
> like our nightly export: it exports and updates harvest studies (set
> membership) or narrow the scope somewhat and export a single dataverse
> but then you need to run update harvest studies afterwards.
>
> We will be be revisiting this functionality and how it works in a future
> release.
>
> Regards,
>
> Kevin
>
>

> ------------------------------------------------------------------------
> *From:* dataverse...@googlegroups.com

> [dataverse...@googlegroups.com] on behalf of Joerg Messer
> [joerg....@gmail.com]

> *Sent:* Wednesday, February 11, 2015 1:25 PM
> *To:* dataverse...@googlegroups.com
> *Subject:* [Dataverse-Users] Re: Getting odd behaviour when trying to

> <mailto:dataverse-commu...@googlegroups.com>.

> To post to this group, send email to
> dataverse...@googlegroups.com

> <mailto:dataverse...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dataverse-community/290a9ecf-a78b-4798-80c0-4e47e3bf27c4%40googlegroups.com

> <https://groups.google.com/d/msgid/dataverse-community/290a9ecf-a78b-4798-80c0-4e47e3bf27c4%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Dataverse Users Community" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dataverse-commu...@googlegroups.com

> <mailto:dataverse-commu...@googlegroups.com>.

> To post to this group, send email to
> dataverse...@googlegroups.com

> <mailto:dataverse...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/dataverse-community/8552BDDD3DDE294C995A0182F32F976F0168829B30%40HARVANDMBX02.fasmail.priv
> <https://groups.google.com/d/msgid/dataverse-community/8552BDDD3DDE294C995A0182F32F976F0168829B30%40HARVANDMBX02.fasmail.priv?utm_medium=email&utm_source=footer>.