Changes to OAI implementation to support additional sets

[Mark Triggs]

Hi everyone,

I'm doing some work to help a customer get their AtoM database
harvested by the Trove service of the National Library of Australia
(http://trove.nla.gov.au/). From the research I've done so far, it
appears that I'll need to extend AtoM's OAI implementation a little,
so I thought I'd get in touch sooner rather than later.

Our goal here is to allow Trove's OAI harvester to harvest the records
from my customer's AtoM installation, but only the collection-level
records (that is, just the top-level information objects). The reason
for this is that item-level record records don't make much sense once
flattened and presented in isolation, so from my customer's
perspective it would be best if they weren't harvested at all.

Trove's OAI harvester can restrict its harvest to an OAI set, so that
seems like a natural fit here: if we had an OAI set that returned only
top-level information objects, we could point them at that and then
everything would be great.

My understanding of AtoM's current OAI implementation is that it
exposes one OAI set per collection. My thought is to preserve this
behavior, but to extend the implementation to allow other sets to be
defined (in code) as well. Perhaps this new set type could be given a
prefix to distinguish it from the existing collection sets--something
like "virtual:collection_records" for example--and I could add a
configuration option to allow people to selectively enable these new
OAI sets (so that existing harvesters that assume one set == one
collection wouldn't be surprised when the new OAI sets suddenly
appeared).

I've had a bit of a dig through the code, and I think I can see a way
to implementing what I'm suggesting. I think the changes I'm
considering would also fix
https://projects.artefactual.com/issues/6436 as a happy side-effect.

I'm happy to make a start and to send a pull request for discussion
if this all sounds OK. I just thought I'd check in first to see if I
could benefit from everyone's collective wisdom :) Any feedback would
be greatly appreciated!

Thanks, and apologies for the essay,

Mark

--
Mark Triggs
<ma...@teaspoon-consulting.com>
http://teaspoon-consulting.com/

[mi...@artefactual.com]

Hi Mark,

That sounds interesting and would make things a lot more flexible... definitely submit a pull request and we'll check out your solution!

Cheers,

Mike Cantelon

Artefactual Systems

[Mark Triggs]

Will do! Thanks Mike.

Mark

mi...@artefactual.com writes:

> Hi Mark,
>
> That sounds interesting and would make things a lot more flexible...
> definitely submit a pull request and we'll check out your solution!
>
> Cheers,
> Mike Cantelon
> Artefactual Systems
>
> On Wednesday, March 18, 2015 at 5:04:50 PM UTC-7, Mark Triggs wrote:
>>
>> Hi everyone,
>>
>> I'm doing some work to help a customer get their AtoM database
>> harvested by the Trove service of the National Library of Australia
>> (http://trove.nla.gov.au/). From the research I've done so far, it
>> appears that I'll need to extend AtoM's OAI implementation a little,
>> so I thought I'd get in touch sooner rather than later.

[Dan Gillean]

Hi Mark,

You may want to keep an eye on a related thread, and check out the two new issues I have filed as a result:

• https://groups.google.com/d/msg/ica-atom-users/6BmOLdLEkPs/CqGjTliPSO8J

The resulting issue tickets can be found here:

• #8158, to improve the way we implement resumptionTokens in AtoM
• #8159, to improve the way that oai_dc is implemented in AtoM

The first ticket may especially be of importance to you to ensure that Trove could resume a long request - we haven't had the opportunity so far to test AtoM's OAI repository module against an OAI Harvester, but based on the comments in the other User Forum thread, this might be worth investigating along the way.

I've added some detailed notes and examples in the issue tickets to try to make life a little easier for anyone willing to take this on :)

Cheers,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

http://teaspoon-consulting.com/

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at http://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/87egodiquw.fsf%40teaspoon-consulting.com.

For more options, visit https://groups.google.com/d/optout.

[Mark Triggs]

Thanks Dan. It does indeed sound like I'll be hitting this issue, so
I'll check back once I'm done looking at the OAI set modifications.
Maybe I can lend a hand here too.

Cheers,

Mark

Dan Gillean <d...@artefactual.com> writes:

> Hi Mark,
>
> You may want to keep an eye on a related thread, and check out the two new
> issues I have filed as a result:
>

> - https://groups.google.com/d/msg/ica-atom-users/6BmOLdLEkPs/CqGjTliPSO8J

>
> The resulting issue tickets can be found here:
>

> - #8158 <https://projects.artefactual.com/issues/8158>, to improve the

> way we implement resumptionTokens in AtoM

> - #8159, <https://projects.artefactual.com/issues/8158> to improve the

> way that oai_dc is implemented in AtoM
>
> The first ticket may especially be of importance to you to ensure that
> Trove could resume a long request - we haven't had the opportunity so far
> to test AtoM's OAI repository module against an OAI Harvester, but based on
> the comments in the other User Forum thread, this might be worth
> investigating along the way.
>
> I've added some detailed notes and examples in the issue tickets to try to
> make life a little easier for anyone willing to take this on :)