Strathclyde University - OAI-PMH to harvest archive data to Primo.

[Helen Cooper]

Hello,

 

I am a developer at Strathclyde University, and am currently looking into the possibility of harvesting our archive data from AtoM to Primo using OAI-PMH.

 

I have tried the following URL’s (which we had used successfully some time ago),  But these now return a 404 Page not found error.

http://ica-atom.org/strathclyde/index.php/oai/oai?verb=Identify

http://ica-atom.org/strathclyde/index.php/oai/oai?verb=ListRecords&metadataPrefix=oai_dc

 Can you possibly tell me the current base URL for OAI-PMH?

Also I believe there were some concerns about using OAI within AtoM 2.0 , see attached email from Dan Gillean dated October 2013.

Is there still a problem exposing data sets?

Is there some up-to-date documentation I can read?

 

Apologies with bombarding you with so many questions!

 

Many Thanks

Helen

----------------------------------------
Helen Cooper

Library Systems Analyst/Developer

Information Services Directorate

University of Strathclyde

Tel: 44 (0) 141 548 4611
Email: helen....@strath.ac.uk

 

 

The University of Strathclyde is a charitable body, registered in Scotland, with registration number SC015263.

 

[José Anjos]

Maybe you have disabled arOaiPlugin for some reason?
Hope it helps

On Thursday, 26 March 2015 11:38:22 UTC, Helen Cooper wrote:

Hello,

 

I am a developer at Strathclyde University, and am currently looking into the possibility of harvesting our archive data from AtoM to Primo using OAI-PMH.

 

I have tried the following URL’s (which we had used successfully some time ago),  But these now return a 404 Page not found error.

http://ica-atom.org/strathclyde/index.php/oai/oai?verb=Identify

http://ica-atom.org/strathclyde/index.php/oai/oai?verb=ListRecords&metadataPrefix=oai_dc 

http://10.10.1.3/                       index.php/;oai?verb=ListRecords&metadataPrefix=oai_dc

[Hutchinson, Tim]

Hi Helen,

 

The routing for OAI has indeed changed, although I believe there is also a typo in your earlier URL.

 

This should work:

http://strathclyde.ica-atom.org/index.php/;oai?verb=Identify

 

Have you enabled OAI in the settings? I still get a 404 error when I try that. If you’ve enabled the setting, it could relate to the name of the module changing – something to check if you’re still having problems is that the setting_i18n table should have arOaiPlugin rather than qtOaiPlugin.

 

Tim

 

 

Tim Hutchinson
Head, University Archives & Special Collections
University Library, University of Saskatchewan

Tel: (306) 966-6028  Fax: (306) 966-6040

Email: tim.hut...@usask.ca

Web: http://library.usask.ca/archives/

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at http://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1CFA2FC3908BE241BCB30328F268FAE447AFFB45%40EX2010-MBX3.ds.strath.ac.uk.
For more options, visit https://groups.google.com/d/optout.

[Dan Gillean]

Hi Helen,

I will wait to see if the above suggestions help resolve your issues before suggesting anything further.

Re: status and documentation of OAI-PMH in AtoM...

Basically, OAI-PMH support was quickly added to a very early version of ICA-AtoM but was not maintained, and never properly documented. Since that time, we have managed to complete a number of fixes and are slowly rendering the module useful again for exposing metadata. AtoM does not currently have the ability to act as a harvester.

Here is a post in the user forum from about a year and a half ago, to give you some context for where we were at prior to the 2.0.0 release: https://groups.google.com/d/msg/ica-atom-users/neungj7cgGI/NfhPkZlllxoJ

We have since managed to fix some of the outstanding bugs related to OAI-PMH - there are a couple of others still floating around, but this should give you a sense of what tickets we have, which have been verified and which are outstanding in our issue tracking system. The comments on #3888, #3889 and #3890 should give you a few samples of some of the verbs that can be used at the moment.

There are also a couple of new features that will be added in 2.2 for OAI-PMH - see tickets #8143 and #8144.

This means that, as the module has finally become more functional and is even seeing recent development, I need to prioritize getting some official documentation completed for this. I will likely be adding it to the 2.2 documentation. My apologies that we have not had the resources and opportunity to undertake this sooner. I hope this helps a bit in the meantime.

Regards,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6B26EA65847C3D47A5A1450ABE5A06884FE85EBA%40CAMPUSMB3.usask.ca.

[Jenny Mitcham]

Hi Helen,

Do let me know how you get on and whether you are successful in doing this. At the University of York we are going to be looking at doing something very similar in the future.

Best of luck,

Jen

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6B26EA65847C3D47A5A1450ABE5A06884FE85EBA%40CAMPUSMB3.usask.ca.

For more options, visit https://groups.google.com/d/optout.

--

Jenny Mitcham
Digital Archivist
Borthwick Institute for Archives

University of York
Heslington
York
YO10 5DD

Telephone: 01904 321170

Borthwick Institute website: http://www.york.ac.uk/borthwick/
Digital archiving blog: http://digital-archiving.blogspot.co.uk/

Twitter: @Jenny_Mitcham

[João Pereira]

Hi all!

Like to inform that the OAI-PMH AtoM has a bug. The resumption token is not working because the harvesting servers fail to make the harvester beyond the first page. To see this situation go to the OAI-PMH Validator site in http://validator.oaipmh.com/ and test your AtoM repository (do not forget to activate the plugin AtoM - arOaiPlugin and enable in settings the OAI Repository). I also have some doubts about the dublin core generated, see some examples of DC generated in DSPACE (http://repositorio-tematico.up.pt/oaiextended/request?verb=ListRecords&metadataPrefix=oai_dc&set=rap) and other similar platforms (http://arquivo.cm-vidigueira.pt/OAI-PMH/oai/?verb=ListRecords&metadataPrefix=oai_dc).

Tahnk you,

João Pereira

To post to this group, send email to ica-at...@googlegroups.com.

Visit this group at http://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1CFA2FC3908BE241BCB30328F268FAE447AFFB45%40EX2010-MBX3.ds.strath.ac.uk.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at http://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6B26EA65847C3D47A5A1450ABE5A06884FE85EBA%40CAMPUSMB3.usask.ca.
For more options, visit https://groups.google.com/d/optout.

[Hutchinson, Tim]

Hi João,

 

Are you referring to the “request time too much” error? I haven’t been able to test with a real harvester, so wasn’t sure if that was a blocker. Does changing the resumption token limit help at all?

 

Regarding the resumption token, see also this note about needing to properly encode it:

https://groups.google.com/d/topic/ica-atom-users/paj51fk51wE/discussion

 

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/cc9a9285-d777-4e64-a29f-d4fab3df6820%40googlegroups.com.

[Dan Gillean]

Hi all,

Thanks for this interesting and engaging thread - it's nice to see an under-developed feature in AtoM getting some attention.

In response to feedback received from João and Tim, I have filed two new feature requests in AtoM outlining some of the issues raised in this thread:

• #8158, to improve the way we implement resumptionTokens in AtoM
• #8159, to improve the way that oai_dc is implemented in AtoM

There remain other open tickets as well, and there might be more issues to be discovered and reported - as I said, this module has so far been underdeveloped in AtoM, and we have not yet received much feedback from anyone who is actively exposing AtoM metadata to an OAI harvester. If your institution is doing that work, please feel free to tell us more!

At this point, the two new tickets will likely require community support for us to be able to address them in AtoM - either via a community-created pull request submitted for inclusion in a future public AtoM release, or via development sponsorship. Feel free to ask questions on-list if you intend to work on a pull request; feel free to contact me off-list if your institution is willing to sponsor enhancements to the OAI repository module.

Regards,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6B26EA65847C3D47A5A1450ABE5A06884FE87033%40CAMPUSMB3.usask.ca.

[João Pereira]

Hi Dan,

Thank you for creating the threads #8158 and #8159.

For the thread #8159, I think is easy to resolve the problem of the dublin core. I made some modifications to the file _dc.xml.php in "[AtoM diretory]\plugins\sfDcPlugin\modules\sfDcPlugin\templates" of my localhost AtoM, then where is:

<oai_dc:dc xmlns="http://purl.org/dc/elements/1.1/"
    xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">

  <title><?php echo esc_specialchars($resource->title) ?></title>

  <?php foreach ($resource->getCreators() as $item): ?>
    <creator><?php echo esc_specialchars($item) ?></creator>
  <?php endforeach; ?>

  <?php foreach ($dc->subject as $item): ?>
    <subject><?php echo esc_specialchars($item) ?></subject>
  <?php endforeach; ?>

  <description><?php echo esc_specialchars($resource->scopeAndContent) ?></description>

  <?php foreach ($resource->getPublishers() as $item): ?>
    <publisher><?php echo esc_specialchars($item) ?></publisher>
  <?php endforeach; ?>

  <?php foreach ($resource->getContributors() as $item): ?>
    <contributor><?php echo esc_specialchars($item) ?></contributor>
  <?php endforeach; ?>

  <?php foreach ($dc->date as $item): ?>
    <date><?php echo esc_specialchars($item) ?></date>
  <?php endforeach; ?>

  <?php foreach ($dc->type as $item): ?>
    <type><?php echo esc_specialchars($item) ?></type>
  <?php endforeach; ?>

  <?php foreach ($dc->format as $item): ?>
    <format><?php echo esc_specialchars($item) ?></format>
  <?php endforeach; ?>

  <identifier><?php echo url_for(array($resource, 'module' => 'informationobject'), true) ?></identifier>

  <identifier><?php echo esc_specialchars($resource->identifier) ?></identifier>

  <source><?php echo esc_specialchars($resource->locationOfOriginals) ?></source>

  <?php foreach ($resource->language as $code): ?>
    <language xsi:type="dcterms:ISO639-3"><?php echo strtolower($iso639convertor->getID3($code)) ?></language>
  <?php endforeach; ?>

  <?php if (isset($resource->repository)): ?>
    <relation><?php echo url_for(array($resource->repository, 'module' => 'repository'), true) ?></relation>
    <relation><?php echo esc_specialchars($resource->repository->authorizedFormOfName) ?></relation>
  <?php endif; ?>

  <?php foreach ($dc->coverage as $item): ?>
    <coverage><?php echo esc_specialchars($item) ?></coverage>
  <?php endforeach; ?>

  <rights><?php echo esc_specialchars($resource->accessConditions) ?></rights>

</oai_dc:dc>


I modified to:

<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">

  <dc:title><?php echo esc_specialchars($resource->title) ?></dc:title>

  <?php foreach ($resource->getCreators() as $item): ?>
    <dc:creator><?php echo esc_specialchars($item) ?></dc:creator>
  <?php endforeach; ?>

  <?php foreach ($dc->subject as $item): ?>
    <dc:subject><?php echo esc_specialchars($item) ?></dc:subject>
  <?php endforeach; ?>

  <dc:description><?php echo esc_specialchars($resource->scopeAndContent) ?></dc:description>

  <?php foreach ($resource->getPublishers() as $item): ?>
    <dc:publisher><?php echo esc_specialchars($item) ?></dc:publisher>
  <?php endforeach; ?>

  <?php foreach ($resource->getContributors() as $item): ?>
    <dc:contributor><?php echo esc_specialchars($item) ?></dc:contributor>
  <?php endforeach; ?>

  <?php foreach ($dc->date as $item): ?>
    <dc:date><?php echo esc_specialchars($item) ?></dc:date>
  <?php endforeach; ?>

  <?php foreach ($dc->type as $item): ?>
    <dc:type><?php echo esc_specialchars($item) ?></dc:type>
  <?php endforeach; ?>

  <?php foreach ($dc->format as $item): ?>
    <dc:format><?php echo esc_specialchars($item) ?></dc:format>
  <?php endforeach; ?>

  <dc:identifier><?php echo url_for(array($resource, 'module' => 'informationobject'), true) ?></dc:identifier>

  <dc:identifier><?php echo esc_specialchars($resource->identifier) ?></dc:identifier>

  <dc:source><?php echo esc_specialchars($resource->locationOfOriginals) ?></dc:source>

  <?php foreach ($resource->language as $code): ?>
    <dc:language xsi:type="dcterms:ISO639-3"><?php echo strtolower($iso639convertor->getID3($code)) ?></dc:language>
  <?php endforeach; ?>

  <?php if (isset($resource->repository)): ?>
    <dc:relation><?php echo url_for(array($resource->repository, 'module' => 'repository'), true) ?></dc:relation>
    <dc:relation><?php echo esc_specialchars($resource->repository->authorizedFormOfName) ?></dc:relation>
  <?php endif; ?>

  <?php foreach ($dc->coverage as $item): ?>
    <dc:coverage><?php echo esc_specialchars($item) ?></dc:coverage>
  <?php endforeach; ?>

  <dc:rights><?php echo esc_specialchars($resource->accessConditions) ?></dc:rights>

</oai_dc:dc>


Because I'm not

--

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/cc9a9285-d777-4e64-a29f-d4fab3df6820%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[João Pereira]

But because I am not a developer, I ask to test these changes and evaluate its possible application.

Thank you all!

Best regards,

João Pereira

[Dan Gillean]

Hi all,

First, João, thanks for your useful comments! I have added them to ticket #8159.

I have some other good news on the OAI front as well:

I have just tested and verified a fix for #6434 (earliestDateStamp not formatted correctly). We've also resolved #7964 (from and until parameters overwriting each other in OAI request handling) These fixes will be included in our upcoming 2.2 release.

We have a community developer who is looking into developing new functionality for AtoM's OAI repository module - he has mentioned in this recent thread that he hopes to be able to address #6436 as well. We're sure crossing our fingers! 

I also have the initial OAI repository documentation completed in our 2.2 documentation branch - you can't see this yet on our website (we'll put up the 2.2 docs when 2.2 is released), but for those curious now, you can see it in our GitHub repository. Note that GitHub markdown is different from the Sphinx reST we are using to write the docs, so the presentation version will not render correctly in GitHub's page - but it is still readable (minus images), and you can get a sense of what has been added. You can also view the raw reST version directly if desired:

• GitHub markdown rendered version: https://github.com/artefactual/atom-docs/blob/2.2/user-manual/import-export/oai-pmh.rst
• Original raw (reST) version: https://raw.githubusercontent.com/artefactual/atom-docs/2.2/user-manual/import-export/oai-pmh.rst
• The commit: https://github.com/artefactual/atom-docs/commit/d717b7b284036581bfd0c8448a7ca6aa15aabae5
  

This is where we currently are with OAI-PMH related issues in our AtoM project issue tracker. 5 issues; only one outstanding bug! #8158 and #8159 are currently listed as features because there are workarounds, though for fully automated harvesting they may be necessary. The other two feature requests are development ideas that might extend the usefulness of the OAI repository module. Likely, all tickets currently marked feature requests would require some form of community support or sponsorship for Artefactual to be able to address.

Nevertheless, we have slowly made progress. Helen, I hope this gives you some of the answers you were looking for in your original email!

Regards,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/caac5ea4-7ca0-4a38-96b2-c25de72b5215%40googlegroups.com.

[João Pereira]

Great news Dan!

Best regards,

João Pereira

[Mark Triggs]

Hi everyone,

Just a quick message to note that I'm currently looking at a couple of these issues. For this one:

https://projects.artefactual.com/issues/8158

I have enhanced the resumption token implementation to keep track of the OAI set used, and have modified the encoding scheme to use base64-encoded JSON (which fits the bill for not requiring any special escaping when harvested).

I also hit the same issue as João Pereira documented in this issue:

https://projects.artefactual.com/issues/8159

so I've incorporated similar handling of special characters in the OAI XML responses.

With the changes I've made so far, I've got jOAI automatically harvesting its way through one of the new OAI sets I added recently, so things are looking better. Once I've had a chance to do some more testing I'll clean up my work and send a pull request for review. Next week, hopefully!

Cheers,

Mark

Dan Gillean <d...@artefactual.com> writes:

We have a community developer who is looking into developing new functionality for AtoM's OAI repository module - he has mentioned in this recent thread that he hopes to be able to address #6436 as well. We're sure crossing our fingers! 

[...]

This is where we currently are with OAI-PMH related issues in our AtoM project issue tracker. 5 issues; only one outstanding bug! #8158 and #8159 are currently listed as features because there are workarounds, though for fully automated harvesting they may be necessary. The other two feature requests are development ideas that might extend the usefulness of the OAI repository module. Likely, all tickets currently marked feature requests would require some form of community support or sponsorship for Artefactual to be able to address.

Nevertheless, we have slowly made progress. Helen, I hope this gives you some of the answers you were looking for in your original email!

-- 
Mark Triggs
<ma...@teaspoon-consulting.com>
http://teaspoon-consulting.com/

[Dan Gillean]

Mark, this is great news!

Thank you so much for working with us so that the entire community can benefit from the enhancements you are undertaking. Seeing this kind of collaboration in action makes me proud to be a part of the AtoM community.

We look forward to checking our your pull requests!

Cheers,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at http://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/87oamwfong.fsf%40teaspoon-consulting.com.

[Mark Triggs]

Thanks Dan! I must in turn thank the Australian National University Archives for sponsoring this work and giving me a good reason to dive in :)

Cheers,

Mark

Dan Gillean <d...@artefactual.com> writes:

Mark, this is great news!

Thank you so much for working with us so that the entire community can benefit from the enhancements you are undertaking. Seeing this kind of collaboration in action makes me proud to be a part of the AtoM community.

We look forward to checking our your pull requests!

Cheers,

-- 
Mark Triggs
<ma...@teaspoon-consulting.com>
http://teaspoon-consulting.com/

[Creighton Barrett]

Amazing. This is so cool. Looking forward to hearing your progress. Great work all around!

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at http://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/87k2xjeh2h.fsf%40teaspoon-consulting.com.

[Victoria Peters]

Yes, excellent!  Thanks very much.

 

Victoria

 

 

From: ica-ato...@googlegroups.com [mailto:ica-ato...@googlegroups.com] On Behalf Of Creighton Barrett
Sent: 11 April 2015 20:39
To: ica-ato...@googlegroups.com
Cc: Dan Gillean
Subject: Re: [ica-atom-users] RE: Strathclyde University - OAI-PMH to harvest archive data to Primo.

 

Amazing. This is so cool. Looking forward to hearing your progress. Great work all around!

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAHueW_WKtXRGwS5eAWF7Fd7QiL4PkRWSOjR8GWWFgvh_0R7u_A%40mail.gmail.com.

[Dan Gillean]

Hi all,

One more positive update:

Mark Triggs has implemented João's suggestions on #8159, I've tested them locally, and Mark has tested them with 2 different OAI harvesters. We've added this as the last feature that will make it into the 2.2 release!

Note that this fix has an inadvertent positive side effect - it makes our DC XML more compliant throughout the application :D

At this point, looking in our issue tracker, there are currently no open bug tickets for OAI-PMH - only a couple enhancements/feature requests to add future functionality.

A big thanks to everyone involved in this thread for working together to help outline the issues, come up with strategies, and implement them! Thanks especially to João for the suggestions on #8159 (which was filed in response to his testing and comments on this thread), and to Mark for implementing them and sharing the code with the AtoM community!

Regards,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

[Creighton Barrett]

Fantastic work, all around! We've been super interested in harvesting AtoM with Primo and I'm thrilled to see that this now looks possible. I'd love to hear from Mark and anyone else who has successfully harvested some AtoM records into a different system using these new OAI tools. We plan on testing in 2.2 when we get it installed.

Well done!

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZKKi8TxY3-3tW%2BnEfRBZWJyj8wJXHB0KvD4icWhGQHZRg%40mail.gmail.com.

[João Pereira]

Fantastic job of everyone involved!

Thank you all!

João Pereira

--
You received this message because you are subscribed to a topic in the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ica-atom-users/6BmOLdLEkPs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ica-atom-user...@googlegroups.com.

To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at http://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAHueW_Ug%2B9pcV5WoxB2U-%2B6fe3yPUjChrYMHPYCxm5gSTXmKVg%40mail.gmail.com.

[Victoria Peters]

Great news indeed!

 

We will be testing too, once we have 2.2.

 

Best wishes

 

Victoria

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAOJOS7%2BNohx%3Dus%3D99%3DHT9JK6JGhB5%3Ddaq3s%2BTXEYZRD0uRbvkw%40mail.gmail.com.

[Mark Triggs]

Hi Creighton,

My testing so far has been with the jOAI harvester and Trove.

With the jOAI harvester (http://www.dlese.org/oai/docs/harvester.jsp), I've confirmed I can successfully run full and partial harvests. This doesn't do anything much with the harvested records, but at least checks that the mechanics of the OAI repository are working.

In conjunction with the Australian National University Archives and the Trove team (http://trove.nla.gov.au/), I've also been working on getting an AtoM repository harvested by Trove. That's looking good too: Trove has been able to harvest records from AtoM and add them to its collection, so that's a good indicator that it will work with harvesters "in the wild" too. It will be interesting to hear whether the Primo harvester works too.

Happy to answer any questions if you want to shoot me an email (either on or off list is fine).

Cheers!

Mark

Creighton Barrett <csba...@gmail.com> writes:

Fantastic work, all around! We've been super interested in harvesting AtoM with Primo and I'm thrilled to see that this now looks possible. I'd love to hear from Mark and anyone else who has successfully harvested some AtoM records into a different system using these new OAI tools. We plan on testing in 2.2 when we get it installed.

-- 
Mark Triggs
<ma...@teaspoon-consulting.com>
http://teaspoon-consulting.com/

[Creighton Barrett]

Awesome, thanks Mark. Great to hear about your early successes. I will definitely get in touch once 2.2 is released and we get to the upgrade and make time for testing in Primo. Could be a little while, but it's definitely high on our to-do list.

[Gomes Silva]

Hello everyone,

There are two things I need to change in _dc.xml.php file.

1- The first thing are related to dates. The Harvester I want to subscribe uses this dc scheme for dates (a start date and a end date):

<dc:date>1901</dc:date>

<dc:date>1909</dc:date>

So, with the help of João we did this:

<?php foreach ($resource->getDates() as $item): ?>

     <dc:date><?php echo $item->startDate ?></dc:date>

  <?php endforeach; ?>

  <?php foreach ($resource->getDates() as $item): ?>

     <dc:date><?php echo $item->endDate ?></dc:date>

  <?php endforeach; ?>

Result:

<dc:date>1901-00-00</dc:date>

<dc:date>1909-00-00</dc:date>

It works fine, except when months and days don’t exist. If months and days are not filled it will show zeros on them. I need the startDate and endDate but how can I hide month and day values when they are not filled?

In the default code, years, months and days behave as expected. Is there a way to make these startDate and endDate behave like the default code (hide month and day values when empty)? 

2- The second thing is related to thumbnails.

The Harvester I want to subscribe is able to generate thumbnails if you place the digital object URL on <dc:relation> element,

so according to João we had to do this:

<?php if (isset($resource->digitalObjects[0])): ?>

   <dc:relation>http://<?php echo $sf_request->getHost().$sf_request->getRelativeUrlRoot().$resource->digitalObjects[0]->getFullPath() ?></dc:relation>

<?php endif; ?>

However, instead of providing the Master's URL (which is a heavy file, not always available to everyone and not always JPEG) we wanted to provide the Thumbnail URL instead which basically is the same URL but it ends on _142.jpg. What should be changed in this code?

I don’t know if there’s an easy solution for these problems, but if they are quick to solve I appreciate your help.

Thank you very for your support!

[sbr...@artefactual.com]

Hello Gomes

Regarding your first question on the dates:

You should be able to use our renderDates() function like so:

foreach ($io->getDates() as $date)
{
  print("Date: " . Qubit::renderDate($date->startDate) . "\n");
}

Output:  Date: 2017

You second point is a little more difficult for me to help you with as I do not know how the path to the digital object needs to be manipulated - you say it needs to end in '_142.jpg' but I suspect that number would need to change per object.  You could certainly append anything you want to the path string, but I suspect it's not as simple as this.

Steve

[Gomes Silva]

Excelent!

The Qubit::renderDate function worked perfectly.

The first question is solved. Thank you very much for your support!

[Gomes Silva]

Just to clarify the second point:

I don't know if archivematica's DIP have the same behavior, but AtoM 2.x creates files like this:

Filename                                            Size

derivada_PT-CMALB-NI-DNPDN-DNPDN-4940.pdf   13636KB This is the master file

derivada_PT-CMALB-NI-DNPDN-DNPDN-4940_141.jpg 49KB This "_141.jpg" is the reference representation

derivada_PT-CMALB-NI-DNPDN-DNPDN-4940_142.jpg 36KB This "_142.jpg" is the thumbnail representation

Note: They are always stored on the same folder. I think this is AtoM's default behavior.

If there's an easy way to show thumbnail's path it would be great.

In any case, the first question was what concerned me most and you already solved it.

Thank you again for all your help.

[Dan Gillean]

Hi Gomes, 

You might find the following post (and it links to another thread as well) useful in figuring out how the digital objects are stored, and how you might figure out the file path you need. See: 

• https://groups.google.com/d/msg/ica-atom-users/3SRd50y9bEM/sld_BBDjBgAJ
  

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e7be7cf9-fecb-4f5a-aeac-0429be1cf5af%40googlegroups.com.

[Gomes Silva]

Hello Dan,

Thanks to your and Sbreker's tips I managed to solve the thumbnail issue with rtrim() and suchlike. So, the issues are pretty much solved and we thank you both for that.

If I'm not mistaken AtoM is designed around the standards-based principle of non-duplication of information at lower levels of description so we have to enter the name of the "Creator" at the highest level of description only. I believe the same is true for "Repository" and "LangMaterial" elements.

So, is there a way to make DublinCore XML grab a toplevel creator, repository and langmaterial information?

These are mandatory elements to that national Harvester, but they don't appear in lowerlevel descriptions.

Thank you again

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

[Dan Gillean]

Hi Gomes, 

There's no easy way to do this without overhauling how the DC XML and OAI Repository plugin are currently implemented, unfortunately. However, I did want to let you know about a new feature coming in 2.4 that might be of use to you. In 2.4, you will be able to expose EAD 2002 XML via the OAI Repository module. Since EAD 2002 XML contains a complete archival hierarchy, you won't need to perform multiple queries to get all the different levels of description and then try to figure out how they relate to each other after. Docs here: 

• https://www.accesstomemory.org/docs/2.4/user-manual/import-export/oai-pmh
  

Often, EAD 2002 XML exports can be too big to generate on the fly and serve to a harvester - so AtoM 2.4 will also include the ability to pre-generate the  EAD XML every time you create or edit a description via the user interface, and then cache the XML so it can be served up when users try to download the file or a harvester requests oai-ead as the metadata format. See: 

• https://www.accesstomemory.org/docs/2.4/user-manual/import-export/oai-pmh/#cache-xml-oai
  
• https://www.accesstomemory.org/docs/2.4/user-manual/administer/settings/#cache-xml-setting
  

There is also a command-line task available that will allow you to generate and cache the XML for all existing descriptions: 

• https://www.accesstomemory.org/docs/2.4/admin-manual/maintenance/cli-tools/#cache-xml-cli
  

I hope this might be of use for your needs!

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/2971923a-8aaf-4250-a8c3-f09860d67c1d%40googlegroups.com.

[arquivo.a...@gmail.com]

Hi,

For those who are interested in providing the Thumbnail URL we can use $digitalObject->getChildByUsageId(QubitTerm::THUMBNAIL_ID);

Regards,

Helder

[Gomes Silva]

Hello again,

Thank you for the reply. 

The EAD will surely solve those problems. I just hopped it would be as easy as it was for the identifier element. By default, the _dc.xml.php uses $resource->identifier which only shows the identifier of the current record, but if we replace it with $dc->identifier it will bring all identifiers from the upper levels as well, giving us a complete hierarchical reference code. Maybe a better example would be with the CSV export where those empty fields (creator, repository and LangMaterial) are automatically filled based on the upper levels in spite of CSV not being an hierarchical format.

Just one more question, regarding to <dc:relation>. We managed to make it retrieve the master, reference and thumbnail URLs, but is there a way to (make it) check if the associated digital object (e.g. master) is accessible to public?

The idea is that if $resource->digitalObjects[0] cannot be accessed/seen by anonymous users it would echo something else, like a $digitalObject->getChildByUsageId(QubitTerm::REFERENCE_ID) instead, for instance.

Thank you again for your patience.

[Gomes Silva]

Hello, 

Just a quick update to tell that the Repository issue has been solved.

So, now the only thing that remains is the Creator. After this the Dublin Core export will totally be compatible with  AtoM's (and ISAD's) standards-based principle of non-duplication of information at lower levels of description.

1. How to enable the inherited reference code

This is AtoM's original code:

<?php echo esc_specialchars(strval($resource->identifier)) ?>

Change to this:  

<?php echo esc_specialchars(strval($dc->identifier)) ?>

2. How to enable the inherited Repository

This is AtoM's original code:

<?php echo esc_specialchars(strval($resource->repository->authorizedFormOfName)) ?>

Change to this:

<?php echo esc_specialchars(strval($resource->getRepository(array( 'inherit' => true)))) ?>

3. How to enable the inherited Creator

This is AtoM's original code:

  <?php foreach ($resource->getCreators() as $item): ?>

    <dc:creator><?php echo esc_specialchars(strval($item)) ?></dc:creator>

  <?php endforeach; ?>

Change to this:

  ...

Well, I still don't know how to do this one, but I believe it's possible since that information is already being shown at lower levels in the archival descriptions view just like Repository did. If I manage to find out I'll update here. Meanwhile if a developer could provide the code, it would be awesome.

Best regards

[Dan Gillean]

Hi Gomes, 

The developer who is best suited to answer this question is currently on vacation, but I've sent him a note and asked him if he can provide any guidance when he returns. 

Hopefully we'll be able to point you in the right direction! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/a7e70a7a-50e8-4d0c-a4db-784970e84c46%40googlegroups.com.

[José Raddaoui]

Hi Gomes,

You'll need to traverse the ancestors (and self) going backwards and get the first resource with existing creators. This is the code used in the archival descriptions view page:

https://github.com/artefactual/atom/blob/qa/2.5.x/apps/qubit/modules/informationobject/actions/creatorDetailComponent.class.php#L24-L30

I hope that helps.

Regards.

[Ricardo Pinho]

Hi José,

Based on your tip I've managed to find this solution:

https://github.com/atom-pt/atom_cmoaz/blob/097d96ccf3064701524eabf0c1e72370817d65a1/plugins/sfDcPlugin/modules/sfDcPlugin/templates/_dc.xml.php#L373

Could you please have a look at it and coorect if needed.

Thank you.

Cheers,

Ricardo Pinho

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/f00e46a1-cc63-420e-bc6d-c2d82f55aacc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Ricardo Pinho

[José Raddaoui]

Hi Ricardo,

That should print only the first creator if founds in the ancestors. If that's the intended behavior, it looks good to me. Otherwise, if you want to print all the creators of the first ancestor with some, you could remove the first "break;". I'd also change "item" for "ancestor" and "ancestor" for "creator", as it makes more sense when reading the code, but that shouldn't change the final result.

Best regards.

[Gomes Silva]

You guys are amazing. Thank you all for your support!

As promised, let me share what was found out:

1. How to enable the inherited reference code

This is AtoM's original code:

<?php echo esc_specialchars(strval($resource->identifier)) ?>

Change to this:  

<?php echo esc_specialchars(strval($dc->identifier)) ?>

2. How to enable the inherited Repository

This is AtoM's original code:

<?php echo esc_specialchars(strval($resource->repository->authorizedFormOfName)) ?>

Change to this:

<?php echo esc_specialchars(strval($resource->getRepository(array( 'inherit' => true)))) ?>

3. How to enable the inherited Creator

This is AtoM's original code:

  <?php foreach ($resource->getCreators() as $item): ?>

    <dc:creator><?php echo esc_specialchars(strval($item)) ?></dc:creator>

  <?php endforeach; ?>

Change to this:

 <?php

  foreach ($resource->ancestors->andSelf()->orderBy('rgt') as $ancestor)

  {

    foreach ($ancestor->getCreators() as $creator)

    {

      if (0 < count($creator))

      {

        echo "<dc:creator>" . esc_specialchars(strval($creator)) . "</dc:creator>";

        break; //Comment this line if you want to show multiple creators

      }

      break;

    }

  }

 ?>

4. How to enable the inherited Language

 This is AtoM's original code:

 <?php foreach ($resource->language as $code): ?>

    <dc:language xsi:type="dcterms:ISO639-3"><?php echo esc_specialchars(strval(strtolower($iso639convertor->getID3($code)))) ?></dc:language>

  <?php endforeach; ?>

Change to this:

<?php

  foreach ($resource->ancestors->andSelf()->orderBy('rgt') as $ancestor) {

    if (0 < count($ancestor->language)){

    foreach ($ancestor->language as $code){

      if (isset($code)) {

        echo "<dc:language>";

        echo esc_specialchars(strval(strtolower($iso639convertor->getID3($code))));

        echo "</dc:language>";

      }

      //break; #uncomment for single language (shows only the first in alphabetical order)

    }

    break;

    }

  }

?>

Best regards

Visit this group at <a href="https://groups.google.com/group/ica-atom-users" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/group/ica-atom-users';return true;" onclick="this.href='https://groups.google.

 

[Ricardo Pinho]

Hi José,

> That should print only the first creator if founds in the ancestors.

> If that's the intended behavior, it looks good to me.

> Otherwise, if you want to print all the creators of the first ancestor with some, you could remove the first "break;".  

That's correct! We are now testing both behaviors to see witch one is the best.

>  I'd also change "item" for "ancestor" and "ancestor" for "creator", as it makes more sense when reading the code, but that shouldn't change the final result.

Good point, I've just done that for better understanding. Thanks!

https://github.com/atom-pt/atom_cmoaz/commit/482a483aefb73f9ab3ed7e03ab15380586cf29f3#diff-d4e1b81ba0473ef8a80e6c6aaa6b00d3

Cheers,

Ricardo

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e6c5b476-7285-4d23-a75b-663d24e265ed%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Ricardo Pinho