Did somebody know about 'how to do automatic harvest from another AtoM user?'

74 views
Skip to first unread message

Satrio Rizki

unread,
Feb 23, 2018, 3:36:35 AM2/23/18
to AtoM Users

Hello everybody,

I work with AtoM 2.3.1, I have atom as server, and another atom clients (its more than 90 until today). Currently we harvest metadata from atom clients using export tool, and import them to the server manually. I wish if anybody here know something that we could use to harvest automatically from all of our clients to our server. We will glad to received any information useful from all of you. Thank you

Sorry for my English :(

PS: Below, there are two schemes both existing and proposed system. Best regards!

EXISTING SYSTEM


PROPOSED SYSTEM



Dan Gillean

unread,
Feb 23, 2018, 11:58:15 AM2/23/18
to ICA-AtoM Users
Hi Satrio, 

This is an interesting issue and one that has come up a lot in the Canadian context, where each province/territory has a portal site (most of which now use AtoM), and each portal sends its data to the national archival portal (also running AtoM). This particular use case was unfortunately not considered during the original application design 10 years ago - and the technology landscape has changed a lot since then. We hope to be able to incorporate a solution for this directly into AtoM3 in the future, but for now it will require some development and creative research. I can offer you a few starting places to explore. 

First, it sounds like it is possible for Elasticsearch to create a new index that searches over multiple existing ES indices - so it might be possible to to create a portal front-end that aggregates data from all of your "client" sites. However, this is not something we have explored yet, and I'm not sure how difficult it would be to combine 90+ indices all on different machines - you'll have to do some research! 

You might be interested in looking at the University of British Columbia's OpenCollections website. They are aggregating data from multiple different platforms into one common, custom-built search interface. To gather data from AtoM, they are pulling it directly from Elasticsearch's API. 

You can see here that an archival result gives returns basic information, and points the user to the relevant AtoM installation for further context and additional information: 

Another option might be to see if you can use either AtoM's API, or the OAI harvesting module to automate data extraction from your client systems. See: 
Finally, we do have some users who are making use of our replication script to manage a 2-site deployment - some large sites will use one read/write AtoM site internally for data creation and editing, and another read-only site for public use, combined with a replication script (that copies the database, digital objects, and search index to the public site) to sync them whenever it is triggered. Our replication script is available here: 
None of these solutions will solve your use case out of the box - all will require analysis and further modification/development, but hopefully they might act as starting points for consideration. 

We have previously discussed OpenCollections, as well as Simon Fraser University (who use the 2-site/replication script solution), in the following slide deck, in case it is of interest: 
Note that in the Canadian context, some of the new import/export functionality included in the 2.4 release was added to offer better support for migrating select data from one AtoM instance to another. For example, you can configure what levels of description are included in an export, whether or not draft records are included, and more. See the import/export documentation for more details. It is a temporary solution, as so much manual data duplication is undesireable, but it will certainly require a lot more analysis and development to improve, which in turn will require community support for Artefactual to be able to address. 

I hope this helps provide ideas, at least! Keep us posted on whatever solution you settle upon! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/24d98820-44ba-4538-be9e-ef6dae0f1b46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Satrio Rizki

unread,
Feb 25, 2018, 9:26:37 PM2/25/18
to AtoM Users
Dear Mr. Dan Gillean,

Thanks for the response, it's so meaningful for us. Maybe our condition is similar like Canada where each province has a portal site. We will continue our research along with your guide and recommendation, we hope that we could provide a good result for us all in the future.

If we found something good, we will post them immediately here. Again, thank you very much

abe...@utoronto.ca

unread,
Feb 27, 2018, 10:06:33 AM2/27/18
to AtoM Users
Hi Dan,

Seems like the following link is not working:
https://open.library.ubc.ca/collections/johnkeenlyside/items/1.0127376

Could you provide us another example?

Thank you.

Abraham

Dan Gillean

unread,
Feb 27, 2018, 11:40:18 AM2/27/18
to ICA-AtoM Users
Hi Abraham, 

Sorry about that! Not sure what has happened to poor Keenlyside. Here is a different example: 

And, the source records in the Rare Books and Special Collections AtoM instance: 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/da4e8b07-7bff-40fc-90b9-e03e995b46da%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages