Audit trail for deleted archival description records

46 views
Skip to first unread message

li...@orangeleaf.com

unread,
Mar 11, 2020, 11:29:30 AM3/11/20
to AtoM Users
When a user deletes an archival description record from AtoM is the record completely deleted from the Atom database or does it get retained in the database with a field signifying that it is 'deleted'?

We are trying to work out how we can create a report of archival description records that users have deleted from an AtoM installation for audit purposes.

Thanks

Linda


Dan Gillean

unread,
Mar 16, 2020, 12:15:59 PM3/16/20
to ICA-AtoM Users
Hi Linda, 

Unfortunately, AtoM doesn't currently track deletions - when a record is deleted, it is removed from the database. In general, unless it is built into the business logic as a feature from the initial design, relation databases tend to only maintain information about the current state of the world, not how the "world" looked throughout its history. Once you delete something from a relational database, it's gone (unless you've designed specific processes and tables to store information on that action). 

That said, one thing you might be able to use to glean this type of information would be MySQL's binary log. From the MySQL 5.7 documentation: 

The binary log contains “events” that describe database changes such as table creation operations or changes to table data. It also contains events for statements that potentially could have made changes (for example, a DELETE which matched no rows), unless row-based logging is used. The binary log also contains information about how long each statement took that updated data. The binary log has two important purposes:
  • For replication, the binary log on a master replication server provides a record of the data changes to be sent to slave servers. The master server sends the events contained in its binary log to its slaves, which execute those events to make the same data changes that were made on the master. See Section 16.2, “Replication Implementation”.
  • Certain data recovery operations require use of the binary log. After a backup has been restored, the events in the binary log that were recorded after the backup was made are re-executed. These events bring databases up to date from the point of the backup. See Section 7.5, “Point-in-Time (Incremental) Recovery Using the Binary Log”.  

In the long term, Artefactual has been doing research into Event Sourcing as a design pattern that might be relevant to future versions of AtoM (think AtoM3 goals), and how we might take an incremental approach to getting there by building new components around AtoM's core stack, rather than trying to re-architect the entire application in one go. For example: a sidecar application that processes AtoM's MySQL binlog and parses these into events that have meaning within the business rules of the application, and then stores these in a separate append-only log. This way, other services (both within AtoM, and wherever we want to integrate with other tools) can subscribe to the event log and only listen for events that are relevant to their workflows. Additionally, since all events in the system are permanently stored, you can figure out the state of your data at any point of time, allowing for features such as temporal queries, versioning and rollbacks, etc. 

This is a big topic, and one we are already exploring via the Enduro project with Archivematica - but it will require a lot of careful analysis, research, and design to hit upon the right solution for AtoM. 

In the meantime, there may be ways to get the information you need from the binlog without building so much infrastructure around it! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him


--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/88803a94-7c1d-4707-b125-0728f0b71baa%40googlegroups.com.

James Grimster

unread,
Mar 16, 2020, 12:22:53 PM3/16/20
to AtoM Users
Hi Dan

Does this mean that AtoM is not a deleting repository for OAI ?  

<identifier status="deleted">oai:</identifier>

I thought it was, which is why I suggested to Linda that deleted identifiers must be tracked somewhere.  I guess I was wrong.  How are harvesters managing deletes from AtoM catalogues?

Thanks

James

Dan Gillean

unread,
Mar 16, 2020, 12:43:26 PM3/16/20
to ICA-AtoM Users
Hi James, 

That's correct, unfortunately - at this time, AtoM's' OAI repository functionality has no way of letting harvesters know about deletions. I'm not sure how users are working around this issue - I suspect it's by contacting the maintaining administrators directly to request deletions in portals or other harvested aggregates. 

We would like to see this supported in AtoM in the future, but it will require feature development to be able to support. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c3e57478-224b-42cb-a31e-1b1f0ff6dbfe%40googlegroups.com.

Karl Goetz

unread,
Mar 19, 2020, 6:42:28 PM3/19/20
to ICA-AtoM Users
Hi,
Looking at the standard s 2.5 inc. 2.5.1 the status attribute is optional
An item may be organized in one set, several sets, or no sets at all. In the example above, it is conceivable that an individual item is organized in both subject and institution:florida.A harvester should not assume that harvesting every set in a repository will retrieve metadata from all items in the repository. Items may also be assigned to interior nodes in the set hierarchy.
2.5.1 stipulates that "

If a record is no longer available then it is said to be deleted. Repositories must declare one of three levels of support for deleted records in the deletedRecord element of the Identify response:

  • no - the repository does not maintain information about deletions. A repository that indicates this level of support must not reveal a deleted status in any response.
...
If a repository does not keep track of deletions then such records will simply vanish from responses and there will be no way for a harvester to discover deletions through continued incremental harvesting.
"

Which appears to be AtoMs behaviour.
~/source/atom~/source/atom$ grep -r deletedRecord plugins/arOaiPlugin/
plugins/arOaiPlugin/modules/arOaiPlugin/actions/identifyComponent.class.php:    $this->deletedRecord = 'no';
plugins/arOaiPlugin/modules/arOaiPlugin/templates/_identify.xml.php:    <deletedRecord><?php echo $deletedRecord ?></deletedRecord>

SO harvesting sites should be deleting of their own accord.

thanks,
kk

From: ica-ato...@googlegroups.com <ica-ato...@googlegroups.com> on behalf of Dan Gillean <d...@artefactual.com>
Sent: Tuesday, 17 March 2020 3:43 AM
To: ICA-AtoM Users <ica-ato...@googlegroups.com>
Subject: Re: [atom-users] Re: Audit trail for deleted archival description records
 



University of Tasmania Electronic Communications Policy (December, 2014).
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.

Karl Goetz

unread,
Mar 19, 2020, 6:43:20 PM3/19/20
to ica-ato...@googlegroups.com
Hi Linda,
What drives the need for your report? It will change how deeply you may need to dive.

thanks,
kk

From: ica-ato...@googlegroups.com <ica-ato...@googlegroups.com> on behalf of li...@orangeleaf.com <li...@orangeleaf.com>
Sent: Thursday, 12 March 2020 2:29 AM
To: AtoM Users <ica-ato...@googlegroups.com>
Subject: [atom-users] Audit trail for deleted archival description records
 
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages