Hi,
We have a use case of AtoM where we have a large number of Archival descriptions (around 3.9 mio at the moment) and we do many single description imports per day (around 500 new descriptions per day) using XML EAD using CURL requests (so using the "UI" import and not the CLI import).
The biggest bottleneck has always been the nested set update that occurs when importing data.
Since this nested set update can be disabled for many of the imports using the command line (--skip-nested-set-build), we decided to make a small change to the import XML code so that nested set updating is also disabled there and felt relatively safe doing so since this was a behavior already implemented. We then use a cron job that rebuilds the nested set nightly.
This works great for us as we are OK with not having a consistent nested set on the day of the import since everything is correctly available the day after.
We now have the issue that we sometimes want to delete many descriptions because of some errors unrelated to AtoM (bad data was sent to import usually).
Since the deletion also updates the nested set, it can sometimes take more than a minute to complete a single description, which makes it quite unusable. Even worse, since a deletion does not happen as a background task, but in the "foreground" and it locks the tables it updates, all our import tasks fail when a deletion is being performed.
We would like to solve this issue in a similar way that we did for the import, this means also disabling the nested-set updating during the deletion of a description by modifying a bit the code of the deletion process.
However, since this is not something that is done by other CLI tasks (as was the case for the import), we are unsure if there are some side effects that may occur when deleting and importing data on an "inconsistent" nested set.
We did a few tests and did not see any issues for now, but it would be great if someone from Artefactual could chime in and give some insight if it is an acceptable solution.
I have attached the patch of the changes we did.
Thanks a lot for your input,
Best,
Raphaël