AtoM import/export

[specialb...@gmail.com]

Hello,

We recently upgraded to 2.4 and have been trying out the new CSV import/export features. Artefactual staff recommends that during UI CSV imports, we should limit other AtoM activities such as adding/editing descriptions. Doing so may interrupt the import. We want to clarify how the import/export pipeline is structured in order to prevent job interruptions.

If we initiate a UI CSV import and a public user is using the clipboard at the same time to export to CSV, does this affect the import process? How about if an AtoM user is trying to export using clipboard at the same time as an import?

Are export and import jobs done using separate pipelines? In general, should we refrain from starting a new import/export job while another one is running or at the same time?

Thanks,
Jacky

[Dan Gillean]

Hi Jacky, 

Both import and export tasks from the clipboard will use the job scheduler. As I understand it, Gearman should be able to handle multiple atom workers running jobs in parallel. If an import job is already running and no other Gearman workers are available, then theoretically the new task will be queued, and won't begun running until another atom-worker is available. 

Technically this should be fine, but in reality, and especially with larger task (e.g. large imports, etc), it is possible for the memory available to the job scheduler to be used up, which can cause the atom-worker to stall. When that happens, the job will show in the user interface as in progress, indefinitely. 

In that case, since we don't currently have a task that can end one specific job, the only option is to clear all jobs with the following command-line task: 

• php symfony jobs:clear

The result of this would be that not only the stalled job is cleared, but all your other queued jobs as well - meaning you'd have to go and restart them by manually re-adding the records to the clipboard, etc. 

The reason for limiting other activities is to minimize the risk of impacting shared resources in the database. In AtoM's current database schema, importing data results in expensive database operations mostly around updating hierarchical data. This causes concurrent access to the database to be suboptimal, which in turn has the potential to cause issues. 

Essentially jobs are handled separately, but because they share the same database and the same resources (such as available memory), doing too much at once can cause concurrency issues, leading to other errors and problems in the database. 

For this reason, I might recommend not starting another import or export job while a large UI import is already running. 

It is possible to implement Gearman in different ways, and change some of its defaults - I believe that you can make more workers available, and you should be able to assign more system resources (such as memory) - you'd likely have to dig in the Gearman documentation. You could also change your PHP execution limits. See the following pages: 

• Job scheduler default configuration: https://www.accesstomemory.org/docs/latest/admin-manual/installation/asynchronous-jobs/#gearman-job-worker
• PHP execution limits: https://www.accesstomemory.org/docs/latest/admin-manual/installation/execution-limits/
• Gearman protocol documentation: http://gearman.org/protocol/

In terms of queueing there are likely improvements we could make with community support, such as adding support for persistent queues. See: 

• http://gearman.org/manual/job_server/#persistent_queues
  

To ensure that all operations in AtoM are atomic and avoid potential conflicts would require a deeper analysis of AtoM's current implementation, and some refactoring. 

Finally, note that another AtoM community user has shared a potential solution to prevent the atom-worker from going idle after too long a period of inactivity - see the following thread: 

• https://groups.google.com/d/msg/ica-atom-users/X-07V3RUusw/3sHhpsrzCAAJ
  

Please note that Artefactual has not tested this proposed solution, and as such, cannot officially recommend this - proceed at your own risk if you choose to try this, and let us know how it goes! 

Regards,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c03e8f33-884f-4e3b-a119-4d8a221af72a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[specialb...@gmail.com]

Hi Dan,

Thanks so much for the detailed explanation. We will note these recommendations for future imports/exports. I also passed this to our IT support team so they are aware of these as well.

Jacky

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.