Managing the dashboard (transfer and ingest)

[Andrew Berger]

Hi all,

For those of you running Archivematica in production, I'm curious to know if you have made policies for managing the transfer and ingest tabs on the dashboard. In testing, I've found that there's a certain point where the amount of information about previous transfers/SIPs on the dashboard starts to affect page load times, at least on computers with a relatively small amount of processing power.

As I understand it, once an ingest is complete and you've done whatever analysis you were going to do (if any) of the dashboard output for that transfer/SIP, it's safe to delete the corresponding entries on the dashboard. Since the information remains in the MCP database (I think), you can still retrieve it if you really need to, but it doesn't get pulled into the page when you load the transfer/ingest tabs. I suppose this is also true of failures/rejections, but you may want to keep those logs around longer for debugging and evaluation.

If you are clearing out dashboard entries for successful ingests on a regular basis, are you doing so immediately after each completed ingest? After a set amount of time has elapsed?

And if you've decided not to delete dashboard entries, or if it's actually a bad idea to delete dashboard entries, I'd appreciate hearing about that too.

Thanks,
Andrew

[Sarah Romkey]

Hi Andrew,

An interesting topic, and relevant I think to the recent thread about reporting in some ways as well.

I am very interested to hear what the current practice is from users. Our general recommendation is that the dashboard should be cleaned up "from time to time" to prevent browser time-out issues. I don't think there is a way to keep transfers/ingests sitting in the dashboard indefinitely and have the browser consistently work as your production scales up.

Cheers,

Sarah

Sarah Romkey, MAS,MLIS

Systems Archivist
Artefactual Systems
604-527-2056

@ArchivesSarah

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/d/optout.

[e w]

We have found that cleaning up the Dashboard is essential-- like you say, as the list gets bigger the Dashboard becomes less responsive and less usable. 
Our general workflow is that if the package goes through successfully and everything looks good, remove it from the dashboard at the end of your work session.  If there is some error, we usually leave it on the list until we sort out the problem.  It is important to keep on top of this clean up, because it becomes extremely tedious to remove entries when the list is long.

I would love to see a more efficient method to clean up the Dashboard!

[Andrew Berger]

Thanks for your reply. We've had Archivematica up and running in production now for just about 10 days and already I can see the need keep up with the Dashboard. We keep the "staging" copies (i.e. the folders in our transfer source location) until the corresponding AIP has been replicated in our storage, so I think what we'll do is delete the dashboard entry for each AIP at the time we also delete the staging folder for that AIP.

Andrew

--

[Ben Fino-Radin]

Same as above – we clear out the dashboard as things safely arrive the repository.

[L Snider]

I am really curious as to why leaving things in the dashboard causes a slowdown? Is it because the program checks each of them again and again if they are left there?

Cheers

Lisa

[Justin Simpson]

I can answer that question.  The Transfer and Ingest tabs of Archivematica have some javascript code in them, that polls the Archivematica web server, every 5 seconds.  It asks for a complete list of all Transfers (or SIPs, depending on the tab you are in).  The Archivematica web server looks in the database and returns a complete list of all Transfers (or SIPs), with details of all the Micro-Services for each Transfer (or SIP) and all the Jobs within each Micro-Service.

The javascript code runs in your browser, and it goes through this big dump of data from Archivematica and does things like create that little red icon showing you that there are User Inputs required. 

As the list of Transfers (you guessed it, or SIPs)  grows, it takes longer and longer to parse the data.  At a certain point (more than say 30 or 40 Transfers or SIPs, more if you have a fast computer) it can take longer than 5 seconds to ask for the data, receive it and parse it. 

In the Archivematica database, there is a table called Transfers (and another one called SIPs), that lists all the Transfers that have been processed by the local pipeline.  There is a column called 'hidden', when the value of that column is 1, that Transfer is not returned to the javascript in that big data dump, and therefore is not displayed in the browser.

If you have a ton of Transfers (or SIPs) showing in your dashboard, you can click on the red icon beside each one to 'hide' it, or you can log into your Archivematica machine, and at the command line run something like:

update Transfers set hidden = 1;

to hide all transfers.

This could be done via phpMyAdmin, rather than from the command line, if you don't want to hand out ssh credentials to your Archivematica server.

I will note finally that this javascript code is something that we would really like to replace, with something that performs better.  Artefactual is doing work on a new tab for Archivematica (see: https://www.archivematica.org/wiki/Appraisal_Arrangement_tab ).  This work is just starting, but the javascript being developed for this tab is based in Angular.js.  It is possible that this work could at some point be extended, to produce a better Transfer tab or Ingest tab.  We are a long way away from that, but trying to start down that road at least.

Justin Simpson
Director of Archivematica Technical Services
www.artefactual.com
604-527-2056

[L Snider]

Hi Justin,

Thanks for the detailed explanation, now I get it!

Cheers

Lisa

[Jenny Mitcham]

That is useful info - thanks Justin.

Also, can I just say the Appraisal and Arrangement tab proposal looks fantastic. To have some visual reporting to help show what you've got in your transfer would be so useful. Thanks to Bentley for sponsoring this! 

We had been thinking about some automated process to see which files hadn't been identified and produce a report (or 'naughty step'!) to highlight these files and have an option of triggering a process of doing something about them (ie: triggering a report or creating a sample to send to the Pronom folks). It would be nice if there was a way of directly feeding into the enhancement of the identification tools for the benefit of the whole community. Is this something that has been explored before?

Cheers,

Jen

--

Jenny Mitcham
Digital Archivist
Borthwick Institute for Archives

University of York
Heslington
York
YO10 5DD

Telephone: 01904 321170

Borthwick Institute website: http://www.york.ac.uk/borthwick/
Digital archiving blog: http://digital-archiving.blogspot.co.uk/

Twitter: @Jenny_Mitcham

[Sarah Romkey]

Hello Jen,

I think that's definitely on the table for the Arrangement/Appraisal report. One of Bentley's desires is to have some manner of alert pop up in the reports when there is a file which Archivematica does not have a normalization path for- this allows the archivist to either create a normalization command before continuing, or at least arms the archivist with more information for how they want to continue processing. Files that couldn't be identified would fall into this category too.

The Appraisal/Arrangement tab is part of a larger project that Bentley is undertaking to integrate Archivematica with ArchivesSpace (and also DSpace). ArchivesSpace has kindly let us hijack a section of their Jira account in order to capture user stories :) If anyone is interested in reading them, you can follow this link:

https://archivesspace.atlassian.net/projects/AASWF/issues

And also keep up with the Bentley project via their blog:

http://archival-integration.blogspot.ca/

Cheers,

Sarah

Sarah Romkey, MAS,MLIS

Systems Archivist
Artefactual Systems
604-527-2056

@ArchivesSarah

[Max Eckard]

Thanks for the shout-out, Sarah! We're very excited about this as well! 

We'd like for everyone to feel free to send feedback and/or comments on this feature, the Appraisal and Arrangement tab (as it is developed) or the larger Archivematica-ArchivesSpace-DSpace Workflow Integration project to the blog or our project e-mail: bhl-mell...@umich.edu

Thanks again! Have a nice weekend!

Max

Max Eckard
Assistant Archivist for Digital Curation

Bentley Historical Library

1150 Beal Ave.
Ann Arbor, MI 48109-2113
734/763-7518

http://bentley.umich.edu/