Store DIP options

[Courtney Whitmore]

Hello again,

So we've just updated our production environment to the newest version but are having a smallish problem. In our previous setup, when we stored our AIPs, somehow a DIP was also stored. And the DIP had the same sequence as the AIPs.

So for example, if this was our AIP's top folder:

 

Our old dips would look like this:

However, now the DIPs do not automatically store, we have to tell them to do so in the dashboard. Additionally, they have the same folder structure as the AIPs and a unique sequence. 

We have had some staff change over and I have looked through our documentation and the documentation on the website, but am unable to find out how we originally had it set up to do that. So I thought I would simply ask and see if anyone here had a similar set up and/or some advice. 

I would be grateful for any advice or direction.

Kind regards,

Courtney Tang

[Sarah Romkey]

Hi Courtney,

Just to clarify, which version did you update to? I assume 1.5.1.

We did change the DIP upload/store workflow in this release- see: https://wiki.archivematica.org/DIP_storage_to_designated_location

However, in previous versions, the DIP was not stored in the same way that you're being prompted for in the dashboard- it was one of several upload options, but if you chose to upload to say, ArchivesSpace, you wouldn't also have the choice of uploading to a storage location as defined by the Storage Service.

I think what you're thinking of is that a copy of the DIP was sent to the uploadedDIPs directory. This is still true. The only circumstance under which this doesn't happen is if you choose to store instead of upload from the first menu of options.

I know it's a little confusing! I hope this makes sense. I think the most clear explanation we have is the diagram on the wiki that I linked to above.

Cheers,

Sarah

Sarah Romkey, MAS,MLIS

Archivematica Program Manager
Artefactual Systems
604-527-2056

@archivematica / @accesstomemory

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematica+unsubscribe@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at https://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/d/optout.

[Courtney Whitmore]

Hello Sarah,

Thank you for this information. Actually, it seems that we are only at V 1.5.0.

What you're saying makes sense, especially in the context of what I am seeing. What I am trying to determine, however, is why the file structure of our stored DIPs has changed and whether or not it's something we've inadvertently done on our side, and if it really is just a change to the system. 

I think I am explaining it poorly, however. When our DIPs are storing locally now, the folder is at the bottom of a directory that is configured in the same structure as when our AIPs store. 

Before, when our DIPs stored, they looked like this: 

Just a folder in our desired destination.

Now they store at the ninth level of a directory:

Our AIPs store this way, but as a zipped folder at the ninth level. They have always done that, however, so it wasn't unusual. I have also noted that the sequence of the AIP and DIP folder names both match the collective names of the directory folders for the AIP. However, the series of folders for the DIP is different. Actually, that the folders in the sequence are different makes sense to me, but I thought I should point it out.

We are just trying to figure out why. We thought maybe it was a default setting that we had changed but we cannot identify the cause. Likewise, this used to be the workflow:

1. I would tell the AIP to store.

2. It would store in the selected drive on the server. 

3. I would not select any instructions for the DIP; however, a DIP would also store on the server, in the manner of the first image that I showed above. 

Now I have to do as you might expect.

1. Store the AIP

2. Then choose to store a DIP

Before, those two things would happen simultaneously, without my having to use any of the DIP commands in the dashboard. 

I am not particularly upset about that change in functionality. What concerns me is that we aren't sure why it's changed, the additional directory levels for the DIP feels cumbersome, and for now we've altered the makeup of our identifiers to reflect that change, so they aren't consistent. 

But basically, I just want to confirm if this is something because of the updated version, or does it seem more like we have inadvertently altered our own settings on our end? 

In any case, I am sure this is probably a confusing situation, so I want to stress that I am grateful for any insight or information you might provide. 

Best,

Courtney Tang

--
You received this message because you are subscribed to a topic in the Google Groups "archivematica" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/archivematica/g3-LKAUEbFI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to archivematica+unsubscribe@googlegroups.com.

[Sarah Romkey]

Hi Courtney,

Did someone set up some custom configuration for you previously? Because what you are describing as the previous functionality differs from standard installations. There hasn't been a way to pre-configure DIP storage/upload options, and if choosing the Store DIP option it has always been stored in quad directories in the location you configured in the Storage Service. The uploadedDIPs directory however does not use the quad directories and it is on your server- I wonder if that's what you were relying upon previously? If you choose an access system (CONTENTdm, AtoM, etc) as the DIP upload target, a copy gets sent to uploadedDIPs. The only change in functionality that I can think of is previously if you choose Store DIP a copy of the DIP would also be placed in uploadedDIPs, which is no longer the case- it seemed redundant to store two copies, one in uploadedDIPs and one in whatever DIP storage location you configured in the Storage Service.

I hope this helps clarify, and please let me know if I'm misunderstanding! And if you want to experiment with a 1.4.1 server, sandbox.archivematica.org is still on that version. (login: de...@example.com, password demodemo)

Cheers,

Sarah

Sarah Romkey, MAS,MLIS

Archivematica Program Manager
Artefactual Systems
604-527-2056

@archivematica / @accesstomemory

[Tatiana Canelhas]

Hi Sarah,

Can you please explain with a little more detail why the AIP directories are broken down into UUID quad directories (for efficient storage and retrieval)????

What is the logic behind this?

I need to explain this to my bosses.

Thanks,

Tatiana Canelhas

To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.

To post to this group, send email to archiv...@googlegroups.com.
Visit this group at https://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "archivematica" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/archivematica/g3-LKAUEbFI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to archivematic...@googlegroups.com.

To post to this group, send email to archiv...@googlegroups.com.
Visit this group at https://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "archivematica" group.

To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.

[Sarah Romkey]

Hi Tatiana- here is some detail from the wiki:

"Some file systems limit the number of items allowed in a directory, Archivematica uses a directory tree structure to store AIPs. The tree is based on the AIP UUIDs. The UUID is broken down into manageable 4 character pieces, or "UUID quads", each quad representing a directory. The first four characters (UUID quad) of the AIP UUID will compose a sub directory of the AIP storage. The second UUID quad will be the name of a sub directory of the first, and so on and so forth, until the last four characters (last UUID Quad) create the leaf of the AIP store directory tree, and the AIP with that UUID resides in that directory."

Hope that helps,

Cheers,

Sarah

Sarah Romkey, MAS,MLIS

Archivematica Program Manager

She/her
Artefactual Systems
604-527-2056

@archivematica / @accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/1b79c127-6c10-471c-89eb-2496868ba9d2n%40googlegroups.com.

[Tatiana Canelhas]

Hi Sarah, thanks. I already read this explanation, but I wasn't sure if that was the reason.

So, it's because of some file system limit of items.

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/CAAr2QtvDqzD%3Db33s8i6KbuT8%3D0e9jN%2BWRDeFb9_0qt3zw30yMg%40mail.gmail.com.

[Ross Spencer]

HI Tatiana - just an observer here  - I've never quite got to grips around file-system limits like this myself even though I am a frequent visitor to this page on Wikipedia!. 

The limits might be quite high in practice, but performance drops trying to list directories with long listings. You can often see it manifest through a file-system front-end GUI as you get into thousands of files. I feel this is quicker with programmatic access but without decent indexing, 16^36 is quite a large number (and represents the number of combinations of directory you can have with lots of different UUIDs). A quad-directory has an averaging affect and limits permutations to 16^4 (65535) (if I am calculating these numbers correctly!) and maintains that affect down the tree. So it improves indexing and thus our ability to walk/access it as required. The AIPstore uses the same structure. (I guess that's a lot of potential AIPS we were planning for ;) ) But yep. Limits might come into play, but I think I'd consider performance (and potential user-friendliness) first. 

Best,

Ross