Mac .ds_store files

[L Snider]

Hi Everyone,

Those of you who use Macs and do fixity checks with programs like Bagit, do you find that .ds_store files are a problem? Do you turn them off all together on the computer/devices? What script worked for you?

I am finding that they mess with my checksums when I try to move files from an external hard drive to the computer, as the .ds_store is added and then verification can't work because the oxum is changed (in Bagit).

Cheers

Lisa

[Matt Schultz]

Hi Lisa,

As part of our qa-tools for BagIt-based ingests in MetaArchive we developed a simple Python script for analyzing a target directory of files - looking primarily for characters that are not URL-safe, as well as things like hidden files. Among other problematic file types and file naming conventions .ds_store files can be identified quickly with the script and remedied both prior to "bagging" content and prior to "validating".

There may be other solutions that institutions have cooked up to help manage this particular issue, but you'd be welcome to test and use our script. It does assume a basic familiarity with use of command-line. See below:

https://github.com/MetaArchive/metaarchive-qa-tools

--
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-curati...@googlegroups.com.
To post to this group, send email to digital-...@googlegroups.com.
Visit this group at http://groups.google.com/group/digital-curation.
For more options, visit https://groups.google.com/d/optout.

--

Matt Schultz
Program Manager
Educopia Institute, MetaArchive Cooperative
http://www.metaarchive.org

@metaarchive

matt.s...@metaarchive.org

616-566-3204

[ben_fin...@moma.org]

Hi Lisa,

We've encountered this, as our conservators are Mac users. I've also seen the problem occur with the Thumbs DB file Windows creates. You can disable your workstation from writing .DS_store files to network shares, and there are some drastic measures that allow you to disable it locally. The latter isn't always practical, especially in an environment with various people handling the material prior to ingest.

One partial solution is to simply not view the bag in Finder – until it is on a network share, or some write protected situation – use your terminal instead. While I wouldn't advise using pre-ingest QA to remove any .DS_store files (they could be a legitimate artifact of the source material), if they are not in the manifest, it is a safe bet to automate their removal as a matter of policy.

There's always Linux…

Best,

Ben

[Matt Schultz]

Ben's comments about some conditions under which you might not want to remove .DS_store files are warranted. And his front-end suggestion for managing their creation is great. StackExchange also has a brief thread that puts the issue of deleting these files in some further context.

https://apple.stackexchange.com/questions/69467/consequences-of-deleting-ds-store

So, if your goal is to eventually restore the content on the same or similar Mac system and retain any folder display/view settings and Spotlight comments via Finder then you would want to think twice about removing the files. Otherwise, if the content itself is just of importance for preservation, irrespective of platform, the presence of these files are of little consequence.

The Wikipedia article on .DS_Store files also touches on the extent to which these files have an impact:

https://en.wikipedia.org/wiki/.DS_Store

I'll definitely use this as an opportunity to add to our QA process documentation for BagIt-based ingests and make sure that this further context gets communicated. Whatever the appropriate remedy might be for your .DS_Store files, you can still feel free to make use of the find-bad-files.py script to analyze a set of content to surface their existence.

Thanks,

Matt

[Simon Spero]

Xattr fact?

You might could want to pull the extended metadata out and save it somewhere.  Parallel hierarchy to data/, with one metadata data file per directory?

https://pypi.python.org/pypi/xattr

[Al Matthews]

Do we know, have .DS files changed at all recently, now that spotlight queries are copied to the cloud? This is an obscurish corner case.

Al

On Feb 12, 2015 9:52 AM, "Matt Schultz" <matt.s...@metaarchive.org> wrote:

[L Snider]

HI Matt,

Thanks so much, I will check that out. 

Cheers

Lisa

[L Snider]

Hi Ben,

Yes, I saw a few tools that allow me to disable the ds_store, however I have been told it can make the Mac unstable (when it tries to read the files without it).

Hmm, so I am using the python window to run bagit.py. I have to open the external to see where the file was put, but other than that I haven't been opening it in the finder. The ds_store I was having issues with was definitely created during this process. Although my testing on Friday revealed no ds_store created-so it may have been a weirdness with my computer. The tests all came back clean, no checksum issues at all.

I am trying to go PC/Linux, but won't have that for a while...however then I have the thumbs issue too!

Thanks again,

Lisa

Lisa Snider

Archivist

Canadian Museum for Human Rights

All views are my own!

[L Snider]

Hi Matt,

Thanks for the link!

The goal may be to restore on a Mac or PC, we use both here. That is why this one is tough, remove and possibly face weirdness with a Mac.

I am definitely going to play with that script, thanks again.

Cheers

Lisa

[L Snider]

Hi Simon,

Cool, thanks for the link! Looks very interesting.

Cheers

Lisa

Lisa Snider

Archivist

Canadian Museum for Human Rights

All views are my own!

[L Snider]

Hi Al,

Good question! From my research, they seem to work in a similar way but I am not 100% sure.

Cheers

Lisa

Lisa Snider

Archivist

Canadian Museum for Human Rights

All views are my own!

[Chris Adams]

To clarify, nothing breaks without a .DS_Store file – they're only used to store custom desktop settings and the most which would happen without them is that you'd lose a custom background or sort order. Many years ago I managed to find a few corrupt ones which cause the Finder to crash but Apple fixed that relatively early in the OS X development cycle.

Resource forks (i.e. ._ files on non-HFS filesystems) are far more of a concern because classic Mac applications often stored important user data in them – the classic example being text documents where the regular file fork had only plain text but the resource fork contained styling, images, etc. which are critical for displaying the document as actually authored. Developers have been migrating away from that approach for years but I would be extremely hesitant not to preserve them unless you are certain that none of the apps in question use them. Cross-platform apps are usually fine since multiple streams per filename was rare outside of Mac circles[1] but e.g. some of the scientists which I used to work with created journal figures using an obscure graphics package which had never been ported to another platform and would refuse to open a document with a missing or empty resource fork. 

Chris

1. Technically, this feature existed on other platforms (Windows NT, OS/2, Novell, etc.) but other implementations weren't commonly used by developers the way they were on Mac OS; the strongest reasons to implement it was usually so the OS could be used a file server for Mac desktops. These days, the uses tend to be of minimal preservation value – e.g. storing the flag used for the “This file was downloaded from the internet. Are you sure you want to run it?” prompts or caching icons. 

[L Snider]

HI Chris,

Thanks so much for clarifying that for me, makes sense with the OSX updates. Very good to know about resource forks as well, thanks!

That was great, it makes a lot of sense of some of the things I have been researching.

Cheers

Lisa

[Jay Gattuso]

Hi All,

Thanks for your various contributions to this thread. Its a really interesting problem, and one that we've been dealing with also. Its been really useful to see how other folks are addressing this problem.

Our current approach is to not ingest the .DS_Store files, as they are not regarded by as an artefact of the collection, more as an artefact of the system the collection came from.

Of course this view changes the moment we regard the original file store to be the collection, rather than an arrangement mechanism for the collection. That is to say, the moment we have a stated intent to capture the way a donor viewed their files on their original machine, these files become a collection item in their own right.  How we might go about interpreting that information in a meaningful way is interesting  question – is there a “decode” method for .DS_store files that exposes thumbnails and viewer rules?

The larger impact we have with them is the way they are handled differently by various file handling operations - this is in part I guess because of their "hidden" nature from a system perspective, but also because some tools are built to ignore them as far as we can tell. This means we can end up with different file counts when working on complex sets, resulting in a "find the delta" mission that can consume many hours as we seek to ensure that the missing files are not really missing and can be processed properly.

We have a related set of concerns with thumbs.db and desktop.ini. We also have a whole other set of dramas with MAC OS resource forks…

 

Thanks,

 

J  

Jay Gattuso | Digital Preservation Analyst | Preservation, Research and Consultancy

National Library of New Zealand | Te Puna Mātauranga o Aotearoa

[Chris Adams]

On Feb 17, 2015, at 3:55 PM, Jay Gattuso <jay.g...@googlemail.com> wrote:
> Of course this view changes the moment we regard the original file store to be the collection, rather than an arrangement mechanism for the collection. That is to say, the moment we have a stated intent to capture the way a donor viewed their files on their original machine, these files become a collection item in their own right. How we might go about interpreting that information in a meaningful way is interesting question – is there a “decode” method for .DS_store files that exposes thumbnails and viewer rules?

I’m not aware of a built-in command-line tool to view the contents but there do appear to be some open-source libraries:

https://pypi.python.org/pypi/ds_store/1.0.1
http://search.cpan.org/~wiml/Mac-Finder-DSStore/DSStoreFormat.pod

The docs for ds_store seem fairly sane:

http://ds-store.readthedocs.org/en/latest/#usage

… but, of course, be prepared to find something unexpected since Apple considers this a proprietary format and I wouldn’t be surprised to see things change in future OS releases.

Chris

[Jay Gattuso]

Thanks! thats great, I'm working on a collection at the moment that have these resource files - I'll have a poke and see what I can see....