Exporting files using the disk image access tool

[Jessica Moran]

Hi all,

I've noticed that when I export individual files from a disk image using the Bitcurator access tool that the process doesn't preserve the original last access and last modified date properties of the file. Instead I get the date of the export as the new dates. Am I doing something wrong? We've been experimenting with using a script to move the files and preserve the dates, but I'm wondering if I've maybe I've just missed something or not got my settings set up right? Anyone else noticed something similar?

Thanks,
Jessica Moran

[Cal Lee]

The timestamps are subject to change whenever files are written to a new
filesystem. This is true in any computing environment, not just the
BitCurator environment. The timestamps in the DFXML (fiwalk) metadata
will reflect those of the original disk, assuming you used a write
blocker and then created a disk image.

This volatility of the filesystem metadata is one of the primary
motivations for both creating forensic disk images and for exporting the
filesystem into a static representation (the DFXML generated when you
"run all" in the BitCurator reporting tools or run fiwalk
independently). The DFXML timestamp values will remain fixed and can
serve as a reference point.

- Cal

[Alex Nelson]

It is possible to use `touch` to adjust timestamps.  Is this an interface option BC should offer?

--Alex

--
You received this message because you are subscribed to the Google Groups "BitCurator Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcurator-use...@googlegroups.com.
To post to this group, send email to bitcurat...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/54E55890.2040002%40email.unc.edu.
For more options, visit https://groups.google.com/d/optout.

[Kam Woods]

Background reference: the disk image access tool was designed only to provide access (not preservation) copies - things you could share, move around, allow people to download, and so on.

As Alex notes, you can recreate the original timestamps of the original file (as it exists in the source file system) on the new file (as it exists it the newly exported location), but this is messy and introduces a potential for additional error.

A critical question is "What are you doing with the files next?" If you have a tool that, say, prepares a SIP by examining the files individually and creating its own metadata then...ok, yes, this might be useful. But that metadata is already in the DFXML (as Cal mentioned), and would be relatively simple to parse out without taking this additional step.

If you're going to copy them somewhere else, the timestamps will be lost in *that* step (unless you're doing something like rsync with timestamp preservation). If you're going to Bag select files (say, serializing into a zip file) then there may be additional snags. My understanding (and I'm happy to be corrected) is that Bag-capable libraries that use the zip specification for serialization have two-second precision and an earliest timestamp of 1980-01-01 00:00:00 (local), which is fine for FAT volumes but may be problematic for others.

These are just examples, of course, but the general rule is that the further you get from the source material, the higher the risk of contamination or alteration. It would be helpful to know what the next step in your workflow is...

Kam

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAFdnda-PLjkH3xZptLCEAA8SDFO0tsAxn7qQo%2B-JrS0CnmPtMA%40mail.gmail.com.

[Jessica Moran]

Thanks Cal and Kam,

To give a little more information about our workflow, yes our end goal is to deposit the individual files into our preservation system (rosetta) which does include a tool to examine individual files and extract the metadata, so we do want to preserve as much of that metadata with the file as possible, rather than having to parse it out and link to the objects as a separate step.

We were experimenting a little yesterday with moving files from a disk image and did find that using python the shutil.copy2 command could move files to a new location while preserving most of the metadata, including the timestamps. So if there are methods of copying (or exporting) files from a disk image packaged into BitCurator, why not use a method that seeks to preserve that original metadata with the object rather than a method that doesn't? I can understand that the tool was just designed to provide access to the files, but I think there are probably going to be cases where once doesn't want to preserve the disk image and the individual files will become the preservation master.

Jessica 

To unsubscribe from this group and stop receiving emails from it, send an email to bitcurator-users+unsubscribe@googlegroups.com.
To post to this group, send email to bitcurator-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/54E55890.2040002%40email.unc.edu.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BitCurator Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcurator-use...@googlegroups.com.
To post to this group, send email to bitcurat...@googlegroups.com.

[Kam Woods]

It's on the near-term todo list now.

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/82218372-fef2-43b9-a419-7177565d7c84%40googlegroups.com.

[Jessica Moran]

cool :)

[Jarrett Drake]

Hi Kam,

I'm following up to see if there have been any new developments related to this thread. It's certainly something we're considering now that we've completed a survey of all our legacy media dating back to the late 80's.

Best,

Jarrett

[Kam Woods]

Hi Jarrett,

As you may have noticed on GitHub, I began cleaning up and revising the existing BitCurator code a couple of weeks ago. This update is still on the to-do list.

Regards,

Kam

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/1799430d-3bee-4b42-a79a-0392206d3aac%40googlegroups.com.

[Jessica Moran]

We've been developing a script Jarrett that you, and maybe others, will find useful for exporting from BitCurator and just generally moving files around. It's still in development, but you can find it here https://github.com/jayGattusoNLNZ/Safe_mover

Jessica

[Kam Woods]

We can certainly add this and include a setuptools installer for it in the BitCurator environment when it's ready, and if the license permits.

I've mentioned this on-list before, but for those who might be fuzzy on why you'd use the BitCurator Disk Access Tool versus mounting and simply copying files out, the answer is simply that BDAT provides access to possible deleted fragments on disk, along with their file metadata, and extracts what it can directly from the bitstream. No mount and no file copy operations are involved - even resident files are simply reconstructed when an export is requested. This is one of the reasons we don't currently do a "file permission preserving" cp.

In short, if your use case is "mount the disk image and copy files out of it", there's no particular reason to use BDAT unless you want to examine the disk for possible deleted materials. In the future, BDAT will provide additional export and reporting options, but for right now the "file permission preserving export" is the only update in the work queue.

Kam

On Sat, Jun 13, 2015 at 11:49 PM, Jessica Moran <jmm...@sbcglobal.net> wrote:

We've been developing a script Jarrett that you, and maybe others, will find useful for exporting from BitCurator and just generally moving files around. It's still in development, but you can find it here https://github.com/jayGattusoNLNZ/Safe_mover

Jessica

--
You received this message because you are subscribed to the Google Groups "BitCurator Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcurator-use...@googlegroups.com.
To post to this group, send email to bitcurat...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/7b708266-66ad-40d7-8d8e-9331e0ae2ca5%40googlegroups.com.

[Jarrett Drake]

Thanks Jessica and Kam for your responses.

Jessica, thank you for sharing the link; I'll take a look at it and will likely incorporate some aspect of it. We wrote a bash script to accomplish some of the things your python script does; when it's ready for GitHub, I'll share the link here.

Kam, I preempted your advice and removed the BDAT step from our workflow because we aren't interested in recovering deleted files at the time of accessioning. Even if we wanted to do so later, we would refer back to the disk image and use BDAT then. For anyone interested, attached you will find documentation for how we conclude the step preceding the disk image as well as disk imaging itself.

Jarrett

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAAOjFxBAcFVPbj9NQcE3%3DDpKdiv3_L3iJuznV1MQPceDmBQt3g%40mail.gmail.com.