Guymager at 3% after 20 hours

[Jane G]

We are running bitcurator 1.8.12 on virtualbox (mac os sierra host machine).  Started up Guymager to image the contents of an external drive (about 1TB of video files) more than 20 hours ago, but it's still only at 3%.  Even for the size, this seems abnormal -- but I'm new to this process.  There doesn't seem to be a way to increase memory within virtualbox settings.

[Kam Woods]

A few of points:

- You can change the memory settings for the VM *only* when it is powered down.

- Guymager should be memory-constant during the run. If the VM hasn't run out of memory a few minutes it, it's unlikely that memory is the issue.

- To where are you writing the image? The VM itself will auto-expand its disk only to 256GB; once it hits that limit (or the host disk fills up), writes are disallowed.

- Imaging large disks using the VM is, in general, A Bad Idea, as you take a significant performance hit in read/write speeds when operating over the virtualized USB interface.

- In tandem with that: can't say exactly without looking at your logs, but it's possible you're hitting some kind of read/write buffer hangup with VirtualBox. We've seen this before when people try to image large disks using the VM.

Short version: don't try to image large disks using the VM. Even when it *does* work, it's a huge waste of time due to the limitations of the virtualized USB interface. Use a dedicated host, or, failing that, use FTK Imager on your nearest Windows box.

Kam

On Fri, Dec 1, 2017 at 12:53 PM, Jane G <jane.gl...@disneyanimation.com> wrote:

We are running bitcurator 1.8.12 on virtualbox (mac os sierra host machine).  Started up Guymager to image the contents of an external drive (about 1TB of video files) more than 20 hours ago, but it's still only at 3%.  Even for the size, this seems abnormal -- but I'm new to this process.  There doesn't seem to be a way to increase memory within virtualbox settings.

--
You received this message because you are subscribed to the Google Groups "BitCurator Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcurator-users+unsubscribe@googlegroups.com.
To post to this group, send email to bitcurator-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/73262185-fbfa-41c7-986d-248babf004de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jane Glicksman]

Hi Kam -

I'm writing the image to an external drive (2TB).  Yes, it's obvious now that trying to image a large disk is futile.  No Windows box here; we are a Linux/Mac house.   If I use a dedicated host, will there be problems running reports w/n bitcurator in VirtualBox?  I believe I read a post in which this was a problem.

Thank you

On Fri, Dec 1, 2017 at 10:54 AM, Kam Woods <kamw...@gmail.com> wrote:

A few of points:

- You can change the memory settings for the VM *only* when it is powered down.

- Guymager should be memory-constant during the run. If the VM hasn't run out of memory a few minutes it, it's unlikely that memory is the issue.

- To where are you writing the image? The VM itself will auto-expand its disk only to 256GB; once it hits that limit (or the host disk fills up), writes are disallowed.

- Imaging large disks using the VM is, in general, A Bad Idea, as you take a significant performance hit in read/write speeds when operating over the virtualized USB interface.

- In tandem with that: can't say exactly without looking at your logs, but it's possible you're hitting some kind of read/write buffer hangup with VirtualBox. We've seen this before when people try to image large disks using the VM.

Short version: don't try to image large disks using the VM. Even when it *does* work, it's a huge waste of time due to the limitations of the virtualized USB interface. Use a dedicated host, or, failing that, use FTK Imager on your nearest Windows box.

Kam

On Fri, Dec 1, 2017 at 12:53 PM, Jane G <jane.glicksman@disneyanimation.com> wrote:

We are running bitcurator 1.8.12 on virtualbox (mac os sierra host machine).  Started up Guymager to image the contents of an external drive (about 1TB of video files) more than 20 hours ago, but it's still only at 3%.  Even for the size, this seems abnormal -- but I'm new to this process.  There doesn't seem to be a way to increase memory within virtualbox settings.

--
You received this message because you are subscribed to the Google Groups "BitCurator Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcurator-users+unsubscribe@googlegroups.com.
To post to this group, send email to bitcurator-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/73262185-fbfa-41c7-986d-248babf004de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "BitCurator Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bitcurator-users/Wfze5d61LrU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bitcurator-users+unsubscribe@googlegroups.com.

To post to this group, send email to bitcurator-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAAOjFxBRzxtQDh_d4P-RenRauDuxciRn0VzyA21k889ZkcqjwQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Jane Glicksman

Digital Archivist, Animation Research Library

Walt Disney Animation Studios
818.544.4183

[Kam Woods]

Sorry, it's not clear to me what you're asking in "will there be problems running reports w/n bitcurator in VirtualBox?". If you installed BitCurator on a dedicated box, you wouldn't be using VirtualBox.

Kam

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAA5x21YT68XxMV16_irae3SHUQZjs-6%2BgxunHmWqUHkB%2BYjDMQ%40mail.gmail.com.

[Jane Glicksman]

I was hoping to make a disk image using a utility outside of BitCurator and then import the disk image into BitCurator for reporting.  

On Fri, Dec 1, 2017 at 11:38 AM, Kam Woods <kamw...@gmail.com> wrote:

Sorry, it's not clear to me what you're asking in "will there be problems running reports w/n bitcurator in VirtualBox?". If you installed BitCurator on a dedicated box, you wouldn't be using VirtualBox.

Kam

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAAOjFxCaFiijvr40sJc3ZrvUV4JY4hb_ET6fBOexTKcuh7emAg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Kam Woods]

Ok, I understand. I could be wrong, but I think you're headed down a path that's going to waste a lot of your time.

The majority of the work that's done by the reporting tool uses the output of bulk_extractor, which will extract virtually no useful information from video files (but will still take many, many hours to run as it scans every block of the image). 

The remainder of the info produced by the reporting tool is derived from fiwalk. If you want *that* output, you'd be better off just installing and running fiwalk by itself on one of your existing linux boxes.

Kam

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAA5x21ZA008d_bYFsQbsAzFuf3xaFcHbkvDy%3DTXWtg1hj3O2fw%40mail.gmail.com.

[Jane Glicksman]

Hi Kam,

I appreciate your insight -- I feel like a burned up a lot of time already!  I'm looking into fiwalk now to see if the information it produces will be useful to useful to us.  I was most interested in locating duplicate files and derivatives.

Jane

On Fri, Dec 1, 2017 at 12:12 PM, Kam Woods <kamw...@gmail.com> wrote:

Ok, I understand. I could be wrong, but I think you're headed down a path that's going to waste a lot of your time.

The majority of the work that's done by the reporting tool uses the output of bulk_extractor, which will extract virtually no useful information from video files (but will still take many, many hours to run as it scans every block of the image). 

The remainder of the info produced by the reporting tool is derived from fiwalk. If you want *that* output, you'd be better off just installing and running fiwalk by itself on one of your existing linux boxes.

Kam

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAAOjFxBiQcU%3DVBS5tVrEshWAboQisAjPh%2BG2Z56xp4EU%3DJkEpA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Kam Woods]

Sure. In BitCurator (or other Ubuntu boxes) you can also use fslint (preinstalled in BitCurator) to scan the live drive prior to imaging *or* the mounted disk image for exact duplicates. And *that* will not take very long. Also produces a handy exportable list.

Kam

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAA5x21Z12-QifMEezJUaPycChTBH%2BHX8FTPZONnm12Br_qXz1g%40mail.gmail.com.

[Jane Glicksman]

That's perfect -- still learning about all the utilities bundled with BitCurator!

On Fri, Dec 1, 2017 at 12:52 PM, Kam Woods <kamw...@gmail.com> wrote:

Sure. In BitCurator (or other Ubuntu boxes) you can also use fslint (preinstalled in BitCurator) to scan the live drive prior to imaging *or* the mounted disk image for exact duplicates. And *that* will not take very long. Also produces a handy exportable list.

Kam

To view this discussion on the web visit https://groups.google.com/d/msgid/bitcurator-users/CAAOjFxDgfKv6wLdb-xEY7jJmjPH6VggpBvJ9ApUrNrhDtkGBtw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.