filesystem seepage and sector wipes

[Rod Pemberton]

A while back, I had an issue where my USB camera drivers
wouldn't access my camera. The problem turned out to be
some cd-burner software by a major producer of such software
that came with the CD-ROM. It was interfering with the camera
drivers. In the long process of tracking down that issue,
I also noticed the CD software was also archiving *everything*
the filesystem handled. It had some type of quick burn
feature which I assumed was the cause. So, I didn't think
this was intentional. I was glad to be rid of both the camera
driver conflict and "abusive" software archiving my machine.


More recently, I noticed another program allocates far more
filesystem space than it uses. It simply writes a marker at the
end of it's data. In the process of doing so, it allocates some
tracks that have been used but free'd for re-use. I.e., those tracks
have data on them: letters, programs, or whatever else you were
working on. Since the filesystem allocates multiple sectors at a
time and what I was working on was small in size, I've noticed
that this file is picking up 70%, 80%, 90%, of whatever I was
working on. That's not good, IMO.

(fixed font)
<------------------file---------------------------------->
<-program data-><-leak-><-clean-><-leak-><-leak-><-clean->

This can result in seepage or leakage of data. I.e., whatever
accesses that file can access data not intended for it. There are
a number of problems regarding this:

a) what you thought was saved might be leaked upon a security
breach or forensic recovery

b) what you thought you deleted or wiped is actually preserved
in a file you were unaware of


I've pondered when the best time for a filesystem to wipe sectors
is: wipe on allocate, wipe on delete, wipe as time permits, ramdisk
everything and strongly avoid any writes until necessary to prevent
as much data from being placed onto the disk in the first place, to
continously write data to disk to on timed intervals to wipe or
obfuscate as much data as possible ala Win7, etc.


What is the best method to ensure deleted sectors actually get wiped?

With HDs, the sectors don't move around. So, writing to them will
clear the data.

With SSDs, the sectors are moved around for wear-leveling, supposedly.
So, writing to them does not clear the data.

So, SSDs complicate the issue. You really can't be sure you've wiped
the correct sectors. I.e., they read from one sector, and (re)write to
another sector and then renumber the sectors. So, all you know is that
you've written a blank sector, but it might or might not overwrite the
intended data. That data most likely is still on some free'd and renumbered
sector somewhere. I.e., this seems to require entire wipes of all free
tracks and unallocated bytes at the end-of-program sectors.

If you wipe sectors when you allocate, then creating files is slow and
doesn't accomplish much since most of those sectors will be rewritten
with other data, for a HD. For a SSD, the data may be be written
elsewhere and so do nothing to wipe the deleted data.

If you wipe sectors when you delete, it's slow but will clear the data,
for a HD. For a SSD, it may cause excessive sector wear for SSD, and
still do nothing to wipe the deleted data.

Any thoughts on how to prevent filesystem seepage? Any thoughts on
how to do it for both HDs and SSDs? Any thoughts on how to do so
without slowing your filesystem to a crawl?


Rod Pemberton

[s_dub...@yahoo.com]

On May 13, 11:56 pm, "Rod Pemberton" <do_not_h...@notemailntt.cmm>
wrote:

> A while back, I had an issue where my USB camera drivers
> wouldn't access my camera.  The problem turned out to be
> some cd-burner software by a major producer of such software
> that came with the CD-ROM.  It was interfering with the camera
> drivers.  In the long process of tracking down that issue,
> I also noticed the CD software was also archiving *everything*
> the filesystem handled.  It had some type of quick burn
> feature which I assumed was the cause.  So, I didn't think
> this was intentional.  I was glad to be rid of both the camera
> driver conflict and "abusive" software archiving my machine.
>

Yup, the practice seems pandemic.

> More recently, I noticed another program allocates far more
> filesystem space than it uses.  It simply writes a marker at the
> end of it's data.  In the process of doing so, it allocates some
> tracks that have been used but free'd for re-use.  I.e., those tracks
> have data on them: letters, programs, or whatever else you were
> working on.  Since the filesystem allocates multiple sectors at a
> time and what I was working on was small in size, I've noticed
> that this file is picking up 70%, 80%, 90%, of whatever I was
> working on.  That's not good, IMO.
>

It absolutely is not good. But, this possibly could be explained away
due to 'random access' allocation where a random write to some max
block number has the side affect of benign intermediate allocation of
intermediate blocks. CP/M did this, and so probably pcdos. Since way
back, the deletion method was to mark the directory as 'deleted', but
not acually overwrite the file's data sectors, for speed. I'm not to
sure about the allocation algorithm, whether a 'never used' sector is
prefered over a 'deleted' sector (or allocation block, ie cluster).
You can see if 'never used' is prefered, 'deleted' hangs around longer
before reuse. Also in cp/m where allocation is by blocks of
blocksize, if a few bytes are stored in a reused block, remnant
contents can exist in the remainder of the block.

> (fixed font)
> <------------------file---------------------------------->
> <-program data-><-leak-><-clean-><-leak-><-leak-><-clean->
>
> This can result in seepage or leakage of data.  I.e., whatever
> accesses that file can access data not intended for it.  There are
> a number of problems regarding this:
>
> a) what you thought was saved might be leaked upon a security
> breach or forensic recovery
>
> b) what you thought you deleted or wiped is actually preserved
> in a file you were unaware of
>
> I've pondered when the best time for a filesystem to wipe sectors
> is: wipe on allocate, wipe on delete, wipe as time permits, ramdisk
> everything and strongly avoid any writes until necessary to prevent
> as much data from being placed onto the disk in the first place, to
> continously write data to disk to on timed intervals to wipe or
> obfuscate as much data as possible ala Win7, etc.
>

You probably want to do this in more than one way. But journaling
deletions is a start, with enforcement at application termination, and
at multitask-(idle) phases.

> What is the best method to ensure deleted sectors actually get wiped?
>

Writing them with a known bit pattern: E5h, F6h, 0FF0000FFh, or some
such, good enough for civilians.

> With HDs, the sectors don't move around.  So, writing to them will
> clear the data.
>
> With SSDs, the sectors are moved around for wear-leveling, supposedly.
> So, writing to them does not clear the data.
>

Unfortuneately, the wear-leveling algorithms are proprietary, so a
risk assessment needs to be deliberated, istm.

> So, SSDs complicate the issue.  You really can't be sure you've wiped
> the correct sectors.  I.e., they read from one sector, and (re)write to
> another sector and then renumber the sectors.  So, all you know is that
> you've written a blank sector, but it might or might not overwrite the
> intended data.  That data most likely is still on some free'd and renumbered
> sector somewhere.  I.e., this seems to require entire wipes of all free
> tracks and unallocated bytes at the end-of-program sectors.
>
> If you wipe sectors when you allocate, then creating files is slow and
> doesn't accomplish much since most of those sectors will be rewritten
> with other data, for a HD.  For a SSD, the data may be be written
> elsewhere and so do nothing to wipe the deleted data.
>
> If you wipe sectors when you delete, it's slow but will clear the data,
> for a HD.  For a SSD, it may cause excessive sector wear for SSD, and
> still do nothing to wipe the deleted data.
>
> Any thoughts on how to prevent filesystem seepage?  Any thoughts on
> how to do it for both HDs and SSDs?  Any thoughts on how to do so
> without slowing your filesystem to a crawl?
>

SSD's are a lost cause. Some idea of journaling deletions in a couple
of different instances (ways) for HD's.

Steve

> Rod Pemberton

[wolfgang kern]

Rod Pemberton wrote:

> A while back, I had an issue where my USB camera drivers
> wouldn't access my camera. The problem turned out to be
> some cd-burner software by a major producer of such software
> that came with the CD-ROM. It was interfering with the camera
> drivers. In the long process of tracking down that issue,
> I also noticed the CD software was also archiving *everything*
> the filesystem handled. It had some type of quick burn
> feature which I assumed was the cause. So, I didn't think
> this was intentional. I was glad to be rid of both the camera
> driver conflict and "abusive" software archiving my machine.

> More recently, I noticed another program allocates far more
> filesystem space than it uses. It simply writes a marker at the
> end of it's data. In the process of doing so, it allocates some
> tracks that have been used but free'd for re-use. I.e., those tracks
> have data on them: letters, programs, or whatever else you were
> working on. Since the filesystem allocates multiple sectors at a
> time and what I was working on was small in size, I've noticed
> that this file is picking up 70%, 80%, 90%, of whatever I was
> working on. That's not good, IMO.

OTOH, any OS which got at least minimal fail-recovery aka UNDO need
to act like: "save the new copy apart from the old original".
Olde DOS does this always and so add alot fragmentation to its FAT.
My trick to overcome this were once to save a file twice, the first
saving just took a new cluster-set and made the previous storage
obsolete, and assuming that size of the newly stored data didn't
change, a second save-order put it exactly were the old one was.

> (fixed font)
> <------------------file---------------------------------->
> <-program data-><-leak-><-clean-><-leak-><-leak-><-clean->

> This can result in seepage or leakage of data. I.e., whatever
> accesses that file can access data not intended for it. There are
> a number of problems regarding this:

> a) what you thought was saved might be leaked upon a security
> breach or forensic recovery

Yeah, nothing new since a while already :)

> b) what you thought you deleted or wiped is actually preserved
> in a file you were unaware of

that's a different story, we can tell windoze to not keep any copies
in the trash-can ...

also for windoze (dunnu loonix) wipe-tools exists since very long,
even not just meant to make previous data invisible, they can/will
speedup any further access. (windoze-defrag keep all deleted
directory entries forever and so grow and grow this dynamic shit).

> I've pondered when the best time for a filesystem to wipe sectors
> is: wipe on allocate, wipe on delete, wipe as time permits, ramdisk
> everything and strongly avoid any writes until necessary to prevent
> as much data from being placed onto the disk in the first place, to
> continously write data to disk to on timed intervals to wipe or
> obfuscate as much data as possible ala Win7, etc.

Without a special tool on M$-systems the only way to get rid of obsolete
data would be formatting and reinstall windoze (what I do every month
to shrink the OS-Image like reg.exe and track-records to its start-size).

> What is the best method to ensure deleted sectors actually get wiped?

I'd say that overwrite unused data with new data should be enough in
most case. Even really expensive police-department tools may have the
opportunity to read what was stored before the new data (background...)
we can disappoint them by writing several times the same to the disk :)

> With HDs, the sectors don't move around. So, writing to them will
> clear the data.

As said above, writing several times can clear the background totally.

> With SSDs, the sectors are moved around for wear-leveling, supposedly.
> So, writing to them does not clear the data.

> So, SSDs complicate the issue. You really can't be sure you've wiped
> the correct sectors. I.e., they read from one sector, and (re)write to
> another sector and then renumber the sectors. So, all you know is that
> you've written a blank sector, but it might or might not overwrite the
> intended data. That data most likely is still on some free'd and
> renumbered
> sector somewhere. I.e., this seems to require entire wipes of all free
> tracks and unallocated bytes at the end-of-program sectors.

Why wipe before write ? this would just delay everything.

> If you wipe sectors when you allocate, then creating files is slow and
> doesn't accomplish much since most of those sectors will be rewritten
> with other data, for a HD. For a SSD, the data may be be written
> elsewhere and so do nothing to wipe the deleted data.

yes.

> If you wipe sectors when you delete, it's slow but will clear the data,
> for a HD. For a SSD, it may cause excessive sector wear for SSD, and
> still do nothing to wipe the deleted data.

As an (of course delaying) option you can overwrite deleted sectors
with XOR(byte) of the previously stored data to make it all zero.
A single write may not be enough to avoid Forensic Analysis succeed.

> Any thoughts on how to prevent filesystem seepage? Any thoughts on
> how to do it for both HDs and SSDs? Any thoughts on how to do so
> without slowing your filesystem to a crawl?

USB-sticks and SSDs might be wornout too early if we write too often
to it just to have old data deleted forever ...
HDs can be cleared easy, by just write any patterns to deleted sectors
three times.

__
wolfgang