Destroy GPT partition scheme absolutely, how?

[Hartmann, O.]

I ran into a very nasty and time consuming problem. Creating a NanoBSD
image with a modified script framework creating GPT partitions, I put
the imaes via "dd(1)" on USB flash or SD flash. Because the images are
usually much smaller than the overall capacity of the USB or SD, the OS
(FreeBSD CURRENT, recently built as of this morning) complains about
the second GPT header isn't in the last LBA. Sometimes, my PCengines
APU2 doesn't boot then, a relief is to issue the command

gpart recover da1

(in that case, the USB flash drive or SD flash is recognized
as /dev/da1).

But I run into a nasty situation, if the image put to the flash is
somehow corrupted. Then I tried to write a second, repaired image over
the first one using dd(1) again and do a recovering as mentioned above
- but this is fatal in two ways. First, the corrupted/broken GPT seems
to be "recovered" and put in replacement of the correct one - so I
guess. Performing no recover leaves the image on flash corrupted
anyway.

Well, to be honest, I didn't exactly know what is going on here. The
phenomenon is that I had a problem creating a NANO_DATASIZE= DATA
partition with an empty NANO_DATASIZE which somehow corrupted the
whole image. The image then never booted, complaining,
that /foo/bar/_.mnt was unmounted unleanly.

This happened multiple times, even if I tried to overwrite the SD or
USB flash with /dev/zero or /dev/random data, but I do stop such a dd
after a couple f minutes, since the SD is 32GB in size and the USB
flash drive is 32 GB, 64 GB and 128 GB - a pain in the ass if you want
to write via USB 2.0. But even with overwriting with a good image then
results in a corrupt image on flash drive, complaining about the GPT
second header not in last LBA and the issue with the uncleanly
unmount _.mnt (from the creation process of the NanoBSD image)!

So I guess there is something magic happening. Some informations are
not lost and I suspect the "recovery" moving those foul data into
active places.

Using a fresh/new SD or USB resolves the problem. But the question
remains: how can I destroy any relevant GPT information on a Flash
drive (or even harddisk) to avoid unwanted remains of an foul image
installation?

First guess was to write the last couple of bytes on such a flash drive
by letting dd(1) counting backwards, but I couldn't figure out how to
let dd(1) do such a procedure. The nightmare didn't end, while trying,
the SD flash card died :-(

thank you very much for your help and thoughts.

Kind regards, thanks in advance,

oliver
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Miroslav Lachman]

Hartmann, O. wrote on 09/26/2016 15:01:

[...]

> Using a fresh/new SD or USB resolves the problem. But the question
> remains: how can I destroy any relevant GPT information on a Flash
> drive (or even harddisk) to avoid unwanted remains of an foul image
> installation?
>
> First guess was to write the last couple of bytes on such a flash drive
> by letting dd(1) counting backwards, but I couldn't figure out how to
> let dd(1) do such a procedure. The nightmare didn't end, while trying,
> the SD flash card died :-(

I use dd if=/dev/zero almost everytime when I need to install system on
used HDD.

I rewrite about 10MB on its start and then about 10MB on the end.
I have some script to do this - simply takes media size from diskinfo
command, substract 10MB and then dd with seek

Hope that it will help you

Miroslav Lachman

[Ernie Luzar]

This little script has been posted before. Maybe it will be what your
looking for. Called gpart.nuke

#! /bin/sh
echo "What disk do you want"
echo "to wipe? For example - da1 :"
read disk
echo "OK, in 10 seconds I will destroy all data on $disk!"
echo "Press CTRL+C to abort!"
sleep 10
diskinfo ${disk} | while read disk sectorsize size sectors other
do
# Delete MBR and partition table.
dd if=/dev/zero of=/dev/${disk} bs=${sectorsize} count=1
# Delete GEOM metadata.
dd if=/dev/zero of=/dev/${disk} bs=${sectorsize} oseek=`expr $sectors
- 2` count=2
done

[Gary Jennejohn]

I'm surprised that this script works without
sysctl kern.geom.debugflags=16

--
Gary Jennejohn

[Nikolai Lifanov]

You could just gpart destroy/create/destroy instead of doing arithmetic.

- Nikolai Lifanov

[Ngie Cooper]

> On Sep 26, 2016, at 22:48, Ernie Luzar <luza...@gmail.com> wrote:

..

> This little script has been posted before. Maybe it will be what your looking for. Called gpart.nuke
>
> #! /bin/sh
> echo "What disk do you want"
> echo "to wipe? For example - da1 :"
> read disk
> echo "OK, in 10 seconds I will destroy all data on $disk!"
> echo "Press CTRL+C to abort!"
> sleep 10
> diskinfo ${disk} | while read disk sectorsize size sectors other
> do
> # Delete MBR and partition table.
> dd if=/dev/zero of=/dev/${disk} bs=${sectorsize} count=1
> # Delete GEOM metadata.
> dd if=/dev/zero of=/dev/${disk} bs=${sectorsize} oseek=`expr $sectors - 2` count=2
> done

Why not just use "gpart destroy -F provider"?

Cheers,
-Ngie

[John Baldwin]

On Tuesday, September 27, 2016 12:36:22 AM Ngie Cooper wrote:
>
> > On Sep 26, 2016, at 22:48, Ernie Luzar <luza...@gmail.com> wrote:
>

> ...

>
> > This little script has been posted before. Maybe it will be what your looking for. Called gpart.nuke
> >
> > #! /bin/sh
> > echo "What disk do you want"
> > echo "to wipe? For example - da1 :"
> > read disk
> > echo "OK, in 10 seconds I will destroy all data on $disk!"
> > echo "Press CTRL+C to abort!"
> > sleep 10
> > diskinfo ${disk} | while read disk sectorsize size sectors other
> > do
> > # Delete MBR and partition table.
> > dd if=/dev/zero of=/dev/${disk} bs=${sectorsize} count=1
> > # Delete GEOM metadata.
> > dd if=/dev/zero of=/dev/${disk} bs=${sectorsize} oseek=`expr $sectors - 2` count=2
> > done
>
> Why not just use "gpart destroy -F provider"?

That doesn't always work. In particular, if a disk was partitioned with GPT
and then you use normal MBR on it afterwards, the 'gpart destroy -F' of the
MBR will leave most of the GPT intact and the disk will come up with the old
GPT partitions, not as a raw disk.

--
John Baldwin

[O'Connor, Daniel]

> On 27 Sep 2016, at 06:21, John Baldwin <j...@FreeBSD.org> wrote:
> That doesn't always work. In particular, if a disk was partitioned with GPT
> and then you use normal MBR on it afterwards, the 'gpart destroy -F' of the
> MBR will leave most of the GPT intact and the disk will come up with the old
> GPT partitions, not as a raw disk.

I wonder how feasible it would be to have a command which runs destroy for every known partition scheme on a particular device..

Sure there would be some duplicate zeroing but it's not likely to be significantly slower and considerably more robust.

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C

[O. Hartmann]

On Mon, 26 Sep 2016 09:48:22 -0400
Ernie Luzar <luza...@gmail.com> wrote:

Thank you very much.

[O. Hartmann]

On Tue, 27 Sep 2016 00:36:22 +0900
Ngie Cooper <yaneu...@gmail.com> wrote:

> > On Sep 26, 2016, at 22:48, Ernie Luzar <luza...@gmail.com> wrote:
>

> ...

Yes, why not :-(

Maybe panic and too much swet on the forehead ... ;-) Thanks.

[Warner Losh]

On Mon, Sep 26, 2016 at 8:46 PM, O'Connor, Daniel <dar...@dons.net.au> wrote:
>
>> On 27 Sep 2016, at 06:21, John Baldwin <j...@FreeBSD.org> wrote:
>> That doesn't always work. In particular, if a disk was partitioned with GPT
>> and then you use normal MBR on it afterwards, the 'gpart destroy -F' of the
>> MBR will leave most of the GPT intact and the disk will come up with the old
>> GPT partitions, not as a raw disk.
>
> I wonder how feasible it would be to have a command which runs destroy for every known partition scheme on a particular device..
>
> Sure there would be some duplicate zeroing but it's not likely to be significantly slower and considerably more robust.

dd of 2MB of zeros to the start and end of the disk. That will destroy
pretty much everything. For SSDs, sometimes you can do the same with
TRIMs only faster (other times they are slower or unreliable).

Warner

[O'Connor, Daniel]

> On 27 Sep 2016, at 14:28, Warner Losh <i...@bsdimp.com> wrote:
> dd of 2MB of zeros to the start and end of the disk. That will destroy
> pretty much everything. For SSDs, sometimes you can do the same with
> TRIMs only faster (other times they are slower or unreliable).

Yeah, but it would be nicer to not have to know that particular magic incarnation :)

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C

[Warner Losh]

On Mon, Sep 26, 2016 at 11:06 PM, O'Connor, Daniel <dar...@dons.net.au> wrote:
>
>> On 27 Sep 2016, at 14:28, Warner Losh <i...@bsdimp.com> wrote:
>> dd of 2MB of zeros to the start and end of the disk. That will destroy
>> pretty much everything. For SSDs, sometimes you can do the same with
>> TRIMs only faster (other times they are slower or unreliable).
>
> Yeah, but it would be nicer to not have to know that particular magic incarnation :)

Disk formatting has always been 3 parts magic, 2 parts luck and 1 part skill :)

It doesn't fit nicely into geom because metadata can live elsewhere.

I forgot to add the caveat not to try this on a disk that is part of a
RAID volume.

Warner

[Andrey V. Elsukov]

On 26.09.2016 23:51, John Baldwin wrote:
>> Why not just use "gpart destroy -F provider"?
>
> That doesn't always work. In particular, if a disk was partitioned with GPT
> and then you use normal MBR on it afterwards, the 'gpart destroy -F' of the
> MBR will leave most of the GPT intact and the disk will come up with the old
> GPT partitions, not as a raw disk.

If you would did `gpart destroy -F` for GPT, then it would not have
appeared again.
This is very strange problem, how did you created MBR if you have not
destroyed GPT? :)
After this commit it seems very unlikely to reproduce
https://svnweb.freebsd.org/base?view=revision&revision=292057

Described problem can be reproduced with BSD label.

--
WBR, Andrey V. Elsukov

[Warner Losh]

On Wed, Sep 28, 2016 at 1:02 PM, Allan Jude <alla...@freebsd.org> wrote:

> I wonder if this issue is related at all to the new 'auto resize' gpart
> bits. That leaves the 'uncommitted' transaction pending, and may require
> a 'gpart undo' before the other commands will work correctly.
>
> I wonder if something like 'zpool labelclear', but for gpart would be
> useful, that just nukes the first and last few MB of the disk. I know in
> the installer we jump through a number of hoops to try to clear out old
> stuff, and having one command would be better.

I thought the auto resize stuff was backed out, precisely because it
left bogons around...

[Andrey V. Elsukov]

On 28.09.2016 22:02, Allan Jude wrote:
> I wonder if this issue is related at all to the new 'auto resize' gpart
> bits. That leaves the 'uncommitted' transaction pending, and may require
> a 'gpart undo' before the other commands will work correctly.

All other commands that do any changes will issue 'commit', so they will
work correctly. I think you are confusing 'auto resize' with 'recovery
needed'.

[Andriy Gapon]

On 28/09/2016 21:08, Andrey V. Elsukov wrote:
> This is very strange problem, how did you created MBR if you have not
> destroyed GPT? :)

Using a tool that's not aware of GPT at all?

--
Andriy Gapon

[John Baldwin]

On Thursday, September 29, 2016 01:01:44 AM Andriy Gapon wrote:
> On 28/09/2016 21:08, Andrey V. Elsukov wrote:
> > This is very strange problem, how did you created MBR if you have not
> > destroyed GPT? :)
>
> Using a tool that's not aware of GPT at all?

I think the drive's arrived with some partition scheme on them from
burn-in testing or the like and when trying to be reinstalled we could
end up "exposing" the GPT when the MBR was destroyed or something like
that.

--
John Baldwin

[Warren Block]

On Mon, 26 Sep 2016, John Baldwin wrote:

> On Tuesday, September 27, 2016 12:36:22 AM Ngie Cooper wrote:
>>
>>> On Sep 26, 2016, at 22:48, Ernie Luzar <luza...@gmail.com> wrote:
>>
>> ...
>>
>>> This little script has been posted before. Maybe it will be what your looking for. Called gpart.nuke
>>>
>>> #! /bin/sh
>>> echo "What disk do you want"
>>> echo "to wipe? For example - da1 :"
>>> read disk
>>> echo "OK, in 10 seconds I will destroy all data on $disk!"
>>> echo "Press CTRL+C to abort!"
>>> sleep 10
>>> diskinfo ${disk} | while read disk sectorsize size sectors other
>>> do
>>> # Delete MBR and partition table.
>>> dd if=/dev/zero of=/dev/${disk} bs=${sectorsize} count=1
>>> # Delete GEOM metadata.
>>> dd if=/dev/zero of=/dev/${disk} bs=${sectorsize} oseek=`expr $sectors - 2` count=2
>>> done
>>
>> Why not just use "gpart destroy -F provider"?
>
> That doesn't always work. In particular, if a disk was partitioned with GPT
> and then you use normal MBR on it afterwards, the 'gpart destroy -F' of the
> MBR will leave most of the GPT intact and the disk will come up with the old
> GPT partitions, not as a raw disk.

Right. So do a gpart destroy -F of whatever is on there, ignoring
errors, then a gpart create -s gpt. Now there is definitely a secondary
GPT, and a final gpart destroy -F removes it cleanly.

[John Baldwin]

It is usually simpler to just dd zeroes over the first N and last N sectors.
The only fool-proof way to ensure you don't have dangling tables in the
middle of the disk is to zero the entire disk, but that takes too long.

--
John Baldwin

[Maxim Sobolev]

For the whole disk destruction, hopefully one day we'd have BIO_DELETE
coalesce code, so that you can batch of lot of operations into handful SATA
commands. I've heard rumours imp@ was doing something along those lines. As
well as SSD disks smart enough to process those requests in the background.
Anyway, just saying. )

-Max

[Steven Hartland]

CAM already does this, doesn't really help with speed as much as you
might think though.

On 04/10/2016 19:53, Maxim Sobolev wrote:
> For the whole disk destruction, hopefully one day we'd have BIO_DELETE
> coalesce code, so that you can batch of lot of operations into handful SATA
> commands. I've heard rumours imp@ was doing something along those lines. As
> well as SSD disks smart enough to process those requests in the background.
> Anyway, just saying. )
>