9.1 and gmirror with GPT?

[fre...@johnea.net]

Hello,

Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table?

Is the fix for this a near term thing, or something in the farther future?

Thanks for any insight!

johnea
_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

[Lucas B. Cohen]

Hi,

On 2012.10.20 20:17, fre...@johnea.net wrote:
> Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table?

I'm curious about what this is about. Could you refer me to an article
or a discussion where this issue is described ?

[Warren Block]

On Sun, 21 Oct 2012, Lucas B. Cohen wrote:

> On 2012.10.20 20:17, fre...@johnea.net wrote:
>> Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table?
>
> I'm curious about what this is about. Could you refer me to an article
> or a discussion where this issue is described ?

The GPT backup partition tables goes at the end of a disk, the same
place gmirror(8) and other GEOM modules keep metadata. If GPT
partitions are created inside a mirror, the backup GPT table is no
longer at the end of the disk. Hiroki Sato created a patch which fixed
the gptboot complaints, but there was concern about the nonstandard
location of the backup table.

At present, MBR partitioning is recommended with gmirror(8).

[fre...@johnea.net]

On 10/21/2012 07:32 AM, Warren Block wrote:
> On Sun, 21 Oct 2012, Lucas B. Cohen wrote:
>
>> On 2012.10.20 20:17, fre...@johnea.net wrote:
>>> Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table?
>>
>> I'm curious about what this is about. Could you refer me to an article
>> or a discussion where this issue is described ?
>
> The GPT backup partition tables goes at the end of a disk, the same
> place gmirror(8) and other GEOM modules keep metadata. If GPT
> partitions are created inside a mirror, the backup GPT table is no
> longer at the end of the disk. Hiroki Sato created a patch which fixed
> the gptboot complaints, but there was concern about the nonstandard
> location of the backup table.
>
> At present, MBR partitioning is recommended with gmirror(8).

Thank you Warren. That sums it up.

Lucas,

I found this blag post informative:
https://koitsu.wordpress.com/2012/09/18/using-freebsd-graid-geom-raid/

There are also several interesting posts on Michael Lucas' blag, such as:
http://blather.michaelwlucas.com/archives/1071

This is a good discussion thread that dives into a specific configuration and the implications:
http://freebsd.1045724.n5.nabble.com/disk-partitioning-with-gmirror-gpt-gjournal-RFC-td4912676.html

I've tried to determine which came first GEOM or GPT. It seems GEOM is actually older, dating from FreeBSD 5, around 2003. While GPT seems to have been integrated with what is now known as UEFI in the later half of that decade.

It also seems "greedy" of GPT to require both the first and last sectors of the disk. This seems to almost guarantee it will have issues with other low level disk formatting tools. Of course, given the history of the "WinTel" partnership, perhaps not interoperating with other tools was a design specification 8-)

https://en.wikipedia.org/wiki/GUID_Partition_Table

In any case, the upcoming wide spread use of UEFI/GPT (i.e. windoze on commodity PCs) compared to the FreeBSD specific nature of GEOM, pretty much insures that it will have to be GEOM that changes to accommodate this conflict.

Even given the denial of who is David and who is Goliath, in the fact that the GEOM developers don't seem to consider this "their" bug:
http://www.freebsd.org/cgi/query-pr.cgi?pr=162147

It seems inevitable that the FreeBSD devs will have to capitulate and find another way to store the GEOM meta-data or we're going to loose the great benefits of whole disk mirrors under GEOM.

[please proceed with tongue in cheek]

This may not occur any time soon, as time progresses at a different rate for BSD than it does with the rest of the world. A great example is this sentence from the Architecture Handbook:

"The Universal Serial Bus (USB) is a new way of attaching devices to personal computers."

Of course USB is roughly 20 years old now 8-)

There are some other great quotes regarding the "new" computer input device, called the "mouse".

[safe to freely operate tongue again]

In any case, it seems my new 9.1-RC2 installation will be MBR partitioned with whole disk GEOM mirror. This motherboard is BIOS based, not UEFI.

It's become fairly de rigueur to accommodate the 4K sector size disks with fdisk and MBR partitioning. As we propel forward into SSDs this may not stay the case.

Any other comments or caveats are very greatly appreciated...

johnea

[fre...@johnea.net]

On 10/21/2012 07:32 AM, Warren Block wrote:
> On Sun, 21 Oct 2012, Lucas B. Cohen wrote:
>
>> On 2012.10.20 20:17, fre...@johnea.net wrote:
>>> Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table?
>>
>> I'm curious about what this is about. Could you refer me to an article
>> or a discussion where this issue is described ?
>
> The GPT backup partition tables goes at the end of a disk, the same
> place gmirror(8) and other GEOM modules keep metadata. If GPT
> partitions are created inside a mirror, the backup GPT table is no
> longer at the end of the disk. Hiroki Sato created a patch which fixed
> the gptboot complaints, but there was concern about the nonstandard
> location of the backup table.
>
> At present, MBR partitioning is recommended with gmirror(8).

Warren,

I've been reading your article on formatting disks in FreeBSD:
http://www.wonkity.com/~wblock/docs/html/disksetup.html

It's a great description of using gpart to create GPT partitions, and using fdisk and bsdlabel to create MBR partitions.

Would you still recommend this method, using fdisk and bsdlabel, for MRB setup?

Do you have any docs on setting up MBR using gpart, to allign for 4K sector size drives?

Thank you for your advice!

[Warren Block]

On Mon, 22 Oct 2012, fre...@johnea.net wrote:

> On 10/21/2012 07:32 AM, Warren Block wrote:
>> On Sun, 21 Oct 2012, Lucas B. Cohen wrote:
>>
>>> On 2012.10.20 20:17, fre...@johnea.net wrote:
>>>> Just wondering if 9.1 will bring any improvement to the situation of
>>>> creating a full disk geom mirror while also using GPT partition table?
>>>
>>> I'm curious about what this is about. Could you refer me to an article
>>> or a discussion where this issue is described ?
>>
>> The GPT backup partition tables goes at the end of a disk, the same
>> place gmirror(8) and other GEOM modules keep metadata. If GPT
>> partitions are created inside a mirror, the backup GPT table is no
>> longer at the end of the disk. Hiroki Sato created a patch which fixed
>> the gptboot complaints, but there was concern about the nonstandard
>> location of the backup table.
>>
>> At present, MBR partitioning is recommended with gmirror(8).
>

> I've been reading your article on formatting disks in FreeBSD:
> http://www.wonkity.com/~wblock/docs/html/disksetup.html
>
> It's a great description of using gpart to create GPT partitions, and using
> fdisk and bsdlabel to create MBR partitions.

Thanks!

> Would you still recommend this method, using fdisk and bsdlabel, for MRB
> setup?

For drives with 512-byte blocks, they are equivalent. Only gpart can
align the bsdlabel partitions to 4K.

> Do you have any docs on setting up MBR using gpart, to allign for 4K sector
> size drives?

This is a copy from the update of the gmirror section I'm planning to
commit to the Handbook. For a single drive, replace mirror/gm0 with
just the drive name, like ada4.

# create the MBR and add a FreeBSD slice
gpart create -s MBR mirror/gm0
gpart add -t freebsd -a 4k mirror/gm0

# create the bsdlabel partitions in slice 1 (s1)
gpart create -s BSD mirror/gm0s1
gpart add -t freebsd-ufs -a 4k -s 2g mirror/gm0s1
gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1
gpart add -t freebsd-ufs -a 4k -s 2g mirror/gm0s1
gpart add -t freebsd-ufs -a 4k -s 1g mirror/gm0s1
gpart add -t freebsd-ufs -a 4k mirror/gm0s1

# put bootcode on the MBR and mark the first slice active
gpart bootcode -b /boot/mbr mirror/gm0
gpart set -a active -i 1 mirror/gm0

# put bootcode on the bsdlabel
gpart bootcode -b /boot/boot mirror/gm0s1

[fre...@johnea.net]

On 2012-10-22 15:54, Warren Block wrote:
> On Mon, 22 Oct 2012, fre...@johnea.net wrote:
>
>> On 10/21/2012 07:32 AM, Warren Block wrote:
>>> On Sun, 21 Oct 2012, Lucas B. Cohen wrote:
>>>
>>>> On 2012.10.20 20:17, fre...@johnea.net wrote:
>>>>> Just wondering if 9.1 will bring any improvement to the situation of creating a full disk geom mirror while also using GPT partition table?
>>>>

>>> At present, MBR partitioning is recommended with gmirror(8).
>>
>> I've been reading your article on formatting disks in FreeBSD:
>> http://www.wonkity.com/~wblock/docs/html/disksetup.html
>>

>> Do you have any docs on setting up MBR using gpart, to allign for 4K sector size drives?
>
> This is a copy from the update of the gmirror section I'm planning to commit to the Handbook. For a single drive, replace mirror/gm0 with just the drive name, like ada4.
>
> # create the MBR and add a FreeBSD slice
> gpart create -s MBR mirror/gm0
> gpart add -t freebsd -a 4k mirror/gm0
>
> # create the bsdlabel partitions in slice 1 (s1)
> gpart create -s BSD mirror/gm0s1
> gpart add -t freebsd-ufs -a 4k -s 2g mirror/gm0s1
> gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1
> gpart add -t freebsd-ufs -a 4k -s 2g mirror/gm0s1
> gpart add -t freebsd-ufs -a 4k -s 1g mirror/gm0s1
> gpart add -t freebsd-ufs -a 4k mirror/gm0s1
>
> # put bootcode on the MBR and mark the first slice active
> gpart bootcode -b /boot/mbr mirror/gm0
> gpart set -a active -i 1 mirror/gm0
>
> # put bootcode on the bsdlabel
> gpart bootcode -b /boot/boot mirror/gm0s1

That helps a lot Warren!

In recent years I've just been creating a swap partition and one big root partition. It just seems as soon as I make all the traditional partitions, one runs out of room.

Do you feel there are any major disadvantages of this approach?

To create a swap and then a root that fills the rest of the disk, must the swap be created first, like this:

gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1

gpart add -t freebsd-ufs -a 4k mirror/gm0s1

Is there any other way to tell gpart to create the / partition using all space except 4G?

Thank You!

johnea

[Warren Block]

On Tue, 23 Oct 2012, fre...@johnea.net wrote:

> In recent years I've just been creating a swap partition and one big root partition. It just seems as soon as I make all the traditional partitions, one runs out of room.
>
> Do you feel there are any major disadvantages of this approach?

Backup of split filesystems can be easier. The traditional
split-filesystem approach kind of separates things by use, and some
people create a separate /home also. The advantage of one big root is
efficient use of free space on small drives.

> To create a swap and then a root that fills the rest of the disk, must the swap be created first, like this:
>
> gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1
> gpart add -t freebsd-ufs -a 4k mirror/gm0s1
>
> Is there any other way to tell gpart to create the / partition using all space except 4G?

I'm afraid it requires one to Use Math(tm). gpart show will at least
show the real capacity of a drive, instead of the diagonally-measured
inflated units used by drive vendors.

[Lucas B. Cohen]

On 2012.10.21 17:44, fre...@johnea.net wrote:
> On 10/21/2012 07:32 AM, Warren Block wrote:
>> On Sun, 21 Oct 2012, Lucas B. Cohen wrote:
>>
>>> On 2012.10.20 20:17, fre...@johnea.net wrote:
>>>> Just wondering if 9.1 will bring any improvement to the situation of
>>>> creating a full disk geom mirror while also using GPT partition table?
>>>

>>> I'm curious about what this is about. Could you refer me to an article
>>> or a discussion where this issue is described ?
>>
>> The GPT backup partition tables goes at the end of a disk, the same
>> place gmirror(8) and other GEOM modules keep metadata. If GPT
>> partitions are created inside a mirror, the backup GPT table is no
>> longer at the end of the disk. Hiroki Sato created a patch which fixed
>> the gptboot complaints, but there was concern about the nonstandard
>> location of the backup table.
>>

>> At present, MBR partitioning is recommended with gmirror(8).
>

> Thank you Warren. That sums it up.
>

> It also seems "greedy" of GPT to require both the first and last sectors
> of the disk. This seems to almost guarantee it will have issues with
> other low level disk formatting tools. Of course, given the history of
> the "WinTel" partnership, perhaps not interoperating with other tools
> was a design specification 8-)

What surprises me is that GEOM mirror provides a logical device that
doesn't abstract the parts that hold its own metadata. It so happens
that GPT wants to use one of those parts, but doesn't creating an MBR
partition that spans the whole provider up to the last logical block
create a similar - but in this case latent - problem, once that last
block is written to by a filesystem living inside that partition ?

[Lucas B. Cohen]

Nevermind, I just got this. It's code working at the physical device
level that gets confused and complains about a missing GPT backup in the
single disks it examines; not code that's working against the provided
GEOM mirror once it's assembled.

My first understanding felt so weird, I knew I was missing something !

I guess Hiroki Sato's patch Warren mentions doesn't answer the "danger
of overwriting gmirror metadata by an "unfriendly" UEFI-BIOS", though.

[fre...@johnea.net]

On 2012-10-23 17:46, Warren Block wrote:
> On Tue, 23 Oct 2012, fre...@johnea.net wrote:

>> To create a swap and then a root that fills the rest of the disk, must the swap be created first, like this:
>>
>> gpart add -t freebsd-swap -a 4k -s 4g mirror/gm0s1
>> gpart add -t freebsd-ufs -a 4k mirror/gm0s1
>>
>> Is there any other way to tell gpart to create the / partition using all space except 4G?
>
> I'm afraid it requires one to Use Math(tm). gpart show will at least show the real capacity of a drive, instead of the diagonally-measured inflated units used by drive vendors.

Thanks for your guidance Warren! I've also been reading a number of threads on the forums on this subject, to which you contributed.

Rather than face the scary prospect of using actual Math(tm) 8-) I was just going to create swap first at 8G, and let the freebsd partition fill the rest of the disk, however as I try to destroy previous non-aligned MBR and gmirror metadata, I'm running into issues:

"mirror/gm0s1 added, but partition is not aligned on 4096 bytes"

Below is a short screen shot of the commands used to destroy and then recreate gmirror. I'm currently running with non-geom ada0 as root, and am attempting to create the aligned partitions of gm0 on ada1.

I'm not sure what to do to cause the MBR scheme to be aligned. Thanks for any feedback!

johnea

orsbackup# mount
/dev/ada0s1a on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)
orsbackup# gmirror status
Name Status Components
mirror/gm0 COMPLETE ada1 (ACTIVE)
orsbackup# gpart destroy -F mirror/gm0
mirror/gm0 destroyed
orsbackup# gpart create -s MBR mirror/gm0
mirror/gm0 created
orsbackup# ls /dev/mirror/
gm0
orsbackup# gpart add -t freebsd -a 4k mirror/gm0
mirror/gm0s1 added, but partition is not aligned on 4096 bytes
orsbackup# gpart show
=> 63 3907029105 ada0 MBR (1.8T)
63 63 - free - (31k)
126 3906994077 1 freebsd [active] (1.8T)
3906994203 34965 - free - (17M)

=> 0 3906994077 ada0s1 BSD (1.8T)
0 3890216960 1 freebsd-ufs (1.8T)
3890216960 16777116 2 freebsd-swap (8G)
3906994076 1 - free - (512B)

=> 63 3907029104 mirror/gm0 MBR (1.8T)
63 63 - free - (31k)
126 3907028979 1 freebsd (1.8T)
3907029105 62 - free - (31k)

=> 0 3907028979 mirror/gm0s1 BSD (1.8T)
0 3890216960 1 freebsd-ufs (1.8T)
3890216960 16777116 2 freebsd-swap (8G)
3906994076 34903 - free - (17M)

[Warren Block]

This is new to me, I have not seen it before. I had the impression that
gpart put the bsdlabel partition table at a misaligned offset so that
the actual filesystems in those partitions would land on an aligned
block, but it never gave that message. That comes from
sys/geom/part/g_part.c, function g_part_ctl_add which starts at line
645:

743 /* Provide feedback if so requested. */
744 if (gpp->gpp_parms & G_PART_PARM_OUTPUT) {
745 sb = sbuf_new_auto();
746 G_PART_FULLNAME(table, entry, sb, gp->name);
747 if (pp->stripesize > 0 && entry->gpe_pp->stripeoffset != 0)
748 sbuf_printf(sb, " added, but partition is not "
749 "aligned on %u bytes\n", pp->stripesize);
750 else
751 sbuf_cat(sb, " added\n");
752 sbuf_finish(sb);
753 gctl_set_param(req, "output", sbuf_data(sb), sbuf_len(sb) + 1);
754 sbuf_delete(sb);
755 }
756 return (0);

[fre...@johnea.net]

On 2012-10-27 14:41, Warren Block wrote:
> On Sat, 27 Oct 2012, fre...@johnea.net wrote:
>
>> On 2012-10-23 17:46, Warren Block wrote:
>>> On Tue, 23 Oct 2012, fre...@johnea.net wrote:

>> orsbackup# gpart add -t freebsd -a 4k mirror/gm0
>> mirror/gm0s1 added, but partition is not aligned on 4096 bytes
>
> This is new to me, I have not seen it before. I had the impression that gpart put the bsdlabel partition table at a misaligned offset so that the actual filesystems in those partitions would land on an aligned block, but it never gave that message. That comes from sys/geom/part/g_part.c, function g_part_ctl_add which starts at line 645:
>
> 743 /* Provide feedback if so requested. */
> 744 if (gpp->gpp_parms & G_PART_PARM_OUTPUT) {
> 745 sb = sbuf_new_auto();
> 746 G_PART_FULLNAME(table, entry, sb, gp->name);
> 747 if (pp->stripesize > 0 && entry->gpe_pp->stripeoffset != 0)
> 748 sbuf_printf(sb, " added, but partition is not "
> 749 "aligned on %u bytes\n", pp->stripesize);
> 750 else
> 751 sbuf_cat(sb, " added\n");
> 752 sbuf_finish(sb);
> 753 gctl_set_param(req, "output", sbuf_data(sb), sbuf_len(sb) + 1);
> 754 sbuf_delete(sb);
> 755 }
> 756 return (0);
>

Thanks Warren!

I ended up just ignoring the not aligned warning from the "gpart add -t freebsd" and went on to add the freebsd-swap and freebsd-ufs partitions with "-a 4k" option.

Do you think I'm aligned?

Thanks!

johnea

orsbackup# gpart create -s BSD mirror/gm0s1
mirror/gm0s1 created
orsbackup# gpart add -t freebsd-swap -a 4k -s 8g mirror/gm0s1
mirror/gm0s1a added
orsbackup# gpart add -t freebsd-ufs -a 4k mirror/gm0s1
mirror/gm0s1b added

orsbackup# gpart bootcode -b /boot/mbr mirror/gm0
bootcode written to mirror/gm0
orsbackup# gpart set -a active -i 1 mirror/gm0
active set on mirror/gm0s1
orsbackup# gpart bootcode -b /boot/boot mirror/gm0s1
bootcode written to mirror/gm0s1

orsbackup#

orsbackup# gpart show
=> 63 3907029105 ada0 MBR (1.8T)
63 63 - free - (31k)
126 3906994077 1 freebsd [active] (1.8T)
3906994203 34965 - free - (17M)

=> 0 3906994077 ada0s1 BSD (1.8T)
0 3890216960 1 freebsd-ufs (1.8T)
3890216960 16777116 2 freebsd-swap (8G)
3906994076 1 - free - (512B)

=> 63 3907029104 mirror/gm0 MBR (1.8T)
63 63 - free - (31k)
126 3907028979 1 freebsd (1.8T)
3907029105 62 - free - (31k)

=> 0 3907028979 mirror/gm0s1 BSD (1.8T)

0 2 - free - (1.0k)
2 16777216 1 freebsd-swap (8.0G)
16777218 3890251760 2 freebsd-ufs (1.8T)
3907028978 1 - free - (512B)

[Warren Block]

On Sat, 27 Oct 2012, fre...@johnea.net wrote:

> I ended up just ignoring the not aligned warning from the "gpart add -t freebsd" and went on to add the freebsd-swap and freebsd-ufs partitions with "-a 4k" option.
>
> Do you think I'm aligned?

...

> => 63 3907029104 mirror/gm0 MBR (1.8T)
> 63 63 - free - (31k)
> 126 3907028979 1 freebsd (1.8T)
> 3907029105 62 - free - (31k)
>
> => 0 3907028979 mirror/gm0s1 BSD (1.8T)
> 0 2 - free - (1.0k)
> 2 16777216 1 freebsd-swap (8.0G)
> 16777218 3890251760 2 freebsd-ufs (1.8T)
> 3907028978 1 - free - (512B)

The slice starts at block 126, and then the swap partition starts an
additional two blocks into the slice, which is block 128, evenly
divisible by 8 (4096 = 512 * 8).

The freebsd-ufs partition starts at 126+16777218, which is also evenly
divisible by 8.

So yes, that looks aligned to me.