LVM - Error: VG xenvg metadata too large for circular buffer

[scakkia]

Hi,
I have another ganeti cluster with 2 nodes and 70 instances running
on.
I know about the limit of drbd (max drbd minor_count=255 == 128
instances), but now I have another problem with LVM (is not ganeti
related).

I can't create any more instances because I reached the max size of
metadata file of lvm2 (near 110KB),
so I can't create any other lvm partition
and I can't "gnt-backup export" too, because I can't create shapshot!

I'm googling about this problem and unfortunately the problem seems
can not be resolved retrostectively:-(

But the problem can originally avoided by indicate a more large size
of metadata file when create volume group:

change in the "Ganeti instalation tutorial" this line:
# pvcreate /dev/sda3

with this one
# pvcreate --metadatasize 1M /dev/sda3

I hope 1 MegaByte is enough!

I hope my experience can help someone else before create "vg" group.

You have some solution for that?

For Ganti Team:
you can update the manual?

Thanks
Daniele

[Guido Trotter]

On Thu, Oct 23, 2008 at 5:52 PM, scakkia <sca...@gmail.com> wrote:

> You have some solution for that?
>

Well, if you're sure your instances and drbds are fine you can run all
instances on one node, repartition the offline node and
replace-disks... Make sure the disks on the online node are fine,
though!!

Guido

[Iustin Pop]

On Thu, Oct 23, 2008 at 09:52:14AM -0700, scakkia wrote:
>
> Hi,

Hi!

> I have another ganeti cluster with 2 nodes and 70 instances running
> on.
> I know about the limit of drbd (max drbd minor_count=255 == 128
> instances), but now I have another problem with LVM (is not ganeti
> related).
>
> I can't create any more instances because I reached the max size of
> metadata file of lvm2 (near 110KB),
> so I can't create any other lvm partition
> and I can't "gnt-backup export" too, because I can't create shapshot!

Hmm, yes, this is a problem.

> I'm googling about this problem and unfortunately the problem seems
> can not be resolved retrostectively:-(
>
> But the problem can originally avoided by indicate a more large size
> of metadata file when create volume group:
>
> change in the "Ganeti instalation tutorial" this line:
> # pvcreate /dev/sda3
>
> with this one
> # pvcreate --metadatasize 1M /dev/sda3
>
> I hope 1 MegaByte is enough!
>
> I hope my experience can help someone else before create "vg" group.
>
> You have some solution for that?

You can try to:
- stop one instance
- activate and mount its disks
- rsync to somewhere else for backups
- remove the instance

Then maybe (not sure) you will be able to gnt-backup. I'm not sure you
will be able even after removal because I've seen LVM break not nicely
in this case (out of metadata). Maybe that was an older version.

> For Ganti Team:
> you can update the manual?

Thanks for the feedback, will do.

But, as a separate note: we don't recommend putting that many virtual
instances on only two nodes.

If a node actually breaks, you will not be able to reestablish
redundancy unless you get another machine. And there are some known
defficiencies when starting up big cluster, just be sure to plan for it
(e.g. speed of the cluster startup, etc.).

regards,
iustin

[scakkia]

Il giorno 23/ott/08, alle ore 19:38, Guido Trotter ha scritto:

For this system with 70 instances I think it's a little dangerous, I
want to stay quiet:-)

I just order the third node and wait for 10 days.
So I can re-create LVM volumes on the old 2 nodes more quietly.

> Guido
Thanks
Daniele

[scakkia]

Hi Iustin,

> You can try to:
> - stop one instance
> - activate and mount its disks
> - rsync to somewhere else for backups
> - remove the instance
>
> Then maybe (not sure) you will be able to gnt-backup. I'm not sure you
> will be able even after removal because I've seen LVM break not nicely
> in this case (out of metadata). Maybe that was an older version.
>

LVM is ok.
lvm version
LVM version: 2.02.07 (2006-07-17)
Library version: 1.02.08 (2006-07-17)
Driver version: 4.7.0

I confim that if you remove one instance then you can gnt-backup.
Then you can re-create the removed instance.

>> For Ganti Team:
>> you can update the manual?
>
> Thanks for the feedback, will do.
>
> But, as a separate note: we don't recommend putting that many virtual
> instances on only two nodes.
>
> If a node actually breaks, you will not be able to reestablish
> redundancy unless you get another machine. And there are some known
> defficiencies when starting up big cluster, just be sure to plan for
> it
> (e.g. speed of the cluster startup, etc.).
>

I just order the third node:-)

> regards,
> iustin

Thanks
Daniele