Some questions about HA and metadata mirroring

[Xavi]

HI,

I've a few quetions about buddy mirrors. I've configured 2 nodes with 1 metadata targets each one and 2 storage targets each one. Then I've configured two buddymirrors with a target storage from one node an another target storage from the other node. From the client I've mounted the beegfs and seems all works ok. Then I've created a directory with metadata mirroring and buddymirror:

Path: /dir_bak
Mount: /mnt/beegfs
EntryID: 0-56CDCA7F-1
Metadata node: biosca1 [ID: 1]
Metadata mirror node: biosca2 [ID: 2]
Stripe pattern details:
+ Type: Buddy Mirror
+ Chunksize: 1M
+ Number of storage targets: desired: 2

Now I want to do some tests about HA:

I've done a dd if=/dev/zero of=/mnt/beegfs/dir_bak/test1 bs=4096 count=1000000 and it finish ok, but if I want to do a dd if=/mnt/beegfs/dir_bak/test1 of=/mnt/beegfs/dir_bak/test2 and stop the storage server on node2 the comand does not finish never. I can see in the mgmt.log that the targets on node2 are set offline. Do you what could be happening?

Another thing is when I try to restore a metadata target form the mirror. The same directory is configured with metadata mirror support, so if I remove the metadata directory target on node1 then is no t possible to restore from the node2. I've tried the 3 ways but has not work.

I try to explain what I'm doing. Once I've deleted the metadata directory on node1 I do a star of the metadata direcotory of node2 and copy the tar file to node1. Then I extract the tar file star, but what I have is al the content of node1 metadata directory and directories of dentries and inodes of node2 inte mirror dir. If I move mirror/2.dentries and mirror/2.inodes to dentries and inodes in the metadir does not work.

Do I have missing something?

Thanks in advance,

Xavi.

[Xavi]

Anyone that has been able to restore a metadata server can explain how to?

Thanks

[Sven Breuner]

Hi Xavi,

Xavi wrote on 25.02.2016 12:29:
> HI,
>
> I've a few quetions about buddy mirrors. I've configured 2 nodes with 1 metadata
> targets each one and 2 storage targets each one. Then I've configured two
> buddymirrors with a target storage from one node an another target storage from
> the other node. From the client I've mounted the beegfs and seems all works ok.
> Then I've created a directory with metadata mirroring and buddymirror:
>
> Path: /dir_bak
> Mount: /mnt/beegfs
> EntryID: 0-56CDCA7F-1
> Metadata node: biosca1 [ID: 1]
> Metadata mirror node: biosca2 [ID: 2]
> Stripe pattern details:
> + Type: Buddy Mirror
> + Chunksize: 1M
> + Number of storage targets: desired: 2
>
> Now I want to do some tests about HA:
>
> I've done a dd if=/dev/zero of=/mnt/beegfs/dir_bak/test1 bs=4096 count=1000000
> and it finish ok, but if I want to do a dd if=/mnt/beegfs/dir_bak/test1
> of=/mnt/beegfs/dir_bak/test2 and stop the storage server on node2 the comand
> does not finish never. I can see in the mgmt.log that the targets on node2 are
> set offline. Do you what could be happening?

The BeeGFS metadata service is not HA yet, only the storage service is. I guess
this is the problem/misunderstanding here. Metadata HA based on a buddy mirror
mechanism similar to what is already available for storage targets is currently
work in progress and we hope to have a first beta release with metadata buddy
mirroring available soon.

> Another thing is when I try to restore a metadata target form the mirror. The
> same directory is configured with metadata mirror support, so if I remove the
> metadata directory target on node1 then is no t possible to restore from the
> node2. I've tried the 3 ways but has not work.
>
> I try to explain what I'm doing. Once I've deleted the metadata directory on
> node1 I do a star of the metadata direcotory of node2 and copy the tar file to
> node1. Then I extract the tar file star, but what I have is al the content of
> node1 metadata directory and directories of dentries and inodes of node2 inte
> mirror dir. If I move mirror/2.dentries and mirror/2.inodes to dentries and
> inodes in the metadir does not work.

I guess the problem here is that you copied all contents of the other server's
metadata directory, including the files in the top-level directory (e.g.
nodeNumID). It's important that the server keeps its original node ID, so that
BeeGFS can identify the server correctly.
So you should either make a backup of the files in the top-level directory or
after deletion initialize the directory with the same ID that you originally
used with "beegfs-setup-meta" and make sure that the ID files are not
overwritten with values from the other server when you copy the inodes and
dentries data.

Best regards,
Sven

[Xavi]

Hi Sven, thanks for the answer but I think I must explain better...

I'm aware that there is only HA for storage targets. When I'm talking about stop the storage server I want to say that I disable the storage targets located in the node. When I stop the storage server the storage targets stop working and I expect the system have to work again once the mgmt set as offline but this does not happen in the situation commented before.
 

> Another thing is when I try to restore a metadata target form the mirror. The
> same directory is configured with metadata mirror support, so if I remove the
> metadata directory target on node1 then is no t possible to restore from the
> node2. I've tried the 3 ways but has not work.
>
> I try to explain what I'm doing. Once I've deleted the metadata directory on
> node1 I do a star of the metadata direcotory of node2 and copy the tar file to
> node1. Then I extract the tar file star, but what I have is al the content of
> node1 metadata directory and directories of dentries and inodes of node2 inte
> mirror dir. If I move mirror/2.dentries and mirror/2.inodes to dentries and
> inodes in the metadir does not work.

I guess the problem here is that you copied all contents of the other server's
metadata directory, including the files in the top-level directory (e.g.
nodeNumID). It's important that the server keeps its original node ID, so that
BeeGFS can identify the server correctly.
So you should either make a backup of the files in the top-level directory or
after deletion initialize the directory with the same ID that you originally
used with "beegfs-setup-meta" and make sure that the ID files are not
overwritten with values from the other server when you copy the inodes and
dentries data.

In this case, I only restore the mirror/2.dentrie and mirror/2.inodes in dentries and inodes directories. The metadata server starts correctly but once the metadata is up I can't see the files.
 

Best regards,
Sven

Thank you very much and sorry for my bad explanation.

Xavi.