BeeGFS meta/storage node advertise IP

[barton...@gmail.com]

Hi,

I'd like to run an instance of meta node in a container, which means that the IP address

of main interface is not the IP address which should be used for communication. 

I'm using different port for the meta node:

connMetaPortUDP              = 9005

connMetaPortTCP              = 9005

I'm getting errors like:

 [emergency] [ERROR from fhgfs-meta meta02.example.com [ID: 49504]: Communication error]

and mgmtd logs:

(2) Feb20 16:05:41 Worker4 [Node registration] >> New node: fhgfs-meta meta02.example.com [ID: 41358]; Ver: 2014.1-19; Source: 172.27.11.13:46285

(1) Feb20 16:05:43 XNodeSync [NodeConn (acquire stream)] >> Connect failed on all available routes: fhgfs-meta meta02.example.com [ID: 41358]

 XNodeSync [Messaging (RPC)] >> Retrying communication with node: fhgfs-meta meta02.example.com [ID: 41358] (Message type: StatStoragePath (2031))

The connection to mgtm node seems to be estabilished:

fhgfs-met 8130 root    6u  IPv4 579861793      0t0  UDP *:9005 

fhgfs-met 8130 root   12u  IPv4 579861795      0t0  TCP *:9005 (LISTEN)

fhgfs-met 8130 root   13u  IPv4 579654494      0t0  TCP 10.0.3.10:46318->172.27.11.11:8008 (ESTABLISHED)

The hostname `meta02.example.com` is translated to correct ip address:

host meta02.example.com has address 172.27.11.13

and with iptables I translate both TCP and UDP to appropriate internal address:

-t nat -A PREROUTING -p tcp -m tcp -d 172.27.11.13 --dport 9005 -j DNAT --to-destination 10.0.3.10:9005

-t nat -A OUTPUT -p tcp -m tcp -d 172.27.11.13 --dport 9005 -j DNAT --to-destination 10.0.3.10:9005

-t nat -A PREROUTING -p udp -m udp -d 172.27.11.13 --dport 9005 -j DNAT --to-destination 10.0.3.10:9005

-t nat -A OUTPUT -p udp -m udp -d 172.27.11.13 --dport 9005 -j DNAT --to-destination 10.0.3.10:9005

Am I missing something?

Thanks,

Tomas

[barton...@gmail.com]

I've increased loglevel:

XNodeSync [NodeConn (acquire stream)] >> Establishing new TCP connection to: fhgfs...@10.0.3.10:9005

(2) Feb20 17:40:34 XNodeSync [Messaging (RPC)] >> Unable to connect to: fhgfs-meta meta02.example.com [ID: 41358]. (Message type: StatStoragePath (2031))

Obviously fhgfs-meta doesn't resolve FQDN to an ipaddress, but uses an ip of primary interface. Can I override this somehow?

[Sven Breuner]

Hi Tomas,

is this what you're looking for?
http://www.beegfs.com/wiki/FAQ#multiple_nics

Best regards,
Sven

barton...@gmail.com wrote on 02/20/2016 05:48 PM:
> I've increased loglevel:
>
> XNodeSync [NodeConn (acquire stream)] >> Establishing new TCP connection
> to: fhgfs...@10.0.3.10:9005
> (2) Feb20 17:40:34 XNodeSync [Messaging (RPC)] >> Unable to connect to:
> fhgfs-meta meta02.example.com [ID: 41358]. (Message type:
> StatStoragePath (2031))
>
> Obviously fhgfs-meta doesn't resolve FQDN to an ipaddress, but uses an
> ip of primary interface. Can I override this somehow?
>
>
>
> On Saturday, 20 February 2016 16:35:33 UTC+1, barton...@gmail.com wrote:
>
> Hi,
>
> I'd like to run an instance of meta node in a container, which means
> that the IP address
> of main interface is not the IP address which should be used for
> communication.
>
> I'm using different port for the meta node:
>
> connMetaPortUDP = 9005
> connMetaPortTCP = 9005
>
>
> I'm getting errors like:
>
> [emergency] [ERROR from fhgfs-meta meta02.example.com

> <http://meta02.example.com> [ID: 49504]: Communication error]

>
> and mgmtd logs:
>
> (2) Feb20 16:05:41 Worker4 [Node registration] >> New node:

> fhgfs-meta meta02.example.com <http://meta02.example.com> [ID:

> 41358]; Ver: 2014.1-19; Source: 172.27.11.13:46285

> <http://172.27.11.13:46285>

> (1) Feb20 16:05:43 XNodeSync [NodeConn (acquire stream)] >> Connect
> failed on all available routes: fhgfs-meta meta02.example.com

> <http://meta02.example.com> [ID: 41358]

> XNodeSync [Messaging (RPC)] >> Retrying communication with node:

> fhgfs-meta meta02.example.com <http://meta02.example.com> [ID:

> 41358] (Message type: StatStoragePath (2031))
>
> The connection to mgtm node seems to be estabilished:
>
> fhgfs-met 8130 root 6u IPv4 579861793 0t0 UDP *:9005
> fhgfs-met 8130 root 12u IPv4 579861795 0t0 TCP *:9005 (LISTEN)
> fhgfs-met 8130 root 13u IPv4 579654494 0t0 TCP

> 10.0.3.10:46318->172.27.11.11:8008 <http://172.27.11.11:8008>
> (ESTABLISHED)
>
> The hostname `meta02.example.com <http://meta02.example.com>` is

> translated to correct ip address:
>

> host meta02.example.com <http://meta02.example.com> has address

> 172.27.11.13
>
> and with iptables I translate both TCP and UDP to appropriate
> internal address:
>
> -t nat -A PREROUTING -p tcp -m tcp -d 172.27.11.13 --dport 9005 -j

> DNAT --to-destination 10.0.3.10:9005 <http://10.0.3.10:9005>

> -t nat -A OUTPUT -p tcp -m tcp -d 172.27.11.13 --dport 9005 -j DNAT

> --to-destination 10.0.3.10:9005 <http://10.0.3.10:9005>

> -t nat -A PREROUTING -p udp -m udp -d 172.27.11.13 --dport 9005 -j

> DNAT --to-destination 10.0.3.10:9005 <http://10.0.3.10:9005>

> -t nat -A OUTPUT -p udp -m udp -d 172.27.11.13 --dport 9005 -j DNAT

> --to-destination 10.0.3.10:9005 <http://10.0.3.10:9005>

>
> Am I missing something?
>
> Thanks,
> Tomas
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com
> <mailto:fhgfs-user+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Tomas Barton]

Hi Sven,

no, that's something different. Inside the container I have single interface which is NATed to the physical one.

I wanted to run multiple instances of same service (meta node) on single machine, and using containers was first idea that came up.  Finally, I've solved it by running multi-modal instances.

Many distributed applications support advertising IP, I thought that BeeGFS should have something like this.

Btw. is there any progress with making BeeGFS open-source? 

Regards,

Tomas

[Sven Breuner]

Hi Tomas,

Tomas Barton wrote on 02/23/2016 07:03 PM:
> no, that's something different. Inside the container I have single
> interface which is NATed to the physical one.
>

> [...]

> Many distributed applications support advertising IP, I thought that
> BeeGFS should have something like this.

ah ok, I see. BeeGFS is trying to be smart here (I know you'll say that
it's not so smart in your case, but at least in 99.9% of the cases it is
smart ;-) ) and thus auto-detects the available NICs and their IPs, so
there is currently no way of manually setting an advertised IP.
But I understand of course that such a possibility would make sense in
your case.

> Btw. is there any progress with making BeeGFS open-source?

http://www.beegfs.com/source-download

So if you have some spare time, I think it should be quite
straight-forward to add the capability to read a
beegfs-advertising-ip.conf instead of doing the auto-detection :-)

Best regards,
Sven

> On Tue, 23 Feb 2016 18:02 Sven Breuner <sven.b...@itwm.fraunhofer.de
> <mailto:sven.b...@itwm.fraunhofer.de>> wrote:
>
> Hi Tomas,
>
> is this what you're looking for?
> http://www.beegfs.com/wiki/FAQ#multiple_nics
>
> Best regards,
> Sven
>

> barton...@gmail.com <mailto:barton...@gmail.com> wrote on

> 02/20/2016 05:48 PM:
> > I've increased loglevel:
> >
> > XNodeSync [NodeConn (acquire stream)] >> Establishing new TCP
> connection

> > to: fhgfs...@10.0.3.10:9005 <http://fhgfs...@10.0.3.10:9005>

> > (2) Feb20 17:40:34 XNodeSync [Messaging (RPC)] >> Unable to
> connect to:

> > fhgfs-meta meta02.example.com <http://meta02.example.com> [ID:

> 41358]. (Message type:
> > StatStoragePath (2031))
> >
> > Obviously fhgfs-meta doesn't resolve FQDN to an ipaddress, but
> uses an
> > ip of primary interface. Can I override this somehow?
> >
> >
> >
> > On Saturday, 20 February 2016 16:35:33 UTC+1, barton...@gmail.com

> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>
> > <mailto:fhgfs-user+...@googlegroups.com
> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>>.

[Tomas Barton]

Hi Sven,

 

but at least in 99.9% of the cases it is smart

 

Yes, it's quite smart and it works on bare metal. Especially when you can specify the preferred interface. But people might run into similar issues 

when trying to use beegfs e.g. on OpenVZ or Docker.

http://www.beegfs.com/source-download

That's a great news! I haven't noticed it before. Could you please put the source code on github.com? It's defacto industry standard. It

makes much easier to track issues, references and see the difference between versions. The model of open-source on demand

it quite outdated in 2016 :) For many people this might be still a reason why don't install the software.

Even to keep the BeeGFS EULA:

3.2.1) LICENSEE must inform users of modified versions about the fact that the
software differs from the original version.

it's much easier to fork the code on github, than send patches via email.

I'm not gonna make anything bad with the code :) Currently I have another issue with starting the client. Having the code makes it

much easier to track.

 Regards,

Tomas

[Michael Ruepp]

Hi Thomas,

 

I am curious what you want to achieve by using Beegfs Services in containers. What advantage would you expect using a system designed for high performance on bare metal in containerized environments? The only thing what makes sense to me is to virtualize Management/Admon services to get failover quickly.

 

In case of respawning of services or failover, one always has to consider the storage stack which will not move along with the containerized environment, so whats the benefit at all?

 

 

Thanks for clearing things up for me,

 

Michael

To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.

[Tomas Barton]

Hi Michael,

I wanted to setup a quick a failover, so that we can safely migrate bare metal machines. Another use-case is for testing purposes, when you have automated configuration

of services, you don't want to test it in production. So, containerized setup is ideal for this.

We use containers for isolation as well. It makes sure that service running inside container will always have given resources (CPU & mem), it has dedicated disk

and the overhead of NAT (veth) is minimal.

Regards,

Tomas

--
You received this message because you are subscribed to a topic in the Google Groups "beegfs-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fhgfs-user/7QEZStIQBwY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fhgfs-user+...@googlegroups.com.

[Sven Breuner]

Hi Tomas,

Tomas Barton wrote on 02/24/2016 10:24 AM:
> it's much easier to fork the code on github, than send patches via email.

github is an option that we're still discussing internally.

But right now we also get very useful feedback from the comments that
people send us through the source code download form.

Best regards,
Sven