Multi-homed servers

[mailg...@gmail.com]

Hi there,

I have a situation here where I have all the BeeGFS services running on one single server that serves two different subnets per two different ethernet NICs.

I'm running into the problem of interface priorities i.e. only one interface may be the primary one and the other will be advertised as fallback route.

Ok I can disable the connection retries for the primary route but it still looks unclean to me.


Is there another solution to this?


Thanks in advance,
Gabriel

[Pete Sero]

Can you explain at bit more by showing the output of "netstat -r” ?

> --
> You received this message because you are subscribed to the Google Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[mailg...@gmail.com]

Hi,

Am Dienstag, 10. Februar 2015 14:48:03 UTC+1 schrieb Pete Sero:

Can you explain at bit more by showing the output of "netstat -r” ?

Well, the storage server has for example two nics eth0 and eth1 with corresponding distinct non overlapping subnets, say 10.1.0.0/16 and 10.2.0.0/16.
Then netstat -r on the storage server would give:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.1.0.0          0.0.0.0            255.255.0.0     U         0       0           0   eth0
10.2.0.0          0.0.0.0            255.255.0.0     U         0       0           0   eth1


In this case the all the serices (fhgfs-mgmt,  fhgfs-meta, fhgfs-storage) will choose one interface to be the primary one and one to be the fallback interface. If a client happens to be on the side of the primary network interface it will have the primary route, lets say eth0. If a client is on the other side (eth1 here) fhgfs-net shows:


mgmt_nodes
=============
storage [ID: 1]
   Connections: TCP: 1 (10.2.0.1:8008 [fallback route]);

meta_nodes
=============
storage [ID: 12250]
   Connections: TCP: 1 (10.2.0.1:8005 [fallback route]);

storage_nodes
=============
storage [ID: 12250]
   Connections: TCP: 1 (10.2.0.1:8003 [fallback route]);


fhgfs-client.log:

(3) *fhgfs_HBeatMgr(3765) [NodeConn (acquire stream)] >> Connect failed: fhgfs...@10.1.0.1:8008 (protocol: TCP)
(3) *fhgfs_HBeatMgr(3765) [NodeConn (acquire stream)] >> Connected: fhgfs...@10.2.0.1:8008 (protocol: TCP; fallback route)


Maybe there's something I'm completely missing here ... so please tell me if it makes no sense at all to you.

Cheers,
Gabriel

[Pete Sero]

That makes it clear. From the view of
standard IP routing everything is simple
(only local subnets), but an application/service
dares to impose own policies on top of this...

It’s a long shot, but you might be able to
trick your 10.2.0.0 clients into connecting
to the 10.2.0.0 service nics right away
by use of
tunePreferredMetaFile
tunePreferredStorageFile
in the client config.

Good look

— Peter

[mailg...@gmail.com]

Hi Pete,

thanks for your reply.

Am Dienstag, 10. Februar 2015 16:18:08 UTC+1 schrieb Pete Sero:

It’s a long shot, but you might be able to
trick your 10.2.0.0 clients into connecting
to the 10.2.0.0 service nics right away
by use of
tunePreferredMetaFile
tunePreferredStorageFile
in the client config.

 
In fact I already tried that without luck ... still have the fallback routes. I guess the servers see that they're preferred but that doesn't affect their network interface priorities.


Gabriel

[mailg...@gmail.com]

I was now trying to make use of the connNetFilterFile directive on the clients and it looks like as if this makes it working for me ...