fhgfs-net reports no connection in a meta node

[Ramon Diaz-Uriarte]

Dear All,

I have recently installed FhGFS on a two-node cluster, where each machine
is both client, server, and metadata server. After rebooting, I sometimes
see that one of the metadata nodes (the non-root one) is reported as
having no connection, but everything works fine and the metadata nodes are
reported as being reachable from both nodes. How could this be the case?
After restarting the metadata service, both meta nodes are reported with a
connection.


This is a sequence of calls (lets call node1 the one that is also
management node):


******
root@node2~ fhgfs-ctl mode=listnodes nodetype=metadata print_details

mgmt_nodes
=============
node1
Connections: TCP: 1 (192.168.3.2:8008);

meta_nodes
=============
node1
Connections: RDMA: 1 (192.168.3.2:8005);
node2
Connections: <none>

storage_nodes
=============
node1
Connections: RDMA: 1 (192.168.3.2:8003);
node2
Connections: RDMA: 1 (192.168.3.1:8003);



## But everything works. Moreover, if I do:


root@node1:~# fhgfs-ctl mode=listnodes nodetype=metadata check_reachability
node1
Reachable: <yes>
node2
Reachable: <yes>


## and fhgfs knows about interfaces and connections
root@node2:/etc/fhgfs# fhgfs-ctl mode=listnodes nodetype=metadata print_details
node1
Ports: UDP: 8005; TCP: 8005
Interfaces: ib0(RDMA) ib0(TCP) eth1(TCP) eth0(TCP)
node2
Ports: UDP: 8005; TCP: 8005
Interfaces: ib0(RDMA) ib0(TCP) eth1(TCP) eth0(TCP)

Number of nodes: 2
Root: node1


## lets restart the service

root@node2:~# /etc/init.d/fhgfs-meta restart

### Things are now what I expected

root@node2:/mnt/fhgfs/R-patched# fhgfs-net

mgmt_nodes
=============
node1
Connections: TCP: 1 (192.168.3.2:8008);

meta_nodes
=============
node1
Connections: RDMA: 1 (192.168.3.2:8005);
node2
Connections: RDMA: 1 (192.168.3.1:8005);

storage_nodes
=============
node1
Connections: RDMA: 1 (192.168.3.2:8003);
node2
Connections: RDMA: 1 (192.168.3.1:8003);


***************


Version of FhGFS: 2011.04.r16.debian6_amd64


Best,


R.

--
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina
Universidad Autónoma de Madrid
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdi...@gmail.com
ramon...@iib.uam.es

http://ligarto.org/rdiaz

[Christian Mohrbacher]

Hi Ramon,
if that happens after restart, this is not a problem at all.

To understand that, you must know that fhgfs-net only shows established connections and the clients only establish a connection to the metadata server, if he needs to send a request to it. So the connection is established the first time data is retrieved from this server. So, it is pretty normal that there is no connection in the beginning.

Regards,
Christian

----------------ursprüngliche Nachricht-----------------
Von: "Ramon Diaz-Uriarte" rdi...@gmail.com
An: "fhgfs-user" fhgfs...@googlegroups.com
Datum: Sat, 26 May 2012 16:44:49 +0200
-------------------------------------------------

>
> Dear All,
>
> I have recently installed FhGFS on a two-node cluster, where each machine
> is both client, server, and metadata server. After rebooting, I sometimes
> see that one of the metadata nodes (the non-root one) is reported as
> having no connection, but everything works fine and the metadata nodes are
> reported as being reachable from both nodes. How could this be the case?
> After restarting the metadata service, both meta nodes are reported with a
> connection.
>
>
> This is a sequence of calls (lets call node1 the one that is also
> management node):
>
>
> ******

> root@node2 ~ fhgfs-ctl mode=listnodes nodetype=metadata print_details

--

=======================================================
Christian Mohrbacher
Competence Center High Performance Computing
Institut fuer Techno- und Wirtschaftsmathematik (ITWM)
Fraunhofer-Platz 1

D-67663 Kaiserslautern
=======================================================
Tel.: (49) 631 31600 4425
Fax: (49) 631 31600 1099
E-Mail: christian....@itwm.fraunhofer.de
Internet: http://www.itwm.fraunhofer.de
=======================================================

[Ramon Diaz-Uriarte]

Hi Christian,

Thanks a lot for the explanation.

Best,

R.