Running QFS on Docker Swarm

[Michael Kamprath]

I've set up a small QFS deployment on a cluster managed over docker swarm. Details of what I did can be viewed here. My question is about how QFS identifies chunk servers. Each node in the cluster gets a single chunk server process, and the data for that chunk server is persistent outside of the Docket container with a volume mount to a specific location on the physical node's disk. Note the the meta server has a similar set up but always placed on eh same node in the swarm. If I brink down my stack that includes QFS and bring it back up, all of the QFS docker containers get different IP addresses then what they had before. I haven't found a way with Docker swarm to assign a specific IP to any given container. QFS seems to work fine with this, but it remembers the old IP addresses of the container and thinks those chunk servers are dead. None the less, the meta server seems to adapt, recognized the chunk servers as "new", and none of the data on the QFS deployment becomes unreachable after the stack redeployment, even though the QFS status page says it has dead chunk servers. 

My question is whether I have anything to be concerned about in this scenario? I am assuming the meta server finds data by chunk IDs and not chunk server IP addresses, but I just wanted to make sure I am not missing something.

Thanks!

Michael

[Michael Ovsiannikov]

Michael,

The short is answer is “no” -- IP address change after restart should not affect normal file system operations, except for the web UI reporting, and the time it takes for the file system to transition to operational state.

Presently chunk servers’ addresses {IP, port} are used as unique chunk server identifiers by chunk inventory synchronization protocol. When IP address (or set of hosted chunks) changes at the time when chunk and meta server are not communicating, the chunk inventory protocol detects that and falls back to full (instead of partial) chunk inventory synchronization at the time chunk server re-connects back to the meta server. With full inventory synchronization chunk server transmits to the meta server information about all available chunks. With large number of chunks full inventory synchronization can consume noticeable amount of wall and CPU time.

The meta server’s knowledge of chunk servers chunk inventory is persistent — it is stored in the transaction log and checkpoint, and replicated to other meta server nodes in the case when redundant meta server nodes configured. The primary goal of partial chunk inventory synchronization is to minimize file system swichower time to backup meta server node when primary meta server node becomes unavailable, and chunk servers and clients have to talk to a different meta server node that becomes primary.

The chunk server IP address by default is assigned by chunk server, and updated every time chunk server connects to the meta server. The IP address is obtained by calling getsockname() on the meta server connection socket, i.e. local IP address as perceived by the chunk server.

Adding the following line to the chunk server’s configuration [and restarting chunk server] changes the default IP address assignment behavior:

chunkServer.hostname = 0.0.0.0

In this mode, meta server obtains chunk server IP by invoking getpeername() with the chunk server connection socket. This mode is intended to handle the cases with NAT between chunk, meta server,  and clients, when the “local" IP address obtained by the default method cannot be used by clients to connect to the chunk servers (for example Kubernetes).

Thank you,

 — Mike.

--
You received this message because you are subscribed to the Google Groups "QFS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qfs-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qfs-devel/c935e0d9-12fa-4f14-aa5e-59265c485180%40googlegroups.com.

[Michael Kamprath]

Michael O.,

Thanks! This was insightful.

Michael K.

To unsubscribe from this group and stop receiving emails from it, send an email to qfs-...@googlegroups.com.

[mic...@kamprath.net]

Michael O.,

Just saw that QFS 2.2.3 was release and has the new chunkServer.nodeId configuration. Since theses to address the issues I brought up earlier this thread, I thought I'd pose my question on it here. So I am using a Docker Swarm with the  chunk server containers being deployed with "global" replication, so in my deployment each chunk server container isn't uniquely configured. So I bound the /etc/hostname file of the host server t the /etc/hostname file in the chunk server container, and then set the chunk server nodeID to "FILE:/etc/hostname" in the configuration file. My question is what evidence do I have this took effect? I ask because when I look at the QFS status page for my cluster, it still enumerates chunk servers by IP address rather than the md5 of the contents of the /etc/hostname file. Or does this change only take effect for newly instantiated QFS clusters?

Thanks!

Michael K.

[Michael Ovsiannikov]

Hi Michael,

I do not think that node ID can be used to solve the problem with non static chunk server IP addresses, discussed in this conversation.

In 2.2.3 release the sole use/purpose of the node id is to allow the QFS client library to determine whether or not the node that is running on, also runs chunk server.

The primary motivation what “object store” (AWS s3) use case where chunk server acts as [s3] access proxy, and does not store data locally.

Presently node ID plays no role in replication and such. For the purposes of replication / re-balancing ip address and port are still used exactly the same way as in previous releases.

It should be possible to inspect chunk server node Id by navigating to chunk servers status page in the QFS web UI and selecting XMeta-node-id checkbox.

To use node ID chunk server, meta server, and QFS client library need to be updated to release 2.2.3.

Thank you,

 — Mike.

To unsubscribe from this group and stop receiving emails from it, send an email to qfs-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qfs-devel/f5c56c82-e7d7-442f-9831-ea63ce6154aen%40googlegroups.com.

[mic...@kamprath.net]

Ah thanks.

Michael K.