gnatsd in docker swarm and routes

[nut]

Hi,
I installed gnatsd with a docker-compose file in a docker swarm (one leader node and 2 "follower") via:

docker stack deploy --compose-file=docker-compose.yml nats

Inside my docker-compose I specify:
 deploy:
    replicas: 3

My problem is: How do I specify the -routes option?

From my docker-compose file:
entrypoint: '/gnatsd -DV -cluster nats://0.0.0.0:6222 -routes nats://???.???.???.???:6222'

According to: https://github.com/nats-io/gnatsd

>snip

When running NATS Servers in different hosts, the command line parameters for all servers could be as simple as:

gnatsd --cluster nats://$HOSTNAME:$NATS_CLUSTER_PORT --routes nats://$NATS_SEED_HOST:$NATS_CLUSTER_PORT

Even on the host where the seed is running, the above would work as the server would detect an attempt to connect to itself and ignore that. In other words, the same command line could be deployed in several hosts and the full mesh will properly form.

>snip

I think I can to set 

NATS_SEED_HOST to the internal docker-IP of my docker-swarm leader.

But how to do this?

In the examples I found only the way to set up one service after another. And the second service can grab the IP from the first service and so on...

Another idea I have is to omit the -routes option and to reconfigure the gnatsd-daemons after a "docker stack deploy" with the proper  
NATS_SEED_HOST. Is this possible?


What I have now is the following:
I have 3 nodes. When I publish a message to the cluster it arrives only at one of the three servers.
I think when all the routes are set right I should have the message on ALL 3 nodes.

[Waldemar Quevedo]

Hi, 

Have you tried out using the `--name` option from docker services?  

With it you could then assemble a cluster by doing something like:

for node in A B C; do

  docker service create --network nats-cluster-example \

    --name   nats-$node nats:0.9.4 \

    -routes  nats://nats-A:6222 \

    -cluster nats://0.0.0.0:6222

done

Not sure whether you can do the same via just replicas declaration in Docker Swarm though... (related issue: https://github.com/docker/swarmkit/issues/1242), but if possible already then you could use `replicas` declaration in the compose file and then something like `-routes nats://nats.1:6222` so that they form a cluster for example.

I have some examples on NATS and Docker integration here in this repo that could walk through on how to set them up:

https://github.com/wallyqs/nats-docker-examples/blob/master/infra/infra-setup.org#prepare-nats-cluster-network-and-smoke-test

Hope this helps!

- Wally

--
You received this message because you are subscribed to the Google Groups "nats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to natsio+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[nut]

Thank you very much for your answer.

I have the following infrastructure:

3 servers say k1, k2, k3.
On every node should run one container with gnatsd.

Because using a docker-composefile with something like
-routes nats://nats.1:6222 seems
not to work (the DNS can not resolve nats.1) I ended up
with the following docker-compose file.

version: '3'
services:
  nats_master:
    image: 'nats:0.9.6'
    networks:
      - 'nats-cluster-net'
    entrypoint: '/gnatsd -DV -cluster nats://0.0.0.0:6222 -routes nats://nats:6222'
    deploy:
      replicas: 1

  nats:
    image: 'nats:0.9.6'
    networks:
      - 'nats-cluster-net'
    entrypoint: '/gnatsd -DV -cluster nats://0.0.0.0:6222 -routes nats://nats_master:6222'
    ports:
      - '8222:8222'
      - '4222:4222'
    deploy:
      replicas: 3

When I start my services with:

docker stack deploy --compose-file=docker-compose.yml nats

'docker service ls'
gives me:

ID            NAME              MODE        REPLICAS  IMAGE
dnxv1nt1kn28  nats_nats_master  replicated  1/1       nats:0.9.6
kij35e6u0e0t  nats_nats         replicated  3/3       nats:0.9.6

On every node started one nats_nats service.

I now wrote a simple ruby subscriber

NATS.start(:servers => ['nats://192.168.0.123:4222','nats://192.168.0.124:4222','nats://192.168.0.125:4222']) do |nats|
  puts "NATS is connected to #{nats.connected_server}"
  nats.subscribe('donkey', :queue => 'donkey-queue') { |msg| puts "Received '#{msg}'" }

and started this client script on every node.

Now I started a little producer:

NATS.start(:servers => ['nats://192.168.0.123:4222','nats://192.168.0.124:4222','nats://192.168.0.125:4222']) do |nats
  puts "NATS is connected to #{nats.connected_server}. Do publishing..."
  nats.publish('donkey', 'Donkey laugh')

The output of the client is as expected.
Every message is only processed by one client.

Now I shut down the k3-node.
I see that the service on k3-node moves to the node k2.
When I now
1) restart my clients at k1 and k2
2) publish new messages to the 'donkey-queue'

I do not see every message on the client side.
I still would expect to receive EVERY published message...

Because I expose the ports 8222 and 4222 in my docker-compose file I would
expect some port clashs when k3-node goes down and moves to k2...

To unsubscribe from this group and stop receiving emails from it, send an email to natsio+un...@googlegroups.com.