use nomad deploy a 3 nodes elasticsearch cluster

[tim.x...@gmail.com]

Hi All,

I would like to use Nomad to deploy a 3 nodes elasticsearch cluster. Here is the job file:

job "logging-elk" {
  datacenters = ["my-little-datacenter"]
  type = "service"

  update {
    stagger = "10s"
    max_parallel = 1
  }

  # - logging-elk - #
  group "logging-elk" {

    count = 3

    # - elasticsearch - #
    task "elasticsearch" {
      driver = "docker"

      template {
        data = <<EOH
cluster.name: "docker-cluster123"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
xpack.license.self_generated.type: basic
        EOH

        destination = "local/elasticsearch.yml"
      }



      config {
        image = "docker.elastic.co/elasticsearch/elasticsearch:6.1.2"
        logging {
          type = "json-file"
        }
        port_map {
          es_rest = 9200
          es_node = 9300
        }

        volumes = [
          "local/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml",
        ]
      }

      resources {
        memory = 3000
        network {
          mbits = 50
          port "es_rest" {
            static = "9201"
          }
          port "es_node" {
            static = "9301"
          }

        }
      }
      
      service {

        name = "elasticsearch"

        tags = ["elastic"]

        port = "es_rest"

        check {
          type = "http"
          path = "/"
          port = "es_rest"
          interval = "10s"
          timeout = "2s"
        }

      }

    }
    # - end elasticsearch - #
  }
  # - end logging-elk - #

}

After I ran the job, the 3 elasticsearch containers can't join as one cluster, does anyone can provide a working example?

[Michael Schurter]

You're using Docker's default bridge network so the Elasticsearch daemons will see and advertise an IP address internal to their container. You'll want to template the IP address and port of the host using the network environment variables at the bottom of this page: https://www.nomadproject.io/docs/runtime/interpolation.html

Alternatively you can use host networking since you're using static port assignments anyway.

A couple other notes:

1. Using minimum_master_nodes=1 in a 3 node cluster will lead to data loss as during a network partition or even just while doing restarts of ES each node may consider itself a complete cluster and accept writes which may be dropped when the node joins the full cluster again.

2. You may want to advertise ES nodes in Consul and use the Consul discovery plugin instead of zen disco: https://github.com/lithiumtech/elasticsearch-consul-discovery

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/30526dfa-1b54-4610-8a20-5f2e905b10a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tim.x...@gmail.com]

Thanks, Michael!   These are very helpful suggestions.  I was able to setup the es cluster with docker host network.

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.