Cannot connect to elasticsearch task on localhost

[Adam Watson]

I'm trying to run a single elasticsearch node locally with nomad and can't seem to get the http port to map correctly.

First, here is my job config:

job "elk-stack" {
  datacenters = ["dc1"]
  type = "service"

  update {
    max_parallel = 1
    min_healthy_time = "10s"
    healthy_deadline = "3m"
    progress_deadline = "10m"
    auto_revert = false
    canary = 0
  }

  meta {
    ES_CLUSTER_NAME = "es-cluster"
  }

  migrate {
    max_parallel = 1
    health_check = "checks"
    min_healthy_time = "10s"
    healthy_deadline = "5m"
  }

  group "es-cluster" {
    count = 1

    restart {
      attempts = 2
      interval = "30m"
      delay = "5m"
      mode = "fail"
    }

    ephemeral_disk {
      size = 300
    }

    task "elasticsearch" {
      driver = "docker"

      config {
        image      = "docker.elastic.co/elasticsearch/elasticsearch:7.0.1"
        port_map {
          rest = 9200
        }
      }

      env {
        "discovery.type" = "single-node"
        "bootstrap.memory_lock" = "true"
        "cluster.name" = "${NOMAD_META_ES_CLUSTER_NAME}"
        "http.port" = "${NOMAD_PORT_rest}"
        "http.host" = "0.0.0.0"
        "http.publish_port" = "${NOMAD_HOST_PORT_rest}"
        "network.host" = "0.0.0.0"
        "network.publish_host" = "${NOMAD_IP_rest}"
      }

      resources {
        cpu    = 500
        memory = 256
        network {
          mbits = 25
          port "rest" {
          }
        }
      }

      service {
        name = "${NOMAD_JOB_NAME}"
        tags = ["global", "db"]
        port = "rest"
        check {
          name     = "alive"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

With this config the task spins up fine and the check passes and I can see the port mapping with docker:

> docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                                            NAMES
e8712c2566c8        docker.elastic.co/elasticsearch/elasticsearch:7.0.1   "/usr/local/bin/dock…"   8 minutes ago       Up 8 minutes        9300/tcp, 127.0.0.1:31531->9200/tcp, 127.0.0.1:31531->9200/udp   elasticsearch-697eb6d4-14b0-a766-c1da-fc3c59761929

If I curl that endpoint I get:

> curl 127.0.0.1:31531
curl: (56) Recv failure: Connection reset by peer

But, if I run that same curl command from within the docker container it returns the expected healthy elasticsearch response:

[root@e8712c2566c8 elasticsearch]# curl localhost:31531
{
  "name" : "e8712c2566c8",
  "cluster_name" : "elashticstack-cluster",
  "cluster_uuid" : "mQfaXVSKR7el4oKOxj1hLw",
  "version" : {
    "number" : "7.0.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "e4efcb5",
    "build_date" : "2019-04-29T12:56:03.145736Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Additionally, I can curl the IP address assigned to it by docker (172.17.0.2 in this case) and get the same response from outside the container. I've been struggling with this for most of the past week and haven't made much progress beyond characterizing the issue I'm having a little more.

Some things I've tried:
Changing the http.host and network.host settings to NOMAD_IP_rest, 127.0.0.1 or commenting them out haven't changed much. One of those combinations made it so I could only curl 127.0.0.1 within the docker container instead of localhost, but the behavior from outside the container was unchanged.
Changing the nomad bind_addr to 0.0.0.0 had no effect on the port mappings shown in docker ps.

I've seen suggestions in other places to switch to host networking but I'd prefer not to do that. I'm trying to really learn how running an ELK stack in Nomad works so would like to be able to expand this to a more production-like configuration eventually and host networking wouldn't work in that case.

So, my question is what config am I missing/doing wrong that I can't connect to elasticsearch using localhost? Any help with that would be greatly appreciated.

[Lowe Schmidt]

Are you on a GNU/Linux host or are you using docker-machine (or similar) ?

--

Lowe Schmidt | +46 723 867 157

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/028e24b1-2417-4a6d-9291-84236ce19a7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam Watson]

I knew I forgot to include something! I'm running Linux Mint 19.

Other version info:

Docker - 18.09.5

Nomad - v0.9.1

Consul - v1.1.0

To unsubscribe from this group and stop receiving emails from it, send an email to nomad...@googlegroups.com.

[Lowe Schmidt]

Is your service considered up in consul? The Web ui should be available on localhost:8500

--

Lowe Schmidt | +46 723 867 157

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/f89edda7-abe6-4789-bbd2-6547db012d90%40googlegroups.com.

[Adam Watson]

It's considered up in Consul, and healthy from Nomad. But if I add another health check like this:

check {
  name     = "rest-http"
  type     = "http"
  port     = "rest"
  path     = "/"
  interval = "5s"
  timeout  = "4s"
}

That one will fail.

[Nick Ethier]

Can you check to see what ip elasticsearch is bound to inside the container? I know it defaults to loopback. You can check this by running ‘netstat -an’. 

-Nick

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/fcd184d2-1494-40b5-92fe-00c1f46b024b%40googlegroups.com.

[Adam Watson]

The docker image doesn't have netstat or apt installed, so I ran netstat in the running container from the host using nsenter as per this StackOverflow question: https://stackoverflow.com/questions/40350456/docker-any-way-to-list-open-sockets-inside-a-running-docker-container

Here's what it spit out:

> docker inspect -f '{{.State.Pid}}' elasticsearch-50fde3aa-18ae-d09a-9ebc-814e87f6d177
4006


> sudo nsenter -t 4006 -n netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.1:9300          0.0.0.0:*               LISTEN    
tcp        0      0 127.0.0.1:27033         0.0.0.0:*               LISTEN    
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ]         STREAM     CONNECTED     44885    
unix  2      [ ]         STREAM     CONNECTED     44363

[Nick Ethier]

Ok this is good. It looks like theres nothing listening on the 'rest' port you've configured in your port map to be 9200. Also these sockets are binding to the loopback interface. You'll want elasticsearch to bind to the container network interface. This is usually easily done by setting the bind IP to '0.0.0.0', in this case that looks to be 'network.host' in elasticsearch's config. 

[Adam Watson]

So I don't think that worked. I changed my env config to the following:

      env {
        "discovery.type" = "single-node"
        "bootstrap.memory_lock" = "true"
        "cluster.name" = "${NOMAD_META_ES_CLUSTER_NAME}"
        "http.port" = "${NOMAD_PORT_rest}"
        "http.host" = "0.0.0.0"

        "network.host" = "0.0.0.0"
      }

at which point netstat showed it was binding on 0.0.0.0 instead of 127.0.0.1, but still listening on the dynamic port mapped by nomad instead of 9200 like it should be. This also exhibited the same behavior I've been seeing, where I can curl the endpoint from within the docker container on the dynamic port from nomad, or at the IP assigned to it by Docker from the host, but not at localhost like one would expect. I also tried changing http.port to 9200 and that made it worse, I couldn't curl elasticsearch in any of the ways I had been able to so far even though netstat showed it was listening on 0.0.0.0:9200

[Nick Ethier]

Great! I'm glad you were able to figure it out Adam! Let us know if there is anything else we can do to help.

-Nick

On Tue, May 7, 2019 at 6:29 PM Adam Watson <adamcw...@gmail.com> wrote:

I GOT IT. So I needed to set http.port to 9200, and set http.publish_port to ${NOMAD_PORT_rest}. it looks like from the elasticsearch documentation the http.publish_port tells the container what port docker wlll map http.port to so that it can configure the network correctly (or something). Once I changed my env config to this:

      env {
        "discovery.type" = "single-node"
        "bootstrap.memory_lock" = "true"
        "cluster.name" = "${NOMAD_META_ES_CLUSTER_NAME}"

        "http.port" = "9200"
        "http.publish_port" = "${NOMAD_PORT_rest}"

        "http.host" = "0.0.0.0"
        "network.host" = "0.0.0.0"
      }

I was able to hit the elasticsearch endpoint from localhost! Finally! Thank you guys so much!!

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/d071b7b4-00cc-40a4-8124-ed3e2e6a4403%40googlegroups.com.

[Adam Watson]

Will do! Sorry for the double reply btw, Google groups said "your message has been deleted" when I submitted both responses....