Nomad doesn't register Consul service

[Vladimir Kozyrev]

Hi,

When I run my nomad job, in the logs it says that Nomad is registering service in Consult.
nomad status $job_name says that something has been registered, but I can't get it using curl, e.g., curl "192.168.11.20:8500/v1/catalog/service/haproxy"

I have the following nomad HCL conifg that I use to run Nomad job:

job "haproxy-nomad" {
  region = "global"
  datacenters = ["aws-eu-west-1"]

  type = "service"

  constraint {
    attribute = "${attr.kernel.name}"
    value     = "linux"
  }

  update {
    stagger = "10s"
    max_parallel = 1
  }

  group "haproxy-dev" {
    count = 1
    restart {
      interval = "1m"
      attempts = 2
      delay    = "15s"
      mode     = "delay"
    }
    task "haproxy" {
      driver = "docker"
      config {
        image    = "example.com:5000/haproxy-rhel:0.1"
        command  = "/usr/bin/supervisord"
        hostname = "haproxy-nomad"
      }
      service {
        name = "haproxy"
        tags = ["global","haproxy"]
        port = "http_haproxy"
        check {
          name     = "tss alive"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
        check {
          name = "haproxy running on port 34180"
          interval = "10s"
          timeout  = "5s"
          type     = "http"
          protocol = "http"
          path     = "/haproxy_status"
        }
      }
      resources {
        cpu = 500 # Mhz
        memory = 512 # MB
        network {
          mbits = 100
          port "http_haproxy" {
            static = 8001
          }
          port "http_elb" {
            static = 34180
          }
        }
      }
    }
  }
}

Consul client config

{
    "server": false,
    "datacenter": "aws-eu-west-1",
    "data_dir": "/var/consul",
    "ui": true,
    "encrypt": "ZeOmrAz/m9s1N6NPqFvbLQ==",
    "ca_file": "/etc/consul.d/ssl/ca.cert",
    "cert_file": "/etc/consul.d/ssl/consul.cert",
    "key_file": "/etc/consul.d/ssl/consul.key",
    "verify_incoming": true,
    "verify_outgoing": true,
    "log_level": "INFO",
    "enable_syslog": true,
    "start_join": ["192.168.15.101","192.168.15.102","192.168.15.103"]
}

Nomad client config

bind_addr = "192.168.11.20"

log_level = "INFO"
datacenter = "aws-eu-west-1"

data_dir = "/var/nomad"

client {
    enabled = true
    network_interface =  "eth0"
    servers = ["192.168.15.101:4647","192.168.15.102:4647","192.168.15.103:4647"]
    options {
        consul.address = "192.168.11.20:8500"
        consul.token = "ZeOmrAz/m9s1N6NPqFvbLQ=="
        consul.auth = "token"
        consul.ssl = true
        consul.verifyssl = true
        docker.auth.config = "/root/.docker/config.json"
        docker.cleanup.image = false
    }
}

Thanks,
Vladimir

[Diptanu Choudhury]

Hi Vladimir,

Can you please share the Nomad client logs with us? It would help if the log level is set to DEBUG.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/a261ae64-7111-4b78-be66-54e569c891f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Diptanu Choudhury

Web - www.linkedin.com/in/diptanu

Twitter - @diptanu

[Vladimir Kozyrev]

Hi Diptanu,

Thank you for looking into this! Here comes the DEBUG log:

[INFO] consul: registering service haproxy with consul.
[DEBUG] consul: error while registering service haproxy with consul: Put https://192.168.11.20:8500/v1/agent/service/register?token=ZeOmrAz%2Fm9s1N6NPqFvb
LQ%3D%3D: http: server gave HTTP response to HTTPS client
[INFO] consul: registering check with ID: da2508106a07129a36662cf2aabeb7e489c89c4c for service: nomad-registered-service-0f12500b-fa9d-b945-4cc4-befc63578
e77
[DEBUG] consul: error while registering check tss alive with consul: Put https://192.168.11.20:8500/v1/agent/check/register?token=ZeOmrAz%2Fm9s1N6NPqFvbLQ
%3D%3D: http: server gave HTTP response to HTTPS client
[INFO] consul: registering check with ID: cf2114426748efaaf2dd97c7ad0f1fda972cb8c0 for service: nomad-registered-service-0f12500b-fa9d-b945-4cc4-befc63578
e77
[DEBUG] consul: error while registering check haproxy running on port 34180 with consul: Put https://192.168.11.20:8500/v1/agent/check/register?token=ZeOm
rAz%2Fm9s1N6NPqFvbLQ%3D%3D: http: server gave HTTP response to HTTPS client

I've tried to figure it out myself.
I've updated consul configuration file adding HTTPS address and port.

"addresses": {
  "https": "0.0.0.0"
},
"ports": {
  "https": 8800
},

I've updated consul's port in nomad's configuration file as well.

consul.address = "192.168.11.20:8800"

Now there are no errors, but services are still not registered.

[INFO] consul: registering service haproxy with consul.
[INFO] driver.docker: created container fdbfd6392581776586433ed5bf2c17399cd04a63e8affc5cc4d6664c0a894519
[DEBUG] client: updated allocations at index 7678 (pulled 0) (filtered 18)
[DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 18)
[INFO] driver.docker: started container fdbfd6392581776586433ed5bf2c17399cd04a63e8affc5cc4d6664c0a894519


Thanks,
Vladimir

пятница, 18 марта 2016 г., 22:32:37 UTC+3 пользователь Diptanu Choudhury написал:

[Vladimir Kozyrev]

Quick update.

Nomad version in use: 0.3.1

Consul version is use: 0.6.4

Nomad doesn't care about it's own configuration, it seems

I've consul.address variable in /etc/nomad.d/client/client.hcl

It's now called  consul.addddress and it's pointing to port 22. Nevertheless, nomad is saying that it's registering service in consul without any errors.

bind_addr = "192.168.11.20"

# Increase log verbosity

log_level = "DEBUG"

datacenter = "aws-eu-west-1"

# Setup data dir

data_dir = "/var/nomad"

# Enable the client

client {

    enabled = true

    network_interface =  "eth0"

    servers = ["192.168.15.101:4647","192.168.15.102:4647","192.168.15.103:4647"]

    options {

        consul.addddress = "192.168.11.20:22"

        consul.token = "ZeOmrAz/m9s1N6NPqFvbLQ=="

        consul.auth = "token"

        consul.ssl = true

        consul.verifyssl = true

        docker.auth.config = "/root/.docker/config.json"

        docker.cleanup.image = false

    }

}

Thanks,

Vladimir

пятница, 18 марта 2016 г., 18:59:07 UTC+3 пользователь Vladimir Kozyrev написал:

[Diptanu Choudhury]

Hi Vladimir,

Sorry for the late response. I didn't realize that your Consul agent's API was exposed over HTTPS. Can you please see if this works if the Consul agent works over HTTP?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/f8347fcf-50fd-41a5-ae2f-ec10721853f6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Vladimir Kozyrev]

Hi Diptanu,

Yes, it works over HTTP. Services and check are getting registered in Consul.

Could you point out what is wrong in my HTTPS configuration?

What additional information can I provide?

Thanks,

Vladimir

[Diptanu Choudhury]

Hi Vladimir,

We will have to add some more options in the Nomad configuration for users to specify their TLS cert file and TLS Key. We can get this done for the 0.3.2 release which will be coming out in a few days, can you please open an issue on Github?

[Vladimir Kozyrev]

Hi Diptanu,

Sure, I can do that.

Could you help me to find the best way to describe my need?

As far as I'm concerned, all options are already in place.

Thanks,

Vladimir

[Diptanu Choudhury]

Hi Vladimir,

We need the user to be able to specify the location of their tls cert and tls key files in the Nomad configuration. Since you are probably using a custom cert while setting up Consul's https, Nomad will have to know about that in order to talk to Consul.

[Vladimir Kozyrev]

Hi Diptanu,

I've created an issue https://github.com/hashicorp/nomad/issues/977

I hope that it describes my problem well.

Thanks for your help!

[Diptanu Choudhury]

Thanks! We have made the necessary changes in the Consul client, we will add a few options in the Nomad configuration, once that happens it would should work fine.

[Vladimir Kozyrev]

Thanks a lot!

[Diptanu Choudhury]

Hi,

This went out with Nomad 0.3.2, please give it a spin and let us know if it works as intended.