Dynamic ports and command line parameters

[David Maze]

I'm trying to do a paper evaluation of deploying our system on Nomad.  There are maybe a dozen internal service processes, all packaged as Docker containers, some but not all of which communicate via HTTP.

The place I'm getting hung up is this: the "Networking" section of https://www.nomadproject.io/docs/drivers/docker.html pretty strongly suggests that I should let Nomad pick arbitrary ports for my process, https://www.nomadproject.io/docs/job-specification/network.html too.  It's easy enough to start my process with -bind 0.0.0.0:${NOMAD_PORT_http}, or to use the Docker port mapping.

When I start my client tasks, where do I tell them to connect?  I can't just say -connectTo http://server.service.consul/ because Nomad almost certainly won't have picked port 80; and I don't have a config file (I need either command-line parameters or environment variables) so it seems like consul-template won't work for me.  I've seen some suggestions to use HAProxy as an intermediary, also helping the case where there is more than one copy of the service, but this seems like it just kicks the problem down the road.

Maybe the right answer is to use a dynamic port for each service instance, then pick static ports for the load balancer (this is hinted in the network { ... } documentation), and then tell clients -connectTo http://proxy.service.consul:12345/?  Should I have one proxy per service, or one per deployment?

("How we do it today" is a hand-rolled set of docker run scripts, where we have in fact picked a static port per service, but that makes upgrades and migrations a problem.)

[Alex Dadgar]

Hey David,

I think you have identified the possible solutions! You can either run a load-balancer/proxy or you will have to get the address to the applications on run-time.

For the first approach I would suggest looking into Fabio: https://github.com/fabiolb/fabio

Fabio will automatically detect service from Consul and you can create routes via tags so it is incredibly easy to use with Nomad’s built in Consul registration features.

For the second approach if you are limited to env-vars/flags you currently have one work around. Generate the config using the template integration and then run a bash script that parses the address and launches your actual task with those values.

Future versions of Nomad will support populating env-vars/cli flags with values retrieved from consul.

Thanks,
Alex Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/6e2b0fea-1f3a-4797-a344-07a8714172b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.