Nomad interaction with AWS autoscaling... how?

[Vic Putz]

Apologies for the denseness, but I don't understand how Nomad integrates with things like AWS autoscaling, even though things like the AWS Quickstart clearly involve autoscaling groups.

I see from recent searches that Nomad still (?) doesn't support auto-scaling of jobs with cluster size.  But could you set a launch template for the AWS autoscaler to start a new job with each new machine that comes up?  Or is that entirely pointless?  It seems like if the autoscaling groups do nothing but join the Consul/Nomad cluster but don't start jobs, there's not a lot of point to them, so I feel like I'm missing something.

(I had some confusion trying out Consul+Registrator with bringing servers up and down and not having them deregister, so I'm a little gunshy).

It's just a bit peculiar.  I love a lot of the things Nomad brings to administration, but if the rest of my team feels that the need for autoscaling dominates, it's hard to justify its use.  

Just curious and no worries either way; I know it's been a work in progress and not top of the priorities.  

[Daniel Middleton]

I may be misunderstanding your question (long day) but no schedulers I’m aware of interact directly with AWS’s ASGs (or similar). It’s down to you to monitor cluster utilisation and up the ASG max count if you feel you can afford to. The AMI (created with Packer or something) would then register itself as a Consul/Nomad Client and Nomad would start balancing Task Groups across them.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/08b2a485-89ac-422c-9901-9d47ec92cc7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vic Putz]

Daniel:

It was less direct interaction than trying to figure out if Nomad would help us at all or work with AWS auto-scaling in any useful way.

I guess I'm confused (which is fine; I spend much time that way).  We have a really simple situation right now--in our current AWS autoscaling, we have one server, wiith app X on it, in an autoscaling group.  I don't like how it's provisioned (a long and seemingly error-prone bash script run as "user data") but it's in a scaling group and supposedly at a given CPU load that's duplicated, a new instance with app X on it comes up, and amazon load balancing kicks in, and we can say we have autoscaling and load balancing.

I don't like that, because a) I don't like the provisioning method (I'd rather move to containers and have load balancing bring up something like a clean docker/nomad/consul server which gets provisioned after it comes up) and b) the only way to do things like rolling updates of an app is to bring the whole shebang down, install the new provisioning script, and restart from there (right now that's done through CodeDeploy)--which takes the 'rolling' out of rolling upgrades.  I'm also trying to add some more complex logging and microservices, so moving toward containerization is a logical step.  I have a deployment working with Consul, but it's pretty rigid.

...and it works, and with our current low user load it works well enough.  But I can't see any way to interface that with a scheduler like Nomad.  Bringing up new nodes in an autoscaling environment which already run app X doesn't seem in keeping, but bringing up new "blank" nodes doesn't do anyone any good until Nomad could add jobs/tasks--but if you have to add those jobs/tasks manually, then the auto is gone from the autoscaling.  Maybe if you can set the job count equal to the number of servers in a group?

It could well be that Nomad's simply not the right tool for us, but I like so much of what I see that I'd rather us switch to a tool that does more than we need in case we need it later (rather than having to switch everything when we're loaded up).

[pre...@hashicorp.com]

Vic, 

autoscaling task group(application) counts is not core functionality within Nomad. However, the Nomad community have built a couple of options for task group auto scaling, check these two projects out to evaluate if they meet your needs.

https://github.com/underarmour/libra

https://github.com/elsevier-core-engineering/replicator/wiki/Job-Scaling

We don't plan to add autoscaling into Nomad itself, however we have job variable interpolation in our roadmap.  That will enable turning instance count/CPU/Memory etc into interpolated variables, so that external tools can use the Nomad API to easily add/remove instances, increase memory resources etc.

Hope this helps

--Preetha (Nomad team at Hashicorp)

[Daniel Middleton]

You might be interested in this video from Hashiconf 2018 by Joyent about true (metrics-based) auto-scaling with Nomad: https://youtu.be/9oqrk-18Dzs

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/f4ec14bc-f75a-4d38-ab0c-96b34927a116%40googlegroups.com.