One-shot Agents, or on-demand Agents? For use with something like AWS.

[Kyle Hayes]

Yesterday, there was a question about spinning up new Agents on demand.  We want to go a little further. We have use for a system that would do the following for a pipeline:

* create new nodes (VMs with specific amounts of RAM, disk etc.) from a resource pool (e.g. OpenStack or AWS).

* configure the nodes into a small cluster.

* deploy new software for test across cluster (e.g. two DB nodes, a few web, a load balancer, ID management etc.).

* run tests (perhaps via yet more temporary nodes).

* save the results.

* tear down all the nodes and release their resources back into the pool.

We want to completely release Agent resources between runs to maximize the flexibility and minimize the cost.  Jobs may vary wildly in terms of the resources they need.  Systems like AWS can cost a lot less if we can release the Agent nodes as soon as the job is done.

One way I was thinking this might work (maybe not very scalable?) would be to have a small number of dedicated Agents watching the job queue.  When a new job appears, an Agent grabs it and sets up the environment (new instances/Agents) for it.  If I could force the new Agents to map to the one job, then I think I could do this.

I'm searching for ideas here. Am I approaching this the wrong way?

Best,

Kyle

[Ken Mugrage]

Actually, this is something the existing committers have been talking about quite a bit lately. I'm planning to send a message to this list in the next couple days talking about a set of features that could support elastic agents (cloud or not) for discussion. 

Right now doing what you're suggesting is possible, but as you've pointed out it's also very manual. Given the existing architecture the approach you mentioned seems reasonable. 

I hate the term "stay tuned", but please "stay tuned" for some discussion about this area :)

Ken

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ken Mugrage - ThoughtWorks Studios OSS Program Manager

Email

kmug...@thoughtworks.com

Telephone

+1 425 835 2056

Twitter

@kmugrage

[Kyle Hayes]

Thanks for the update, Ken,

This would be very interesting for us. It sounds like you are in the initial brainstorming stage?

Best,

Kyle

[Jens Melgaard]

Is there any issues or something else in relation to this at https://github.com/gocd/gocd ??
I tried to look for it as it's something that spikes high interest for me, not for the purpose of being elastic, but rather to always have agents starting in a known state before builds. (Which will come naturally i guess if you create a new agent from e.g. an appliance or similar each time)