Scheduling work after a pipeline run

[Chad La Joie]

We're developing some pipelines that, as part of their run, will ensure that some test infrastructure is spun up in AWS.  This infrastructure may be used again by follow on runs of the same pipeline or runs of other pipelines that use the same infrastructure.

What I'd like to support is automatically shutting down the infrastructure after X amount of time of disuse.  Is there a way to do this?  I was thinking one way would be to write a Material that looked at some infrastructure usage data and kicked off a pipeline when a condition is met.  Such a Material could be used to trigger a shutdown pipeline for my current use case but could also support, for example, scale-up/scale-down pipelines in the future.

Any thoughts on how to do this?

[Jason Whittington]

You would not need to write a material. You could set up code independent of GoCD that runs on a schedule (e.g. a cron job) and query infrastructure. This job could trigger some kind of cleanup pipeline via the GoCD pipeline scheduling API. 

I don't really know that I recommend it though -- GoCD is best at "moving bits through environments" not "random admin tasks".  I think you might be happier if you just set up a cron job that queries the infrastructures and tears it down as needed and just keep the CI/CD tool out of it.

[Chad La Joie]

Yes, I could do that, but I disagree with your characterization of the request.

GoCD claims to be a CI/CD tool.  While there are definitely classes of apps that don't need anything other than a single server on which to build and run its integration tests, there are just as certainly classes of apps where that is not sufficient.  And, I can't think of any apps right now where deployment simply means "run build script on remote server".

So, I don't feel like what I'm asking for is "random admin tasks", rather I feel like it's an integral part of handling the deployment and, in many cases, integration testing of an app.  If not, what differentiates a CI/CD system from a build server?

On Fri, Nov 4, 2016 at 1:34 PM Jason Whittington <jason.wh...@gmail.com> wrote:

You would not need to write a material. You could set up code independent of GoCD that runs on a schedule (e.g. a cron job) and query infrastructure. This job could trigger some kind of cleanup pipeline via the GoCD pipeline scheduling API. 

I don't really know that I recommend it though -- GoCD is best at "moving bits through environments" not "random admin tasks".  I think you might be happier if you just set up a cron job that queries the infrastructures and tears it down as needed and just keep the CI/CD tool out of it.

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/SD9YlLPqIkQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zabil C M]

16.8+ release has the elastic agents beta feature.
You can write you own plugin for scaling up/down agents and how you want to assign work for jobs. More details for that here.
https://plugin-api.go.cd/current/elastic-agents/

To unsubscribe from this group and all its topics, send an email to go-cd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscribe@googlegroups.com.

[Jason Whittington]

GoCD claims to be a CI/CD tool.  While there are definitely classes of apps that don't need anything other than a single server on which to build and run its integration tests, there are just as certainly classes of apps where that is not sufficient.  And, I can't think of any apps right now where deployment simply means "run build script on remote server".

So, I don't feel like what I'm asking for is "random admin tasks", rather I feel like it's an integral part of handling the deployment and, in many cases, integration testing of an app.  If not, what differentiates a CI/CD system from a build server?

Good point.  I am going by some of my own experience with pipelines that do things like switch a server config.  It's tempting to put those tasks into pipelines but I think the key differentiator here is that pipelines are intended to move materials. The GoCD team docs and people tend to say thing like:

One of the reasons Go mandates materials for every pipeline is we want to promote the practise [of] everything (code, scripts, configuration etc.) is checked into SCM / use a package (created by upstream pipeline or pushed into some package repository)

(see, e.g https://github.com/gocd/gocd/issues/876)

The reason I think this task is maybe better served outside of GoCD is that it doesn't seem like something that would depend on materials.  You can kick off that task at any time -- with no inputs. If you try to model that kind of thing in GoCD you'll find yourself adding a "dummy" material to the pipeline which to me is a hint that you're trying to wedge something into GoCD that would be better done elsewhere (I've been down this road myself with pipelines like "RebootServer"). 

You can certainly do this but it will be kind of weird. Running that task isn't really part of "flowing a version of a material through an environment"   It's not a task that would meaningfully appear as part of a larger value chain.  As such I think you might find it more convenient in the long run to just keep it outside of GoCD.(e.g. set up a cron job or a Jenkins job or something similar.

Rock on!

Jason