Feature list / Roadmap for Go

[srinivas upadhya]

We had a discussion to come up with a feature list that we would like to work upon. The proposed features conform with the vision & roadmap for Go & CD in general. While this doesn't mean we will not pick up things that are not in the list, this is what we would like to focus most of our energy on.

API-first design - Goals: We want to make the Go Server UI interact with the backend through APIs, even more than it does now. The design of APIs would be such that the same could be used by any 3rd party to interact with Go Server. This would basically mean Go Server can run in head less mode allowing developers to write their own UI. The order in which we would like to go about doing this:

1. API for pipeline history, stage history, job history - along with pagination & related details like config md5 for that run, status etc.
   
API for agents - listing (along with sort by column name), search, add resource, environments, enable, disable, delete

API for pipeline creation - CRUD for group, pipeline / template, stage, job, task (along with env. var & parameters) config

API for pipeline dashboard & environment dashboard - show latest pipeline instances, trigger pipeline (along  with specific revisions & env. var), pause pipeline

API for search of pipeline based on parameters (checkin - SHA, message, author / pipeline - counter, label)
API for compare pipeline

Better support for Tests - Goals: The biggest time consumer in any CI/CD setup is running tests. It has become fundamental practice to write tests and every new framework supports writing & running tests first. Which means there are more tests to run & make sense of the results. We want to provide features to ease the pain.

Parallelization - out of box support for TLB? custom runner for each type (junit, rspec etc.) which contacts Go server asking for next test to run?


1. Real time reports (how many ran, how many failed & which ones failed - while the job is running)

2. Understand and provide pluggability for all kinds of test reports / migrators for other test report formats

3. Test metrics - make sense of test failures. possibly list flaky tests. Run tests in failed first order etc.
   

Dashboard Revamp / Rewrite - Goals: People spend most of their time on Go on dashboard page. With the number of pipelines & users increasing, the dashboard page needs to be optimized for performance, while giving as much info as possible in the best possible way for its consumption.

rewrite dashboard to hit API instead of getting the whole markup.

move some logic to client side to reduce load on server. more intelligent updates rather than full update?

1. UX improvements
   

Agent improvement - Goals: Improve Agent Server communication to reduce the problem of too many agents pinging the server at same time during start up & too many requests (1 ping + 1 work request) from agents.

1. Thundering herd - implement exponential backoff while registering so that Server is not swamped with requests.

2. Websocket - move from polling to server push model

Agent Management - Goals: In a CI/CD setup agent management is the hardest part. Remembering what is installed on what agent, knowing when an agent goes down, accessing agent machine to do some changes are a few things that would like to give first class support for.

1. Notification of status when a new agent is available (pending) or goes missing
   

2. Capabilities / Resource (add, edit, remove) - detect software available on agent along with versions & auto update resources of agent
   

3. Key management - manage ssh keys
   

4. Console - to make changes on agent without having to ssh to the agent
   

Tech Debt: Old version of JRuby and Rails in codebase - Goal: Go uses a 2.3.x release of Rails, which recently got deprecated. It causes Go to not be able to take advantage of new improvements in Rails. Causes people to have to use antiquated Rails contsructs rather than the new ones. We haven't been able to upgrade till now since rails 3 is not backward compatible & it's a daunting task to migrate all views at once - because of API changes in Rails, dependency on JRuby, RSpec and other gems. It feels like one upgrade was missed a few years ago and then it was too late.

Tasks:

• Will need to first upgrade JRuby and Jetty to the newer revisions, since newer Rails releases do not support the old JRuby revision that is used (corresponds to Ruby 1.8.6). Then, need to do a pretty big upgrade to all Rails code in the codebase. Need to bring in Bundler into the codebase for gem management.

Biggest hurdles will be:

• Default ERB syntax has changed, so most ERB files will not work out of the box, but will silently fail when viewed (will show ERB code on the rendered page).
• RSpec needs to be upgraded, since the old gem will not work with the new JRuby/Rails combination. Most tests will fail.

[chris...@nike.com]

I think this list looks good, I think the bullets under 2 are critical to making Go better for CD.

[josh.k...@gmail.com]

I like this list too.  Couple comments:

Re: #3 Dasboard Revamp/ Rewrite

 I like Go a lot for its focus on visibility of the pipepline state.  However, once you have more than a dozen pipelines it's almost impossible to understand it at a glance from the pipelines page.  In my opinion the great strength of Go is the Value Stream Map.  So make *this* the primary page.  Also, making value stream maps a primitive to work with in general, would be a great improvement.  I have a few other feature requests I'd add from that point of view:

VSM Feature Requests:

 1) A VSM Page (tab) in the view.  Some of us have more than 1 VSM and being able to see them all at once in some form would be great.  I'm not necessarily saying show all the expanded VSMs at once, but maybe a single tile with a progress bar: when you click on the progress bar, it can show the expanded VSM.  This would be *awesome* and is my biggest request of Go at the moment.

 2) Make the VSM a first class citizen in Go.  In other words, I should be able to create a VSM and add custom pipelines or pipeline templates, exactly like I do for a pipeline with Stages, Jobs, and Tasks.  It would be super-powerful if I could define my VSM at a VSM level, because then I can very quickly produce a high-level VSM that I can show to business people for approval and then flesh out the details of the pipelines later once I've gotten approval and funding.  As it stands right now, the VSM is a great tool to see where you *are* but I want to use it also define where I *want* to be.

Re: #5 Agent Management

  This is interesting, but not that great of an idea, in my opinion.  Agent management is a tricky topic and you'll never make it work right because people will want completely different things.  I've written some agent management tools and it just gets hairy really quickly.  The notifications are a nice idea, but resource management is a nightmare.  

  In fact, we are building some tools that abstract away a lot of this pain.  Specifically, we have built a dynamic agent pool that queries the Go server for the agents' status and scales the pool up and down based on some simple heuristics.  So if we have a pipeline that runs and checks our agent states.  If we need more machines, it calls into EC2 and makes  a new instance, installs the Go agent and a few pre-reqs and adds it to an environment.  A subsequent pipeline runs a simple build against that machine to make sure it's working and then adds it to the official environment pool.

  The resource management is still the tricky part so we handle this by making our resources Docker images.  Then we can just call 'docker run <image>' and it will pull the necessary resources and run them from our private (or could be public) Docker registry.  This wouldn't work for Windows (I suppose if you had an SA license, you could use App-V to do a similar approach).  

So my requests for Agent Management to make this easier would be something like:

  1) Auto-scale triggers: e.g. let the admin define a trigger that sends a signal when it happens.  e.g. "If number of idle machines drops below 2, fire AGENT_POOL_LOW signal flare"  We can then write a receiver for that signal and, say, create more EC2 or Azure instances or whatever.  And we could define another trigger that says, "If number of idle machines goes higher than 5, fire AGENT_POOL_HIGH signal flare". You can imagine these triggers potentially becoming more complicated (e.g. scaling machines with time of day, day of week, incoming requests, number of pipelines in progress, etc.) so that you could set up some really efficient predictors of machine usage.  I would just ask that the Go server would run the auto-scale trigger every X minutes where X is configurable and then acts accordingly.  This would give us great hooks to then go manage our agents pools easily.  Note, I'm not asking Go to create/manage AWS or Azure instances for us.  That might be interesting for, but I feel like it's a rabbit hole and VM create calls are all relatively easy.  But if you try to solve it generically for everyone it would be too hard to be worth it, plus I would also want to hook it up to my own VM farm or private cloud provider that wouldn't be on the list.

[srinivas upadhya]

On Fri, Dec 26, 2014 at 8:54 PM, <josh.k...@gmail.com> wrote:

I like this list too.  Couple comments:

Re: #3 Dasboard Revamp/ Rewrite

 I like Go a lot for its focus on visibility of the pipepline state.  However, once you have more than a dozen pipelines it's almost impossible to understand it at a glance from the pipelines page.  In my opinion the great strength of Go is the Value Stream Map.  So make *this* the primary page.  Also, making value stream maps a primitive to work with in general, would be a great improvement.  I have a few other feature requests I'd add from that point of view:

VSM Feature Requests:

 1) A VSM Page (tab) in the view.  Some of us have more than 1 VSM and being able to see them all at once in some form would be great.  I'm not necessarily saying show all the expanded VSMs at once, but maybe a single tile with a progress bar: when you click on the progress bar, it can show the expanded VSM.  This would be *awesome* and is my biggest request of Go at the moment.

​I suggested a dashboard which will show VSMs for pipelines that you care about as carousel.​ There are a few issues with that like performance etc. Will think more on these lines & 

 

 2) Make the VSM a first class citizen in Go.  In other words, I should be able to create a VSM and add custom pipelines or pipeline templates, exactly like I do for a pipeline with Stages, Jobs, and Tasks.  It would be super-powerful if I could define my VSM at a VSM level, because then I can very quickly produce a high-level VSM that I can show to business people for approval and then flesh out the details of the pipelines later once I've gotten approval and funding.  As it stands right now, the VSM is a great tool to see where you *are* but I want to use it also define where I *want* to be.

​Something like this thread?​

Re: #5 Agent Management

  This is interesting, but not that great of an idea, in my opinion.  Agent management is a tricky topic and you'll never make it work right because people will want completely different things.  I've written some agent management tools and it just gets hairy really quickly.  The notifications are a nice idea, but resource management is a nightmare.  

  In fact, we are building some tools that abstract away a lot of this pain.  Specifically, we have built a dynamic agent pool that queries the Go server for the agents' status and scales the pool up and down based on some simple heuristics.  So if we have a pipeline that runs and checks our agent states.  If we need more machines, it calls into EC2 and makes  a new instance, installs the Go agent and a few pre-reqs and adds it to an environment.  A subsequent pipeline runs a simple build against that machine to make sure it's working and then adds it to the official environment pool.

  The resource management is still the tricky part so we handle this by making our resources Docker images.  Then we can just call 'docker run <image>' and it will pull the necessary resources and run them from our private (or could be public) Docker registry.  This wouldn't work for Windows (I suppose if you had an SA license, you could use App-V to do a similar approach).  

So my requests for Agent Management to make this easier would be something like:

  1) Auto-scale triggers: e.g. let the admin define a trigger that sends a signal when it happens.  e.g. "If number of idle machines drops below 2, fire AGENT_POOL_LOW signal flare"  We can then write a receiver for that signal and, say, create more EC2 or Azure instances or whatever.  And we could define another trigger that says, "If number of idle machines goes higher than 5, fire AGENT_POOL_HIGH signal flare". You can imagine these triggers potentially becoming more complicated (e.g. scaling machines with time of day, day of week, incoming requests, number of pipelines in progress, etc.) so that you could set up some really efficient predictors of machine usage.  I would just ask that the Go server would run the auto-scale trigger every X minutes where X is configurable and then acts accordingly.  This would give us great hooks to then go manage our agents pools easily.  Note, I'm not asking Go to create/manage AWS or Azure instances for us.  That might be interesting for, but I feel like it's a rabbit hole and VM create calls are all relatively easy.  But if you try to solve it generically for everyone it would be too hard to be worth it, plus I would also want to hook it up to my own VM farm or private cloud provider that wouldn't be on the list.

​There is talk about this internally. There should be a thread discussing this sometime soon.​

[srinivas upadhya]

Sent it too fast!

I wanted to say "& get back on this thread".

Also, If somone is wondering here is where latest roadmap is kept updated.

[Derek Hammer]

I've a pull request for a dockerized go-agent to Ken's repository here: https://github.com/kmugrage/go-agent-docker/pull/1/files

That repo could be the docker image that GoCD uses internally for autoscaling agents, if we want to build on a container idea (something I would highly recommend.. thoughts below). This would be another killer feature of GoCD that would automate a ton of the pain that I see day in and day out at various clients. Everyone does agent management manually. I'm intrigued that some people find it painful enough to add autoscaling. Not all that surprised, though. One interesting thing is that we could enable this as a subsystem in the GoCD ecosystem that enables another sort of plugin or plugin-like architecture. There are many different types (VM, container, jails, etc) and service providers (Amazon, Rackspace, VMWare, etc) that would make this a simple do-it-one-way kind of thing.

It is my opinion that this subsystem should be built wholly separate from the main GoCD server. It would be a different process altogether and communicate solely through JSON over HTTP. It should use APIs for anything that the subsystem needs from the GoCD server (e.g. list of current agents.. though maybe that moves to this subsystem). And the GoCD server should query it for anything it needs (not sure if anything?). The Rails app can use either RESTful calls to the service directly from the client side (leveraging Angular? Is that in GoCD somewhere??) or through ActiveResource on the Rails side. Either way, the source of truth is in another process. 

Containers fit for gocd agents. Containers isolate away other things that are running on the VM/physical server. The isolation prevents things like filesystem collisions (2 agents writing to the same place) or resource locking (e.g. a hardcoded lock file in your app code that is in /tmp/my.lock and two agents are running it at the same time). It also fits with a grand master plan that I hope I can convince GoCD to go with.. that is small, quickly spun up go-agents that get created on demand for each job/stage/pipeline/whatever. They initialize and start up so quickly that we can create and destroy them with every run (making our runs super-clean). Then the strategy becomes for autoscaling to be managed at container host-machine cluster level and looking a mere resource usage to autoscale. Yay!

--
You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexandre Conrad]

Just to weigh in a vote with regards to the roadmap, I think the API is a huge deal. We're making some important decisions on which CD tool we're going to use at my company (SurveyMonkey) by evaluating many tools and my favorite is GoCD by far, and that's what I was planning on advocating for. We have a pretty large code base with many services that we want to deploy using our next CD pipeline tool. But then I wanted to script a few thing as a proof of concept to show off how easy it would be for us to migrate to Go by using its API but I then realized that there were no CRUD APIs, which is going to be a deal breaker for us. :(

I absolutely love GoCD's philosophy on how it wants to be able to run a headless pipeline eventually but we won't be able to use Go because of the lack of CRUD APIs, and switching CD pipelines is not something you do easily so we'll probably never get to use Go even if that feature landed a few months from now (unless you tell me there a branch pending for merge that we can start playing with).

So if I were to prioritize on to getting serious GoCD adopters, I'd make the CRUD APIs a top priority as it's so fundamental IMO.

--

[srinivas upadhya]

On Sat, Dec 27, 2014 at 4:53 AM, Alexandre Conrad <alexandr...@gmail.com> wrote:

Just to weigh in a vote with regards to the roadmap, I think the API is a huge deal. We're making some important decisions on which CD tool we're going to use at my company (SurveyMonkey) by evaluating many tools and my favorite is GoCD by far, and that's what I was planning on advocating for.

​:)​

We have a pretty large code base with many services that we want to deploy using our next CD pipeline tool. But then I wanted to script a few thing as a proof of concept to show off how easy it would be for us to migrate to Go by using its API but I then realized that there were no CRUD APIs, which is going to be a deal breaker for us. :(

​Yes. It really i​s sad that CRUD APIs aren't available. The problem is to comeup with something that can achieve the following goals for the API:

• should be directly usable for application through JS
• should require minimal work for supporting new tags, attributes etc.
• code should be re-usable - i.e. code written for adding stage & job into pipeline (Pipeline edit API) should be directly reused for adding job into stage (Stage edit API).

We are trying to come up with something. We want to get it right, once & for all. Till then I would suggest directly editing the XML. You will find this & this very interesting.

Now, even if the API was available I would highly recommend "re-think" the whole CI/CD infrastructure as opposed to just scripting the migration. Go is fundamentally different from Jenkins, in that it provides a lot more features (pipeline based modelling vs job based modelling, ability to add multiple materials (SCMs, Package Materials & Pipelines) to a pipeline, ability to fetch artifact from ancestor pipelines, Fan-in & VSM) altogether warrant a re-think.

[Josh Davis]

I agree the API is super important.

Derek, the way I've approached this is to evaluate Go with a smaller project.  We did that as a proof of concept and learning experiment first.  We are now in the stage of posting one of our projects over to Go completely to see what we learn in real life.  We've decided along the way that we want to migrate our whole product line but we will do it step by step and stage by stage.

I would warrant caution on a wholesale migration right away.  If you are using Jenkins, some other tool or something homegrown, you are bound to have some workflows that dont immediately map well to go.  They may be bad workflows, but nevertheless you may have business dependencies.  Go is great, but it *is* a fundamental workflow change (which again, is good), bur your migration will almost certainly not one to one.

Go has helped us uncover a ton of things about our process that were previously unknown or only known by a select few people.  Five yourself time foe that kind of an evaluation to discover those pitfalls.  Maybe if you can get some time to do a proof of concept, you might get more traction.

That said, I would love RESTful API support too. : )

from my mobile

---

From: Alexandre Conrad
Sent: ‎12/‎26/‎2014 6:23 PM
To: srinivas upadhya
Cc: go-c...@googlegroups.com; josh.k...@gmail.com
Subject: Re: Feature list / Roadmap for Go

Just to weigh in a vote with regards to the roadmap, I think the API is a huge deal. We're making some important decisions on which CD tool we're going to use at my company (SurveyMonkey) by evaluating many tools and my favorite is GoCD by far, and that's what I was planning on advocating for. We have a pretty large code base with many services that we want to deploy using our next CD pipeline tool. But then I wanted to script a few thing as a proof of concept to show off how easy it would be for us to migrate to Go by using its API but I then realized that there were no CRUD APIs, which is going to be a deal breaker for us. :(

I absolutely love GoCD's philosophy on how it wants to be able to run a headless pipeline eventually but we won't be able to use Go because of the lack of CRUD APIs, and switching CD pipelines is not something you do easily so we'll probably never get to use Go even if that feature landed a few months from now (unless you tell me there a branch pending for merge that we can start playing with).

So if I were to prioritize on to getting serious GoCD adopters, I'd make the CRUD APIs a top priority as it's so fundamental IMO.

On Dec 26, 2014 8:35 PM, "srinivas upadhya" <srinivas...@gmail.com> wrote:

[Josh Davis]

Derek gave you actually used the Docker Go Agent?  I think it is a fantastic idea but I hit a few hurdles.

1) I want to deploy everything as a Docker container, but since the Go agent is running inside of a container already, that means I am locked inside a container.  I tried experimenting with Docker-in-Docker but it doesn't seem mature enough to risk it and (more importantly) it doesn't mimic production: we have no plans of deploying containers in containers.  Not to mention it seems like an unnecessary management and debugging step.
2) I could deploy the Go agent to the machine and then deploy resources as docket agents but there's almost no easy way to do this since the Go Agent is already containerized and can't communicate with the host to start the other resource containers.  I think kubernetes or something similar can solve this problem, but it's too much overhead for us at this point and too many variables.  We are trying to stay focused on creating our value stream map first and then optimizing it later.
3) I also explored having multiple flavors of a Go agent container with all different kinds and classes of resources but the big problem with that is that the Docker private registry isn't mature enough to make that process easy.
4) I could also install the Go Agent natively which runs as a machine orchestrator and then launches an additional Go Agent in a Docker container  on the machine which does the actual work.  So I'd have a management pipeline that would create and manage additional go agents and other pipelines that use in the Docker go agents.  That just all seemed like a hack though.  If I already have to install and manage a natively installed Go agent I might as well manage all of them natively.

If you have other solutions that work for you, please say so.  The fundamental problem is that we have different resources to manage and no easy way to so this with Go in a Docker container.

I like your vision about agents being created on demand.  One big problem is build time dependencies.  At build time you need your build dependencies available to you.  I wanted to go your route but then realized we have to pull down our maven repo first which can take as much as an hour.  Similarly if you use NuGet and I'm sure it's true for other dependency management systems too.  Keeping it up to date and accessible is a nontrivial task if you are doing on-the-fly agent creation.

Just my thoughts.

jd
from my mobile

---

From: Derek Hammer

Sent: ‎12/‎26/‎2014 5:42 PM
To: srinivas upadhya
Cc: go-c...@googlegroups.com; josh.k...@gmail.com
Subject: Re: Feature list / Roadmap for Go

I've a pull request for a dockerized go-agent to Ken's repository here: https://github.com/kmugrage/go-agent-docker/pull/1/files

That repo could be the docker image that GoCD uses internally for autoscaling agents, if we want to build on a container idea (something I would highly recommend.. thoughts below). This would be another killer feature of GoCD that would automate a ton of the pain that I see day in and day out at various clients. Everyone does agent management manually. I'm intrigued that some people find it painful enough to add autoscaling. Not all that surprised, though. One interesting thing is that we could enable this as a subsystem in the GoCD ecosystem that enables another sort of plugin or plugin-like architecture. There are many different types (VM, container, jails, etc) and service providers (Amazon, Rackspace, VMWare, etc) that would make this a simple do-it-one-way kind of thing.

It is my opinion that this subsystem should be built wholly separate from the main GoCD server. It would be a different process altogether and communicate solely through JSON over HTTP. It should use APIs for anything that the subsystem needs from the GoCD server (e.g. list of current agents.. though maybe that moves to this subsystem). And the GoCD server should query it for anything it needs (not sure if anything?). The Rails app can use either RESTful calls to the service directly from the client side (leveraging Angular? Is that in GoCD somewhere??) or through ActiveResource on the Rails side. Either way, the source of truth is in another process. 

Containers fit for gocd agents. Containers isolate away other things that are running on the VM/physical server. The isolation prevents things like filesystem collisions (2 agents writing to the same place) or resource locking (e.g. a hardcoded lock file in your app code that is in /tmp/my.lock and two agents are running it at the same time). It also fits with a grand master plan that I hope I can convince GoCD to go with.. that is small, quickly spun up go-agents that get created on demand for each job/stage/pipeline/whatever. They initialize and start up so quickly that we can create and destroy them with every run (making our runs super-clean). Then the strategy becomes for autoscaling to be managed at container host-machine cluster level and looking a mere resource usage to autoscale. Yay!

[josh.k...@gmail.com]

​I suggested a dashboard which will show VSMs for pipelines that you care about as carousel.​ There are a few issues with that like performance etc. Will think more on these lines & 

 

Nice.  For what it's worth, I would prefer a vertical carousel and a enlarged view of all VSMs.  Like you point out in the original thread, I think this should be rendered by the client to take load off of the server.  Tangentially, is there a document somewhere that describes 

​Something like this thread?​

 

Something like that, yes :) 

Re: #5 Agent Management ...

​There is talk about this internally. There should be a thread discussing this sometime soon.​

 

 Sounds good.  I remember seeing a roadmap webpage somewhere?  Does that still exist?  I'm most interested because as we start to ramp up on Go and use it more broadly, it would be nice if some of these things have already been spec'd and planned so that when we want to make enhancements and submit pull requests, we can easily do it.  Github is great for capturing the granular issues, but I'm thinking more significant changes: like maybe helping with the VSM viewing pieces for example.  If the work is already planned and scoped, it is much easier for us to jump in and do it.

jd

[Ming Fang]

You can install the Docker client inside the Go Agent container.

The trick is to run the Agent content with the Docker server domain socket bind mounted.

Then the Agent can launch other Docker containers.

Here is an example https://github.com/mingfang/docker-gocd

[Mark Crossfield]

Here’s my late contribution…

I’m loving the API first design—this is absolutely the right approach. We would like to see APIs made available for clients to subscribe to events, so that people can develop plugins which integrate more deeply with Go’s inner workings. For example:

- Pipelines: build being scheduled, job completion, stage completion, state transitions (failed, passed, building…). These should include relevant information around commit messages, versions, repos, users…

- Agents: agent being used, agent becoming free, agent being added, agent being disabled, aggregated numbers of agents available based on specified resource combinations (to monitor whether more or less agents are required)…

- Monitoring: HTTP response code levels, response times percentiles grouped by code or verb or similar, amount of time jobs have to wait for agents…

Going with an event model for things like this could reduce the amount that other systems have to poll API feeds, potentially reducing the workload significantly.

Improving the performance around many agents calling back to Go also sounds like a great idea, having seen how many requests of this type come in (although they seem to be very efficiently handled).

The dashboard seems like a very difficult feature to complete to everyone’s satisfaction. I think improvements can absolutely be made (particularly with hundreds of pipelines), and hopefully with the API first approach it will be possible to efficiently create your own dashboards without hundreds and hundreds of expensive API calls. I think there are some big wins on usability, such as collapsing search results by pipeline groups so you don’t have to scroll through pages and pages of empty groups would be good, and it would be great to be able to get some feedback on clicks and saves in the admin interface (without having to jump into the developer console ;) ).

Agent management and test support seem okay to us at the moment, although it would be good to be able to see more useful information about the state of the agents—the agent version, other disk volume levels etc.

I think more and more our teams are going to become interested in testing Git branches, and being able to have one pipeline config automatically testing new branches would be awesome

Mark

[srinivas upadhya]

On Wednesday, February 4, 2015 at 11:21:31 PM UTC+5:30, Mark Crossfield wrote:

Here’s my late contribution…

I’m loving the API first design—this is absolutely the right approach. We would like to see APIs made available for clients to subscribe to events, so that people can develop plugins which integrate more deeply with Go’s inner workings. For example:

- Pipelines: build being scheduled, job completion, stage completion, state transitions (failed, passed, building…). These should include relevant information around commit messages, versions, repos, users…

- Agents: agent being used, agent becoming free, agent being added, agent being disabled, aggregated numbers of agents available based on specified resource combinations (to monitor whether more or less agents are required)…

Notification end-point should be available in next release. It has stage schedule & completion call backs. We will be adding more events in future releases.

 

- Monitoring: HTTP response code levels, response times percentiles grouped by code or verb or similar, amount of time jobs have to wait for agents…

Going with an event model for things like this could reduce the amount that other systems have to poll API feeds, potentially reducing the workload significantly.

Improving the performance around many agents calling back to Go also sounds like a great idea, having seen how many requests of this type come in (although they seem to be very efficiently handled).

The dashboard seems like a very difficult feature to complete to everyone’s satisfaction. I think improvements can absolutely be made (particularly with hundreds of pipelines), and hopefully with the API first approach it will be possible to efficiently create your own dashboards without hundreds and hundreds of expensive API calls. I think there are some big wins on usability, such as collapsing search results by pipeline groups so you don’t have to scroll through pages and pages of empty groups would be good, and it would be great to be able to get some feedback on clicks and saves in the admin interface (without having to jump into the developer console ;) ).

Agent management and test support seem okay to us at the moment, although it would be good to be able to see more useful information about the state of the agents—the agent version, other disk volume levels etc.

I think more and more our teams are going to become interested in testing Git branches, and being able to have one pipeline config automatically testing new branches would be awesome

There is quiet some interest in this. Ashwanth is working on Github PR builds support. I think you will find it interesting.

[Mark Crossfield]

On Thursday, 5 February 2015 05:15:22 UTC, srinivas upadhya wrote:

Notification end-point should be available in next release. It has stage schedule & completion call backs. We will be adding more events in future releases.

That’s fantastic news. Should I copy my suggestions over to that thread?

[srinivas upadhya]

Yes, please. Do list all types of events you would want Go to provide call back for. We will see how & when we can get it done. Thanks.

[Jakub Veverka]

Hi, 

I've raised topic connected to #5 and would like to ask if my usecase is part of this solution. 

So my proposal is to add possibility of extending agent management by plugin. So say there should be plugin for cloud providers like aws, azure, rackspace, mesos, ... 

Note: I will use 'resource' here, the meaning is usually virtual machine but I wouldn't want to limit to vm's only since there can be plugins for docker containers, maybe even physical hosts, who knows ;) 

Setup of such plugin would require: 

 - credentials to connect to cloud service

 - agent definition ( there can be multiple agent types on cloud provider):

  - agent label

  - template name (in case of vim, would be image name in case of docker)

  - resources per instance (ram,cpu, disk)

  - # of jobs per instance

  - custom setup (how long unused resource is kept alive, when to trigger new resource) 

The plugin would take care of: 

 - resource handling

  - bringing up resources

  - decommission unused resources

  - connecting agent

Workflow from go.cz itself would be:

New job -> define its agent label + other job related stuff

Trigger job -> job is waiting for agent node to be available -> cloud-plugin will detect it (is notified or checks it on its own???) -> cloud-plugin spins up new resource and ensures that agent connects with proper label to go server

No job is running -> plugin detects that there are no jobs on its resources so it decommission these based on custom rules

I hope this makes sense, if not let me know I will try to explain more...

This concept is not my own invention I was simply using jenkins which has such plugins and I must say its just perfect. Namely I was using jenkins-mesos plugin which was creating/deleting docker containers which worked as jenkins slaves. You can imagine that creating docker container on demand when build is needed is quite handy thing and in case of vm there could be some rule stating that there will always be enough resources for N jobs on given agent. 

jenkins mesos plugin is here: https://github.com/jenkinsci/mesos-plugin

also idea is presented on Ebay blogspot here: http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNtBP3VGjUY and here: http://www.ebaytechblog.com/2014/05/12/delivering-ebays-ci-solution-with-apache-mesos-part-ii/

btw this discussion started here: https://groups.google.com/forum/#!topic/go-cd/gU8UN-yJ3TA

Let me know what you think...

Jakub

[Md. Ali Ejaz]

Hi Jakub,

I've raised topic connected to #5 and would like to ask if my usecase is part of this solution.

Your usecase might not exactly be part of #5. It is however a part of the elastic agents that is being considered but not yet in our updated roadmap. One might say it's an extension of the agent capabilities mentioned in the roadmap :)

I hope this makes sense, if not let me know I will try to explain more...

It does make sense. I was also thinking on the similar lines :)

However what was concerning me, was if we have enough users for this feature. Should we invest our time in this or do some other feature!

What I mean is if this should be a first class support at all! It's not that cloud providers cannot be used today to manage Go and what if we document the process in a nice way and let users do the management through their scripts. I know it would be ugly! 

I would have to analyse more to come to any conclusion (As of now, i'm inclined to have this feature).

--

You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

- Ali

@mdaliejaz

[Jakub Veverka]

Hi Ali, 

thanks for your feedback, I understand your concern about not wasting time on features which wouldn't be used. 

I've seen few projects digging into this topic, but it would be superb to have some general example script with features allowing such a dirty job... 

But if you have any kind of voting system, please count my voice for elastic agents.

Jakub

[Jakub Veverka]

I would definitely like to see your python script, I don't have clue yet how this could work in go, so working example is IMO best way to dive in... 

Thanks,

Jakub

On Fri Feb 13 2015 at 12:41:09 AM Josh Davis <josh.k...@outlook.com> wrote:

I like the idea of elastic agents but it really isn't that hard to implement.

I wrote a relatively simple python script that does so.  The trick was querying for idle machines and adding machines when we hit a certain low threshold and deleting when it gets too high.   We only have 1 machine type and so we dont have to think about machine resources.  We use Docker to deploy most dependencies and to make sure the machines stay clean from testing.
 
The problem is actually resource management, environment manipulation, and proving newly created machines.

I would advise against building (yet another) auto scaler when so many exist.  I think rather than building a cloud provider plugin, you should allow for Orchestration plugins like Mesos, Kubernetes, Puppet, etc.  They already solve the machine management problem. Please don't build another.

I love the feature idea; we've built one in house (I can share it of it's of interest to anyone), but I'd rather see the investment in integrating existing tools.

josh
from my mobile

---

From: Jakub Veverka
Sent: ‎2/‎12/‎2015 3:58 AM
To: go-c...@googlegroups.com
Cc: veverk...@gmail.com

Subject: Re: Feature list / Roadmap for Go

You received this message because you are subscribed to a topic in the Google Groups "go-cd-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd-dev/EEMh7VLZV7I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd-dev+...@googlegroups.com.

[Md. Ali Ejaz]

@Josh The idea is not to build a cloud provider but to provide a first class support for integration of existing cloud providers through a plugin. 

Basically, the idea is to alleviate the burden from the user managing the scripts for the orchestrations. Go would manage it based on the plugin configuration. 

The plugin can be used by new users and much of the hassle can be avoided.

My doubts were on if it's a good enough an investment (time wise) to build this feature. Or documenting (with the help of people who are already using cloud orchestration) would be enough.

Also, if the resources are freed up every time, for a big repository it might take a lot of time in material checkout on the agents increasing the overall build time. From what I understand many users have big repositories, so will this feature be used as much!

Moreover, one might require agent logs for debugging issues, and if the resources are cleaned up every time (thinking out loud) do we copy the logs somewhere before the clean up!

I would spend sometime doing a proper analysis of this feature and post it here so others can comment on what they think :)

[Jakub Veverka]

Hi Ali, 

I would move the deciding logic from Go server to plugin, so instead of telling the plugin to spawn/bring down the agent server, it would just tell it how many builds are waiting for given agent type. Plugin would decide himself when to bring up new agent... Thoughts? ;) 

[Md. Ali Ejaz]

I agree with both of you (Josh and Jakub) that the logic should be in the plugin and not in Go. However this makes Go a bit too much dependent on the plugin! But then it's a design implementation and I would think about it later.

My concern on why not doing it, was only on the effort involved to provide the plugin end-points for this (assuming we go for plugin infrastructure). 

My question was if it would be sufficient to provide help in writing the scripts (documenting) like the one Josh is using and provide some more APIs if required!

On a side note, wouldn't the existing Agent API help in finding the states of the agents? I agree, having events on agent state change would help.

[Vasco Figueira]

Hi,

We currently have an external script running on AWS Lambda querying the Agents and Jobs APIs and writing metrics to Cloudwatch. These metrics drive the autoscaling policies for the GoCD agents' autoscaling groups. Very AWS-centric, I know.

Given that this approach has some limitations (Lambda is stateless, polling instead of reacting, etc) that cause some agent churn and suboptimal resource utilisation, we're looking into using a plugin for this.

Assuming the document 

https://docs.google.com/document/d/1Hf_6fpa1QZMM9U2niOwC74On-HKvj0wbQiWLy9WfDxk/edit#

still reflects the current thinking on this matter, I'd have a couple of suggestions regarding the design, mainly simplifications. I don't know, however, at what stage this is nor whether someone has started implementing anything yet.

I've done some initial prototyping and I think we can make do with just the notifications API (stage-status) and querying Go via the GoAccessor. Of course additional hook methods would be welcome. I'm not (yet) familiar with the codebase to know how easy would those be to add.

Comments?

(BTW, great to see the massive improvements in features and user traction in the past two years. I'm very pleased.)

Thank you.

Regards,

Vasco

[Aravind SV]

On Thu, Jul 7, 2016 at 1:25 PM, Vasco Figueira <vasco.f...@gmail.com> wrote:

still reflects the current thinking on this matter, I'd have a couple of suggestions regarding the design, mainly simplifications. I don't know, however, at what stage this is nor whether someone has started implementing anything yet.

I've done some initial prototyping and I think we can make do with just the notifications API (stage-status) and querying Go via the GoAccessor. Of course additional hook methods would be welcome. I'm not (yet) familiar with the codebase to know how easy would those be to add.

Comments?

(BTW, great to see the massive improvements in features and user traction in the past two years. I'm very pleased.)

I see that Ketan pointed you to #2058 on the gitter channel. As he said, it's getting merged soon and should provide you and others a way to (experimentally) try it out.

Any feedback or comments you have can be brought up there, I think.

Cheers,

Aravind