Kubernetes plugin advice

[Carlos Sanchez]

Hi!

I'm creating the Kubernetes plugin [1] to start slaves in Kubernetes or Google Container Engine, which basically means launching Docker containers across multiple hosts with just one API.

Some background: Kubernetes allows launching individual containers (Pods) or creating a ReplicationController, which accepts a Pod template and a number of how many containers you want to run, and can be resized dynamically, a pool of containers.

Initially I though about implementing the plugin using a ReplicationController and resizing it as new slaves are needed, but seems that Jenkins operation doesn't fit well with this idea. It would prevent killing specific slaves, it would just be a pool. So I went ahead and moved to launch Pods one by one.

The issue I have now is about initialization time, where the container is not accepting connections until the image is downloaded and started. The Docker plugin has this comment [2]

// Docker instances may have a long init script. If we declare

// the provisioning complete by returning without the connect

// operation, NodeProvisioner may decide that it still wants

// one more instance, because it sees that (1) all the slaves

// are offline (because it's still being launched) and

// (2) there's no capacity provisioned yet.

//

// deferring the completion of provisioning until the launch

// goes successful prevents this problem.

slave.toComputer().connect(false).get();

Should I do the same?

There is also the issue that Kubernetes would be better suited for swarm slaves. Opening ssh or ports in each pod is more complicated than just doing the connection to the master from the container.

Thanks!

[1] https://github.com/carlossg/jenkins-kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesCloud.java

[2] https://github.com/jenkinsci/docker-plugin/blob/master/src/main/java/com/nirima/jenkins/plugins/docker/DockerCloud.java#L239

[Magnus Sandberg]

Hi,

I think it would be great with an internal extension point for a "scheduler" where you easily can adapt to different sub-systems. I primarily think of AWS ECS at this point but there are others.

/Magnus 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5e641038-b83b-4901-83a8-4bb54255498a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen Connolly]

Carlos, please sync up with me to get my full rant on the deficiencies of the Cloud API. I can point you to our "best" implementations (which sadly for the community are closed source - not my call)

FWIW the current Cloud API is IMHO a crappy extension point and it is almost impossible for anyone to implement correctly. There is, to my knowledge, no correct implementation of the Jenkins Cloud API. The "best" implementations I am aware of are currently closed source but IMHO it's no big loss to the community as even those have issues.

The changes I made in 1.607 correct for some of the issues, but w.r.t. the Cloud API it's more lipstick on a pig.

On my backlog is to replace the Cloud API with a new and much easier to implement extension point. I gave myself the hooks to do that in either 1.592 or 1.588 and it should be possible to implement as a plugin rather than waiting for changes in core.

Every single cloud implementation has issues as soon as you start to put it under load. With oc-cloud I am able to mask a large chunk of the issues by virtue of doing the required coordination from Operations Center and by doing the state management in a local transaction database... but all of that is far far too much overhead to ask every cloud implementation to do. Jenkins *should* be doing it for the cloud implementer...

In short, I believe what the docker-plugin is doing is "a severely sub-optimal path" and it only gets away with it when under light-medium load. I can cause it to "go bad" with some of my stress tests that I used to develop the fixes for 1.607

[Kanstantsin Shautsou]

I think all Cloud plugin developers will be glad to know about Cloud API problems and their changes. Really difficult to follow massive changes in core. Any comments about docker-plugin will be appreciated. Also it will be very cool if you review https://github.com/jenkinsci/jenkins/pull/1448 

[Stephen Connolly]

On Friday, April 24, 2015, Kanstantsin Shautsou <kanstan...@gmail.com> wrote:

I think all Cloud plugin developers will be glad to know about Cloud API problems and their changes. 

So my plan is to do a new cloud-api in a plugin. It makes it easier to update and experiment. I have sufficient changes made in core to allow for a new cloud api in a plugin that targets 1.588+

 

Really difficult to follow massive changes in core. Any comments about docker-plugin will be appreciated. Also it will be very cool if you review https://github.com/jenkinsci/jenkins/pull/1448 

That pull request is just putting more lipstick in the pig. I see the use case you are after and I believe the correct approach is via a new cloud API that handles all the crazy that does need to be handled and removes the crazy that doesn't actually need to be there (but is needed for the current API)

HTH (though probably not what you wanted to hear ;-) )

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0e6ca23c-9fc2-41c0-9e02-b555c0d1bde7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Sent from my phone

[Kanstantsin Shautsou]

On Apr 25, 2015, at 01:13, Stephen Connolly <stephen.al...@gmail.com> wrote:

On Friday, April 24, 2015, Kanstantsin Shautsou <kanstan...@gmail.com> wrote:

I think all Cloud plugin developers will be glad to know about Cloud API problems and their changes. 

So my plan is to do a new cloud-api in a plugin. It makes it easier to update and experiment. I have sufficient changes made in core to allow for a new cloud api in a plugin that targets 1.588+

Sounds like a plan.

 

Really difficult to follow massive changes in core. Any comments about docker-plugin will be appreciated. Also it will be very cool if you review https://github.com/jenkinsci/jenkins/pull/1448 

That pull request is just putting more lipstick in the pig. I see the use case you are after and I believe the correct approach is via a new cloud API that handles all the crazy that does need to be handled and removes the crazy that doesn't actually need to be there (but is needed for the current API)

HTH (though probably not what you wanted to hear ;-) )

Well, feel free to provide cloud-api-plugin for mass review. Will be glad to help.

[Kanstantsin Shautsou]

Hi, Stephen. 

I reviewed code in almost all cloud plugins and it seems to me that jenkins cloud api (DockerSlave -> DockerComputer (Launcher/retention)) looks not suitable for cloud usage at all. Some plugins just creating vm and then passes hardcodes to this implemented classes, other do run in Launcher.launch() that may end in issues.

Do you have any draft versions or real plans for cloud-api plugin?

On Saturday, April 25, 2015 at 1:33:45 AM UTC+3, Kanstantsin Shautsou wrote:

[Stephen Connolly]

We are currently in release close out mode. I hope to get some time to work on the cloud api after we get the release out... So hopefully something in the next couple of weeks.

As I said before, the hooks I need are in Jenkins 1.592+ so this new API can be added in a plugin targeting 1.592+ though obviously the threading issues with deprovisioning will only be resolved on 1.607+ ;-)

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/f238d6f3-a48d-4336-a149-5b0aefee19af%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nigel Magnay]

On Friday, April 24, 2015, Kanstantsin Shautsou <kanstan...@gmail.com> wrote:

I think all Cloud plugin developers will be glad to know about Cloud API problems and their changes. 

So my plan is to do a new cloud-api in a plugin. It makes it easier to update and experiment. I have sufficient changes made in core to allow for a new cloud api in a plugin that targets 1.588+

 

Really difficult to follow massive changes in core. Any comments about docker-plugin will be appreciated. Also it will be very cool if you review https://github.com/jenkinsci/jenkins/pull/1448 

That pull request is just putting more lipstick in the pig. I see the use case you are after and I believe the correct approach is via a new cloud API that handles all the crazy that does need to be handled and removes the crazy that doesn't actually need to be there (but is needed for the current API)

HTH (though probably not what you wanted to hear ;-) )

​Actually, *any* indication of forward movement is good to hear.

That patch is over 6 months old without any attention (which makes me sad, since passing data containers through a build pipeline would be extremely useful), and I'm sure there's been statements along the lines of "cloud API = bad = unusable" for perhaps twice as long as that.​

​I'll happily spend time to adapt docker-plugin to a newer API​.