(another) docker slaves plugin - review request

[Surya Gaddipati]

This is not a hosting request. I am hoping to get some feedback on the plugin and see if its worth hosting/merging into other plugins.

Details are in the readme of the project 

https://github.com/suryagaddipati/jenkins-docker-slaves 

Pls, let me know if you have any questions.

Surya

[Oleg Nenashev]

I agree that independent Docker slave makes sense.
IIRC such functionality is provided by one of existing Docker plugins, but I'm not sure. There're too many of them :(

If no, I would vote for integrating the proposal into Docker-plugin if there is no showstoppers.

Best regards,
Oleg

воскресенье, 13 декабря 2015 г., 21:53:44 UTC+3 пользователь Surya Gaddipati написал:

[Surya Gaddipati]

Hi Oleg, 

Thank you for your feedback. 

Looks like DockerPlugin uses Jenkins clound api ( link)

This plugin uses QueueListener to provision slaves . 

Can these two approaches be integrated together ?

Surya

[Kanstantsin Shautsou]

Yes, i'm working on such thing in my new plugin, will notify when will have something working.

[Yoann Dubreuil]

Hi Surya,

As one of the author of https://github.com/ndeloof/docker-slaves-plugin , I was wondering what was missing to rewrite a simplified version of it.

Maybe we could work together to improve this plugin instead?

Yoann.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/fb9c00be-ad78-48c6-955b-7485e44d20c7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nigel Magnay]

I agree that independent Docker slave makes sense.
IIRC such functionality is provided by one of existing Docker plugins, but I'm not sure. There're too many of them :(

If no, I would vote for integrating the proposal into Docker-plugin if there is no showstoppers.

​docker-plugin already allows starting / stopping images. You may have additional fields (e.g volumes) that it perhaps does not expose.

​I believe docker-build-step also does this.

 

[Surya Gaddipati]

Hi Yoann,

Thank you for your response. 

>  I was wondering what was missing to rewrite a simplified version of it.

So here are couple of things that I found needed to be changed

1. changed to use spotify-docker api client instead of command line, I needed it to talk to docker-swarm to allocated nodes

2. Slaves are allocated based on job label not job property, much like cloud plugins

3. We didn't need to run builds in another image/side image ect, we use docker-compose to run builds. 

4. Had to remove couple of hacks like this, since we don't store builds on disk.

5. We didn't need jenkins user mounting .

This plugin simply allocates a new slave via jnlp/docker api calls. There is no extra magic. 

Surya

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

[Surya Gaddipati]

Hi nigelm, 

The important difference between this and all other docker slaves plugin is that this doesn't use jenkins' cloud api for the reasons listed here 

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

[Kanstantsin Shautsou]

On Dec 15, 2015, at 18:43, Surya Gaddipati <suryap...@gmail.com> wrote:

Hi Yoann,

Thank you for your response. 

>  I was wondering what was missing to rewrite a simplified version of it.

So here are couple of things that I found needed to be changed

1. changed to use spotify-docker api client instead of command line, I needed it to talk to docker-swarm to allocated nodes

Binary wrapping is solution that CloudBees choose for their plugins and seems this plugin follow it to have compatibility. Before they released first *docker* plugins i proposed to cooperate work on some library but they rejected it. 

2. Slaves are allocated based on job label not job property, much like cloud plugins

Thats feature and required to have ability to specify container from job (imagine that you can’t request admin to change images).

3. We didn't need to run builds in another image/side image ect, we use docker-compose to run builds. 

Ok, it just one of features.

4. Had to remove couple of hacks like this, since we don't store builds on disk.

5. We didn't need jenkins user mounting .

This plugin simply allocates a new slave via jnlp/docker api calls. There is no extra magic. 

Surya

P.S. i’m working on having ability to share slave configuration between Cloud configuration and Job Properties. Also with future docker-java release it will be possible to run remoting with pipes (spotify doesn’t support it afaik, dunno why you so want enforce plugins to use it). But that will be in yet-another-docker-plugin. 

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

This is not a hosting request. I am hoping to get some feedback on the plugin and see if its worth hosting/merging into other plugins.

Details are in the readme of the project 

https://github.com/suryagaddipati/jenkins-docker-slaves 

Pls, let me know if you have any questions.

Surya

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/ruikEEsF_bc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/57db95e2-f8df-4015-9d91-3705a852aa3d%40googlegroups.com.

[Kanstantsin Shautsou]

Surya, could you describe your use case because it seems that you want two orthogonal features in one?

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/ruikEEsF_bc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/64c4afda-60ea-4d3c-9e9a-17a9dec91ce1%40googlegroups.com.

[Surya Gaddipati]

Hi Kanstantsin ,

Use case is exactly this https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin 

"The aim of the docker plugin is to be able to use a docker host to dynamically provision a slave, run a single build, then tear-down that slave." 

But without going through Jenkins Cloud Api. 

Surya

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

[Kanstantsin Shautsou]

Hi, that’s wrong info about docker-plugin. docker-plugin support:

- provisioning jenkins slaves 

- building/publish images (duplicated/duplicates docker-build-publish-plugin)

- Docker container controls

All functionality is written via docker-java library layer. 

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/ruikEEsF_bc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d5035c31-39ec-4c34-94f8-457f2ed555bd%40googlegroups.com.

[Kanstantsin Shautsou]

So what is your problem around Cloud API based provisioning? It’s jenkins Cloud API implementation (probably the best in FOSS plugins).

There are some issues for provisioning delay (that is not fully right described in ndeloof’s plugin README).

Could you definitely describe what exactly you want achieve?

On Dec 15, 2015, at 18:59, Surya Gaddipati <suryap...@gmail.com> wrote:

[Surya Gaddipati]

>So what is your problem around Cloud API based provisioning?

1. Delay in provisioning -> There is no technical reason user has to deal with this.  This is totally a problem we can solve . 

2.  Container booting logs should be available to user via build logs, since allocation of container is part of the build. This is not currently possible. 

Surya

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

[Kanstantsin Shautsou]

On Dec 15, 2015, at 19:17, Surya Gaddipati <suryap...@gmail.com> wrote:

>So what is your problem around Cloud API based provisioning?

1. Delay in provisioning -> There is no technical reason user has to deal with this.  This is totally a problem we can solve . 

2.  Container booting logs should be available to user via build logs, since allocation of container is part of the build. This is not currently possible. 

From Jenkins architecture when job is in queue it just an item and there is no build objects. Probably you can copy provisioning logs into build in launcher onAcceptedTasks() step. Or optionally it should be possible attach slave log to build object as action and provide log from separate link (imho is good enough).

Surya

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

This is not a hosting request. I am hoping to get some feedback on the plugin and see if its worth hosting/merging into other plugins.

Details are in the readme of the project 

https://github.com/suryagaddipati/jenkins-docker-slaves 

Pls, let me know if you have any questions.

Surya

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/ruikEEsF_bc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/87834380-5694-41d4-bbb2-25ce55640b12%40googlegroups.com.

[Surya Gaddipati]

So, do you agree that  cloud api is not good enough for docker containers ?

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

[Nigel Magnay]

I haven't fully grokked how the alternative launch scheme works (I assume it's operating as a launcher higher up the food chain). I don't know if this is giving you options of controlling the build environment before it is launched based on the build it is about to execute. 

I would imagine you could add a similar launcher into (say) docker-plugin. 90% of this stuff is the same (UI to capture user settings); the heavy lifting is deferred to a library anyway (e.g docker-java); there's only really a tiny difference in each (cloud api implementation, launcher).

The cloud API is "sufficient" (in that bar the point above) it allows the lifecycle you need. There's plenty of people using it that way. That's not so say that it doesn't have problems. What's a touch depressing is ISTM that the non-cloud-API approach is eventually going to come full-circle and re-implement all the same stuff the cloud API is supposed to :-/ E.g: balancing between hosts based on their current active workload. I would have though that the two things you mention (pause before starting, attaching start logs to the relevant build) ought to be solveable in the core -- that way all cloud provisioners would benefit..

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0c0844f7-8c5f-410a-9781-ca1c04588686%40googlegroups.com.

[Surya Gaddipati]

nigelm, 

nigelm,

It uses queue listener to look for a label and add a slave . 

Docker has many, many load balancing tools , docker swarm for example. You can simply point jenkins to your swarm instance for docker. 

I see no reasons why Jenkins would be doing load balancing with docker containers when there are much more advanced/sophisticated solutions out there. 

Surya

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

[Nigel Magnay]

nigelm,

It uses queue listener to look for a label and add a slave . 

Docker has many, many load balancing tools , docker swarm for example. You can simply point jenkins to your swarm instance for docker. 

I see no reasons why Jenkins would be doing load balancing with docker containers when there are much more advanced/sophisticated solutions out there. 

​Oh, that's easy - because you may not be using swarm. You may have docker hosts that are unable​ to participate in a single swarm. You may have slaves that aren't docker that you'd also like to use.

I think this is besides the point I was trying to make. I'll invert your question. Do you think that the cloud API ought to be good enough for docker containers ?

 

[Surya Gaddipati]

> Do you think that the cloud API ought to be good enough for docker containers ? 

Not quite sure what you mean. But yea cloud api should support single use throwaway slaves that are tied to a single build. 

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

[Yoann Dubreuil]

Hi Surya,

Thanks for your reply, I've a few questions below.

On Tue, Dec 15, 2015 at 4:43 PM, Surya Gaddipati <suryap...@gmail.com> wrote:

Hi Yoann,

Thank you for your response. 

>  I was wondering what was missing to rewrite a simplified version of it.

So here are couple of things that I found needed to be changed

1. changed to use spotify-docker api client instead of command line, I needed it to talk to docker-swarm to allocated nodes

AFAICT, you can use a swarm endpoint with docker CLI,  you don't need a special API for this. The docker API is a fast moving target, so wrapping the CLI was convenient for experimenting.

 

2. Slaves are allocated based on job label not job property, much like cloud plugins

We had that feature in mind too, but didn't implement it yet.

 

3. We didn't need to run builds in another image/side image ect, we use docker-compose to run builds. 

That's interesting, I don't understand why you need a docker slave then ? Is it just to run a SCM checkout and then call docker-compose to do the build ? I imagine that you are bind mounting the docker socket in the slave node so you can run docker-compose?

 

4. Had to remove couple of hacks like this, since we don't store builds on disk.

5. We didn't need jenkins user mounting .

Not sure to understand, do you mean that you don't need to tie a docker volume to a container?

 

This plugin simply allocates a new slave via jnlp/docker api calls. There is no extra magic.

Ok, so you just want a simple docker slave to be able to use docker-compose.

The fundamental idea behind our plugin was to replace the process startup API with docker. Because in the end, what you want to do to build a software is to run some commands in a controlled environment. Currently, Jenkins achieves this by allocating a build slave, by installing all the tools you need in it and by calling execve. This is exactly what Docker is for. From my POV, Docker is an a API to launch a process in a controlled environment. The container story is not interesting here. That's why I don't think the Jenkins Cloud model fits well, Docker is not a way to provide a slave from a light VM system. It is convenient but it's not the optimal way to integrate Docker and Jenkins (again, my POV).

You use case it not that far, the difference is instead of starting a random process from Jenkins after the checkout, you start docker-compose.

Feel free to correct me if I'm wrong. I really would like to understand your use case well so I can understand the gap with existing plugins.

 

Surya

On Sunday, December 13, 2015 at 12:53:44 PM UTC-6, Surya Gaddipati wrote:

This is not a hosting request. I am hoping to get some feedback on the plugin and see if its worth hosting/merging into other plugins.

Details are in the readme of the project 

https://github.com/suryagaddipati/jenkins-docker-slaves 

Pls, let me know if you have any questions.

Surya

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/57db95e2-f8df-4015-9d91-3705a852aa3d%40googlegroups.com.

[Kanstantsin Shautsou]

I missed my favorite question :)

Jenkins Cloud API is not enough for anything IMHO. It was actual 5 years ago when AWS appeared. Today you may want more levels of provisioning:

DC locations/groups/machines/containers in it.

All cloud plugins are hacks and copy-pastes.

Imho issue is in Jenkins design that ends into NodeProvisioner, Labels and locked Queue.

@Nigelm proposed PR for having NodeProvisioner request to have groups of slaves, but it wasn't merged. 

IMHO So the only way of implementing cloud related things is making plugins full of hacks (around QTD, job types, launchers, etc). 

Let's wait for 2.0, 3.0, 4.....?

[Andrew Bayer]

Yeah, it's painful and needs serious revisiting. I'd be up for discussions around that for 2.0 or soon after. I think that'd be valuable.

A.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2479ec52-fafb-4e20-8eb3-39d15810268b%40googlegroups.com.

[Stephen Connolly]

Can you please bash [with a soft cardboard tube] KK on the head (being as how your are physically closer and may actually have a chance to). 

I have been complaining about how unfit for purpose the Cloud API is since Paul Sandoz and I were working on DEV@PE (which turned into CJOC)... i.e. back in 2011... every time he has pushed back with "oh it can't be that bad... I don't hear anyone else complaining about it"... we need to get some time so that we can consolidate the learning we have from CJOC and push that down into a new API that replaces the current "bag of shite"

-Stephen

[Andrew Bayer]

When I get back to California in early January, I will do so. And yeah, let's start trying to pull thoughts/notes together!

A.

[Oleg Nenashev]

+1 for finally including it into Andrew's "Jenkins 3.0"

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/ruikEEsF_bc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPbPdOYmw1gtAK4JqPybRK5VOAL%2BPKuE6UY5x2jRJTbVzb4XiQ%40mail.gmail.com.

[Andrew Bayer]

And yeah, I really need to get my act together and start discussions on the 3.0/Jenkins.NEXT stuff...

A.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLCdBxY7T6YXSs2EEo%2BiVyv6Ru%3D%2BeiBqRHwjv%2BsR0CLquA%40mail.gmail.com.