Accelerating Jenkins development with Jenkins Essentials
164 views
Skip to first unread message
R. Tyler Croy
Feb 6, 2018, 11:03:05 AM2/6/18
to jenkin...@googlegroups.com
After pounding my head against my whiteboard for a couple months, I'm finally
ready to talk about this idea which was first discussed at Jenkins World 2017
in San Francisco.
At the Contributor Summit we held, Kohsuke challenged us to make Jenkins easy
enough for an end-user to go from "zero" (downloading Jenkins) to continuously
delivering a project in under 5 clicks and under 5 minutes.
Jenkins Essentials has this goal in mind, with an additional goal of making
Jenkins development more efficient.
I have proposed a draft of JEP-300 which outlines the problem domain and
overview of Jenkins Essentials more thoroughly
rendered: https://github.com/rtyler/jep/tree/essentials-overview-300/jep/300
pull request to jenkinsci/jep: https://github.com/jenkinsci/jep/pull/48
To be successful with Jenkins Essentials, I'm endeavouring to improve
pre-release verification as well as post-release monitoring. Some of the
initial tasks that my colleague Raul and I are tackling can be found on this
board: https://issues.jenkins-ci.org/secure/RapidBoard.jspa?rapidView=396
I'm still sketching out the Developer Experience aspects of this in another
thread once I have some thoughts written down.
Please take a look at the document and let me know what you think on this
thread :)
Cheers
- R. Tyler Croy
------------------------------------------------------
Code: <https://github.com/rtyler>
Chatter: <https://twitter.com/agentdero>
xmpp: rty...@jabber.org
% gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
------------------------------------------------------
R. Tyler Croy
Feb 7, 2018, 4:28:49 PM2/7/18
to jenkin...@googlegroups.com
In the same vein as my previous email, I've written up a design document which
outlines the current approach for packaging I intend to use for Jenkins
Essentials:
https://github.com/jenkinsci/jep/tree/master/jep/301
I think it should be fairly straight-forward and understandable why I'm taking
the approach described, but if there any ambiguity do let me know.
outlines the current approach for packaging I intend to use for Jenkins
Essentials:
https://github.com/jenkinsci/jep/tree/master/jep/301
I think it should be fairly straight-forward and understandable why I'm taking
the approach described, but if there any ambiguity do let me know.
Stephen Connolly
Feb 7, 2018, 6:14:23 PM2/7/18
to jenkin...@googlegroups.com
Looks good to me.
My only concern is mixing three VMs in the one container. Python, node and java... would be nicer if that could be reduced... but ack that it’s non-trivial to do so
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180207212838.ndyahjf7psu5obk2%40blackberry.coupleofllamas.com.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone
R. Tyler Croy
Feb 7, 2018, 7:12:29 PM2/7/18
to Stephen Connolly, jenkin...@googlegroups.com
(replies inline)
On Wed, 07 Feb 2018, Stephen Connolly wrote:
> Looks good to me.
>
> My only concern is mixing three VMs in the one container. Python, node and
> java... would be nicer if that could be reduced... but ack that it???s
> non-trivial to do so
I understand the desire to reduce dependencies but there are severe tradeoffs
to doing so, unfortunately :(
Java is an obvious dependency of Jenkins. Unfortunately Java is beefy at
runtime (especially with Jenkins), and doesn't have great support for acting as
a Server-sent Events client, along with some other niceities that I need for
the evergreen-client implementation.
As for Python, that's obviously a dependency for supervisord, but it also
shares some of the same limitations as Java for implementing an
evergreen-client. Python would definitely be my preference for implementing the
client tooling but alas. Also, I have not found a more mature, and featureful
process supervisor, in C, Node, etc, other than supervisord. I've been using it
in various projects with great success for a long time and it's well worth (to
me) the disk space occupied by the alpine python package.
Node is unfortunately the best tool for the job of the evergreen-client, and
believe me I'm not happy to be advocating for that position, since it dooms me
to writing much more JavaScript than I would like to :)
All that said, this is definitely not locked in stone. One of the big
advantages to defaulting to the container as the package means we can readily
swap this out in the future if we learn more, or our needs change.
Anywho, thanks for taking a read and providing some feedback! I hope I've
helped explain why there are some similar but different dependencies in the
jenkins/evergreen design.
On Wed, 07 Feb 2018, Stephen Connolly wrote:
> Looks good to me.
>
> My only concern is mixing three VMs in the one container. Python, node and
> non-trivial to do so
I understand the desire to reduce dependencies but there are severe tradeoffs
to doing so, unfortunately :(
Java is an obvious dependency of Jenkins. Unfortunately Java is beefy at
runtime (especially with Jenkins), and doesn't have great support for acting as
a Server-sent Events client, along with some other niceities that I need for
the evergreen-client implementation.
As for Python, that's obviously a dependency for supervisord, but it also
shares some of the same limitations as Java for implementing an
evergreen-client. Python would definitely be my preference for implementing the
client tooling but alas. Also, I have not found a more mature, and featureful
process supervisor, in C, Node, etc, other than supervisord. I've been using it
in various projects with great success for a long time and it's well worth (to
me) the disk space occupied by the alpine python package.
Node is unfortunately the best tool for the job of the evergreen-client, and
believe me I'm not happy to be advocating for that position, since it dooms me
to writing much more JavaScript than I would like to :)
All that said, this is definitely not locked in stone. One of the big
advantages to defaulting to the container as the package means we can readily
swap this out in the future if we learn more, or our needs change.
Anywho, thanks for taking a read and providing some feedback! I hope I've
helped explain why there are some similar but different dependencies in the
jenkins/evergreen design.
Message has been deleted
Kohsuke Kawaguchi
Feb 8, 2018, 2:06:10 PM2/8/18
to jenkin...@googlegroups.com
Yay! I think this is an important step for the Jenkins project. I'm really excited.
We started talking about this in some of the events adjacent to FOSDEM. I think people generally grok the idea and they are happy that we are attacking these challenges.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180206160253.dkz7whf7h74bda64%40blackberry.coupleofllamas.com.
For more options, visit https://groups.google.com/d/optout.
Kohsuke Kawaguchi
Jesse Glick
Feb 8, 2018, 5:17:00 PM2/8/18
to Jenkins Dev
https://github.com/jenkinsci/jep/blob/master/jep/301/README.adoc#plugins
does not make sense to me. A critical benefit of this system as I
understood it was that we could deliver a complete software
package—including Jenkins core and plugins—as a unit. So why are you
treating core and plugins as distinct things? You should either
· Package a particular version of core, and of all “essential”
plugins, together in a particular version of a Docker image, and have
people use a new tag of this image when they want to update anything.
That would be the normal way to manage software in Docker, though it
appears to contradict your intention of using `evergreen-client` to
perform upgrades.
or
· Have the image contain only the supervisor, and have it download
specific versions of Jenkins core and plugins on demand; and update
the image itself only when the supervisor changes (presumably rarely).
That also bypasses the concern noted in #security as the image would
not need to be updated merely because of a core security advisory.
https://github.com/jenkinsci/jep/blob/master/jep/300/README.adoc#sane-defaults
I do not think it is wise to try to automagically detect a cloud
environment. Rather, we should offer variant images which are
preconfigured for specific environments like AWS. Or offer startup
flags etc. to select such preconfigurations, and tailor the
installation scripts for those environments to pass the right flag.
does not make sense to me. A critical benefit of this system as I
understood it was that we could deliver a complete software
package—including Jenkins core and plugins—as a unit. So why are you
treating core and plugins as distinct things? You should either
· Package a particular version of core, and of all “essential”
plugins, together in a particular version of a Docker image, and have
people use a new tag of this image when they want to update anything.
That would be the normal way to manage software in Docker, though it
appears to contradict your intention of using `evergreen-client` to
perform upgrades.
or
· Have the image contain only the supervisor, and have it download
specific versions of Jenkins core and plugins on demand; and update
the image itself only when the supervisor changes (presumably rarely).
That also bypasses the concern noted in #security as the image would
not need to be updated merely because of a core security advisory.
https://github.com/jenkinsci/jep/blob/master/jep/300/README.adoc#sane-defaults
I do not think it is wise to try to automagically detect a cloud
environment. Rather, we should offer variant images which are
preconfigured for specific environments like AWS. Or offer startup
flags etc. to select such preconfigurations, and tailor the
installation scripts for those environments to pass the right flag.
Jesse Glick
Feb 13, 2018, 4:39:59 PM2/13/18
to Jenkins Dev
BTW
On Thu, Feb 8, 2018 at 5:16 PM, Jesse Glick <jgl...@cloudbees.com> wrote:
> why are you treating core and plugins as distinct things?
was meant as a fairly technical comment about the packaging design.
The goal here makes a lot of sense.
On this topic, is there any information on how you would propose new
snapshots of the system (core + plugin versions, I guess) get
“promoted” or “tagged” or whatever by Jenkins devs?
https://github.com/jenkinsci/jep/blob/master/jep/301/README.adoc#infrastructure-requirements
mentions that there would be some kind of build and test
infrastructure, but that is about it. A topic for another JEP?
On Thu, Feb 8, 2018 at 5:16 PM, Jesse Glick <jgl...@cloudbees.com> wrote:
> why are you treating core and plugins as distinct things?
The goal here makes a lot of sense.
On this topic, is there any information on how you would propose new
snapshots of the system (core + plugin versions, I guess) get
“promoted” or “tagged” or whatever by Jenkins devs?
https://github.com/jenkinsci/jep/blob/master/jep/301/README.adoc#infrastructure-requirements
mentions that there would be some kind of build and test
infrastructure, but that is about it. A topic for another JEP?
Kohsuke Kawaguchi
Feb 13, 2018, 4:43:15 PM2/13/18
to jenkin...@googlegroups.com
While this thread is quiet, between the previous rounds of discussions in Jenkins World contributor summit and the recent mailing list discussion, I think there's a consensus of the general problem space. I intend to accept this JEP here within several days. So please roll in your +1s!
Tyler will be working on this area, and he has been championing the education/arming of users as well as making Jenkins accessible to the broader users, on top of all the other things he has worked on over the past decade. I think he's well qualified to drive this effort, and as such I intend to delegate subsequent JEPs in this space to Tyler.
Kohsuke Kawaguchi
R. Tyler Croy
Feb 13, 2018, 7:04:10 PM2/13/18
to Jesse Glick, Jenkins Dev
(replies inline)
On Thu, 08 Feb 2018, Jesse Glick wrote:
> https://github.com/jenkinsci/jep/blob/master/jep/301/README.adoc#plugins
>
> does not make sense to me. A critical benefit of this system as I
> understood it was that we could deliver a complete software
> package???including Jenkins core and plugins???as a unit. So why are you
points you're raising here. I think we're conceptually on the same page, and
it took a bit of thinking and consultation with Kohsuke to help me fully
understand your feedback here.
I think the design document was unclear, and had a hidden premature
optimization/assumption. I have been thus far deriving jenkins/evergreen from
jenkins/jenkins simply to inherit some scripts, permissions, and other bits.
This is definitely not necessary, and I agree the packaging should follow the
second approach you describe, more or less:
1. Come online
2. Grab the latest, greatest, and secure-iest bits
3. Do a Jenkins \o/
I will make the JEP-301 document more clear on that.
> https://github.com/jenkinsci/jep/blob/master/jep/300/README.adoc#sane-defaults
>
> I do not think it is wise to try to automagically detect a cloud
> environment. Rather, we should offer variant images which are
> preconfigured for specific environments like AWS. Or offer startup
> flags etc. to select such preconfigurations, and tailor the
> installation scripts for those environments to pass the right flag.
I /think/ I understand where the reasoning would be for "flavors" depending on
the cloud environment (for example), but to make sure I'm not making any more
assumptions, would you mind expanding a bit on what your concerns are regarding
a singular "all-in-one" package which can include the necessary cloud-detection
auto-configuration tooling?
Thanks for taking the time to review both documents and provide thoughtful
feedback. I'll try to improve on my delay times, work responsibilities, you
know the drillThanks for taking the time to review both documents and provide
thoughtful feedback. I'll try to improve on my delay times, work
responsibilities, you know the drill.
*looks wearily at google calendar*
I'll try to land some updates to the prototype and these design documents with
your feedback tomorrow (Feb 14th).
On Thu, 08 Feb 2018, Jesse Glick wrote:
> https://github.com/jenkinsci/jep/blob/master/jep/301/README.adoc#plugins
>
> does not make sense to me. A critical benefit of this system as I
> understood it was that we could deliver a complete software
> treating core and plugins as distinct things? You should either
>
> · Package a particular version of core, and of all ???essential???
>
> plugins, together in a particular version of a Docker image, and have
> people use a new tag of this image when they want to update anything.
> That would be the normal way to manage software in Docker, though it
> appears to contradict your intention of using `evergreen-client` to
> perform upgrades.
>
> or
>
> · Have the image contain only the supervisor, and have it download
> specific versions of Jenkins core and plugins on demand; and update
> the image itself only when the supervisor changes (presumably rarely).
> That also bypasses the concern noted in #security as the image would
> not need to be updated merely because of a core security advisory.
I apologize for the delay, I needed some time to think about and understand the
> people use a new tag of this image when they want to update anything.
> That would be the normal way to manage software in Docker, though it
> appears to contradict your intention of using `evergreen-client` to
> perform upgrades.
>
> or
>
> · Have the image contain only the supervisor, and have it download
> specific versions of Jenkins core and plugins on demand; and update
> the image itself only when the supervisor changes (presumably rarely).
> That also bypasses the concern noted in #security as the image would
> not need to be updated merely because of a core security advisory.
points you're raising here. I think we're conceptually on the same page, and
it took a bit of thinking and consultation with Kohsuke to help me fully
understand your feedback here.
I think the design document was unclear, and had a hidden premature
optimization/assumption. I have been thus far deriving jenkins/evergreen from
jenkins/jenkins simply to inherit some scripts, permissions, and other bits.
This is definitely not necessary, and I agree the packaging should follow the
second approach you describe, more or less:
1. Come online
2. Grab the latest, greatest, and secure-iest bits
3. Do a Jenkins \o/
I will make the JEP-301 document more clear on that.
> https://github.com/jenkinsci/jep/blob/master/jep/300/README.adoc#sane-defaults
>
> I do not think it is wise to try to automagically detect a cloud
> environment. Rather, we should offer variant images which are
> preconfigured for specific environments like AWS. Or offer startup
> flags etc. to select such preconfigurations, and tailor the
> installation scripts for those environments to pass the right flag.
the cloud environment (for example), but to make sure I'm not making any more
assumptions, would you mind expanding a bit on what your concerns are regarding
a singular "all-in-one" package which can include the necessary cloud-detection
auto-configuration tooling?
Thanks for taking the time to review both documents and provide thoughtful
feedback. I'll try to improve on my delay times, work responsibilities, you
know the drillThanks for taking the time to review both documents and provide
thoughtful feedback. I'll try to improve on my delay times, work
responsibilities, you know the drill.
*looks wearily at google calendar*
I'll try to land some updates to the prototype and these design documents with
your feedback tomorrow (Feb 14th).
R. Tyler Croy
Feb 13, 2018, 7:04:36 PM2/13/18
to Surya Gaddipati, Jenkins Developers
(replies inline)
On Thu, 08 Feb 2018, Surya Gaddipati wrote:
> Hi,
>
> This is exciting new development.
>
> Couple of comments
>
> 1. Packaging supervisord with another executable is a docker anti pattern.
> I assume any serious jenkins deployment is on a scheduler like
> swarm/kubernetes ect where supervisord is not necessary.
I understand the point made but I disagree with the notion that "serious"
Jenkins deployments are using Swarm or Kubernetes. The Jenkins project itself
(for example) has been running Docker in production for over four years now,
without a container orchestrator of any kind. This includes
https://ci.jenkins.io/. We use the garethr/docker Puppet module
(https://forge.puppet.com/garethr/docker) which itself has millions of
downloads by folks managing Docker services "the hard way." That said, we also
have a Kubernetes environment where we're deploying newer applications (e.g.
https://plugins.jenkins.io/) into. So I /agree/ with the point that Kubernetes
is likely a good candidate for replacing some of the Jenkins lifecycle
management work that supervisord is currently performing, which is why I
mention Kubernetes in JEP-301 at all. However, I strongly
disagree that the market is "there" yet, I think making a container
orchestrator a requirement at this point would only inhibit adoption and
learning. I am instead erring on the side of "simple and fast" from a user
perspective, the closest we can get (IMHO) to the `java -jar jenkins.war`
experience, the easier the adoption and experimentation will be for the user
audience I want to reach.
> 2. Scripting should be done via yaml files, not environment variables(
> init.d/groovy). All essential plugins should be made compatible with
> configuration-as-code.
I fully intend on using as much support as possible from the Configuration as
Code effort which Ewelina and Nicolas are working on. I do not intend to
undertake, as part of Jenkins Essentials, patching or updating plugins in order
to support it however.
My order of priority for scripting Jenkins is:
1. Configuration as Code
2. Groovy scripting (bleh)
3. Hopefully very little if any scary hacks.
So I think we're mostly on same page here :)
> 3. Pipeline while powerful is contrary to the goal of getting new users
> started quickly. From my personal experience, very few new users have the
> appetite to learn groovy scripting to setup their builds, they just want to
> something to run `npm test`.
> A declarative yaml file like travisci,circle ect is the *correct*
> approach here.
I agree that this is an important goal for Jenkins as a whole to consider:
making Pipeline easier to use and be successful with. I believe those currently
hacking on Pipeline, like abayer, would agree with that sentiment. What *is* in
the scope of this work is making Jenkins easier to understand "out of the box"
(https://github.com/jenkinsci/jep/tree/master/jep/300#obvious-path) which I
believe is going to entail a number of copy-and-pasteable examples of common
Jenkinsfiles, and other forms of built-in examples and documentation.
Overall, thank you very much for taking the time to read JEPs and provide
feedback. The way I am interpreting your overall feedback, and please correct
me if I'm wrong, is that the direction is promising but want to encourage good
fundamentals on the implementation.
Anywho, thanks again :)
On Thu, 08 Feb 2018, Surya Gaddipati wrote:
> Hi,
>
> This is exciting new development.
>
> Couple of comments
>
> 1. Packaging supervisord with another executable is a docker anti pattern.
> I assume any serious jenkins deployment is on a scheduler like
> swarm/kubernetes ect where supervisord is not necessary.
I understand the point made but I disagree with the notion that "serious"
Jenkins deployments are using Swarm or Kubernetes. The Jenkins project itself
(for example) has been running Docker in production for over four years now,
without a container orchestrator of any kind. This includes
https://ci.jenkins.io/. We use the garethr/docker Puppet module
(https://forge.puppet.com/garethr/docker) which itself has millions of
downloads by folks managing Docker services "the hard way." That said, we also
have a Kubernetes environment where we're deploying newer applications (e.g.
https://plugins.jenkins.io/) into. So I /agree/ with the point that Kubernetes
is likely a good candidate for replacing some of the Jenkins lifecycle
management work that supervisord is currently performing, which is why I
mention Kubernetes in JEP-301 at all. However, I strongly
disagree that the market is "there" yet, I think making a container
orchestrator a requirement at this point would only inhibit adoption and
learning. I am instead erring on the side of "simple and fast" from a user
perspective, the closest we can get (IMHO) to the `java -jar jenkins.war`
experience, the easier the adoption and experimentation will be for the user
audience I want to reach.
> 2. Scripting should be done via yaml files, not environment variables(
> init.d/groovy). All essential plugins should be made compatible with
> configuration-as-code.
I fully intend on using as much support as possible from the Configuration as
Code effort which Ewelina and Nicolas are working on. I do not intend to
undertake, as part of Jenkins Essentials, patching or updating plugins in order
to support it however.
My order of priority for scripting Jenkins is:
1. Configuration as Code
2. Groovy scripting (bleh)
3. Hopefully very little if any scary hacks.
So I think we're mostly on same page here :)
> 3. Pipeline while powerful is contrary to the goal of getting new users
> started quickly. From my personal experience, very few new users have the
> appetite to learn groovy scripting to setup their builds, they just want to
> something to run `npm test`.
> A declarative yaml file like travisci,circle ect is the *correct*
> approach here.
I agree that this is an important goal for Jenkins as a whole to consider:
making Pipeline easier to use and be successful with. I believe those currently
hacking on Pipeline, like abayer, would agree with that sentiment. What *is* in
the scope of this work is making Jenkins easier to understand "out of the box"
(https://github.com/jenkinsci/jep/tree/master/jep/300#obvious-path) which I
believe is going to entail a number of copy-and-pasteable examples of common
Jenkinsfiles, and other forms of built-in examples and documentation.
Overall, thank you very much for taking the time to read JEPs and provide
feedback. The way I am interpreting your overall feedback, and please correct
me if I'm wrong, is that the direction is promising but want to encourage good
fundamentals on the implementation.
Anywho, thanks again :)
Manuel Jesús Recena Soto
Feb 14, 2018, 2:37:11 AM2/14/18
to Jenkins Developers
Hello Tyler,
A great document.--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180207212838.ndyahjf7psu5obk2%40blackberry.coupleofllamas.com.
For more options, visit https://groups.google.com/d/optout.
Manuel Jesús Recena Soto
Feb 14, 2018, 2:39:26 AM2/14/18
to Jenkins Developers
2018-02-08 0:14 GMT+01:00 Stephen Connolly <stephen.al...@gmail.com>:
Looks good to me.My only concern is mixing three VMs in the one container. Python, node and java... would be nicer if that could be reduced... but ack that it’s non-trivial to do so
I agree with this Stephen's comment. I would try to reduce the technology stack to Java (totally needed) and Python.
Regards,
On Wed 7 Feb 2018 at 21:28, R. Tyler Croy <ty...@monkeypox.org> wrote:
In the same vein as my previous email, I've written up a design document which
outlines the current approach for packaging I intend to use for Jenkins
Essentials:
https://github.com/jenkinsci/jep/tree/master/jep/301
I think it should be fairly straight-forward and understandable why I'm taking
the approach described, but if there any ambiguity do let me know.
Please take a look at the document and let me know what you think on this
thread :)
Cheers
- R. Tyler Croy
------------------------------------------------------
Code: <https://github.com/rtyler>
Chatter: <https://twitter.com/agentdero>
xmpp: rty...@jabber.org
% gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
------------------------------------------------------
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180207212838.ndyahjf7psu5obk2%40blackberry.coupleofllamas.com.
For more options, visit https://groups.google.com/d/optout.
--Sent from my phone
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxu9ZkdQJK%3D%2Bq8hj%3DFpHfn81j0eaaPh5dBWo8znuvtiBA%40mail.gmail.com.
--
Oleg Nenashev
Feb 14, 2018, 4:09:35 AM2/14/18
to Jenkins Developers
The proposal looks good to me.
Also +1 for the JEP-300 writeup. My suggestion would be to also create an Evergreen subproject page once it is approved.
BR, Oleg
среда, 14 февраля 2018 г., 8:39:26 UTC+1 пользователь Manuel Recena Soto написал:
Also +1 for the JEP-300 writeup. My suggestion would be to also create an Evergreen subproject page once it is approved.
BR, Oleg
среда, 14 февраля 2018 г., 8:39:26 UTC+1 пользователь Manuel Recena Soto написал:
2018-02-08 0:14 GMT+01:00 Stephen Connolly <stephen.al...@gmail.com>:Looks good to me.My only concern is mixing three VMs in the one container. Python, node and java... would be nicer if that could be reduced... but ack that it’s non-trivial to do soI agree with this Stephen's comment. I would try to reduce the technology stack to Java (totally needed) and Python.Regards,
On Wed 7 Feb 2018 at 21:28, R. Tyler Croy <ty...@monkeypox.org> wrote:
In the same vein as my previous email, I've written up a design document which
outlines the current approach for packaging I intend to use for Jenkins
Essentials:
https://github.com/jenkinsci/jep/tree/master/jep/301
I think it should be fairly straight-forward and understandable why I'm taking
the approach described, but if there any ambiguity do let me know.
Please take a look at the document and let me know what you think on this
thread :)
Cheers
- R. Tyler Croy
------------------------------------------------------
Code: <https://github.com/rtyler>
Chatter: <https://twitter.com/agentdero>
xmpp: rty...@jabber.org
% gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
------------------------------------------------------
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180207212838.ndyahjf7psu5obk2%40blackberry.coupleofllamas.com.
For more options, visit https://groups.google.com/d/optout.
--Sent from my phone
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxu9ZkdQJK%3D%2Bq8hj%3DFpHfn81j0eaaPh5dBWo8znuvtiBA%40mail.gmail.com.
Jesse Glick
Feb 14, 2018, 9:42:55 AM2/14/18
to Jenkins Dev
On Tue, Feb 13, 2018 at 7:03 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> I have been thus far deriving jenkins/evergreen from
> jenkins/jenkins simply to inherit some scripts, permissions, and other bits.
Yeah, that makes sense as a way to get things up and running quickly.
> I have been thus far deriving jenkins/evergreen from
> jenkins/jenkins simply to inherit some scripts, permissions, and other bits.
Once we have an accepted process for tagging a “release” of “Jenkins
Essentials” as a whole, as implied by
https://github.com/jenkinsci/jep/blob/master/jep/300/README.adoc#auto-update
then we can rewrite the image to handle core consistently.
>> I do not think it is wise to try to automagically detect a cloud
>
> […what are your concerns] regarding
> a singular "all-in-one" package which can include the necessary cloud-detection
> auto-configuration tooling?
A general distrust of anything magical and opaque that might or might
> auto-configuration tooling?
not work depending on factors too complex for users to see or
understand. If we have a recommended way for Jenkins to run in AWS,
polish it up and put it on Marketplace, if possible. If we have a
recommended way for Jenkins to run in a self-managed Kubernetes
cluster, publish an official Helm chart for it. But keep the basic
Docker image, i.e., whatever happens when I run something like
docker run --rm -v jh:/var/jenkins jenkins/evergreen
straightforward and predictable.
If you are worried about discoverability of our preconfigurations, you
can always add an `AdministrativeMonitor` that says something like
> Hey there! From sniffing around, it looks like you are running this Jenkins service in Azure, but you are not taking advantage of the preintegration work we have done for you.
> For more information, see: https://azuremarketplace.microsoft.com/etc.
> [[Dismiss]]
nicolas de loof
Feb 14, 2018, 10:03:03 AM2/14/18
to jenkin...@googlegroups.com, Surya Gaddipati
I haven't found any references to supervisord in JEP pull request, neither about kubernetes - did I miss something ?
Anyway I tend to agree with this comment about packaging supervisord with jenkins essential within a docker image as a bad practice. I don't think this has anything to relate to running this as plain docker or with some orchestrator, this just makes the docker image a mutable instance, which should be avoided. Without much details on where this discussion comes from I can't really tell much about alternatives.
> 2. Scripting should be done via yaml files, not environment variables(
> init.d/groovy). All essential plugins should be made compatible with
> configuration-as-code.
I fully intend on using as much support as possible from the Configuration as
Code effort which Ewelina and Nicolas are working on. I do not intend to
undertake, as part of Jenkins Essentials, patching or updating plugins in order
to support it however.
My order of priority for scripting Jenkins is:
1. Configuration as Code
2. Groovy scripting (bleh)
3. Hopefully very little if any scary hacks.
So I think we're mostly on same page here :)
Configuration-as-Code is not ready yet to support this, but for sure we will keep an eye on this and will try to implement this use-case as well when this becomes relevant.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180214000422.hkio3rohqdgzn4pd%40blackberry.coupleofllamas.com.
R. Tyler Croy
Feb 14, 2018, 10:10:37 AM2/14/18
to Jesse Glick, Jenkins Dev
(replies inline)
On Wed, 14 Feb 2018, Jesse Glick wrote:
> On Tue, Feb 13, 2018 at 7:03 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> > I have been thus far deriving jenkins/evergreen from
> > jenkins/jenkins simply to inherit some scripts, permissions, and other bits.
>
> Yeah, that makes sense as a way to get things up and running quickly.
> Once we have an accepted process for tagging a ???release??? of ???Jenkins
> Essentials??? as a whole, as implied by
> >
> > [???what are your concerns] regarding
agree, it's probably a good idea if for no other reason than to reduce the
opportunity for unintend side effects. I'll look at how I can work this nuance
into the JEP-300, I agree it's important to view different target platforms
(AWS, Azure, Kubernetes) differently and worth capturing that in this overview
document.
On Wed, 14 Feb 2018, Jesse Glick wrote:
> On Tue, Feb 13, 2018 at 7:03 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> > I have been thus far deriving jenkins/evergreen from
> > jenkins/jenkins simply to inherit some scripts, permissions, and other bits.
>
> Yeah, that makes sense as a way to get things up and running quickly.
> Essentials??? as a whole, as implied by
>
> https://github.com/jenkinsci/jep/blob/master/jep/300/README.adoc#auto-update
>
> then we can rewrite the image to handle core consistently.
>
> >> I do not think it is wise to try to automagically detect a cloud
> >> environment. [???]
> https://github.com/jenkinsci/jep/blob/master/jep/300/README.adoc#auto-update
>
> then we can rewrite the image to handle core consistently.
>
> >> I do not think it is wise to try to automagically detect a cloud
> >
> > [???what are your concerns] regarding
> > a singular "all-in-one" package which can include the necessary cloud-detection
> > auto-configuration tooling?
>
> A general distrust of anything magical and opaque that might or might
> not work depending on factors too complex for users to see or
> understand. If we have a recommended way for Jenkins to run in AWS,
> polish it up and put it on Marketplace, if possible. If we have a
> recommended way for Jenkins to run in a self-managed Kubernetes
> cluster, publish an official Helm chart for it. But keep the basic
> Docker image, i.e., whatever happens when I run something like
>
> docker run --rm -v jh:/var/jenkins jenkins/evergreen
>
> straightforward and predictable.
>
> If you are worried about discoverability of our preconfigurations, you
> can always add an `AdministrativeMonitor` that says something like
>
> > Hey there! From sniffing around, it looks like you are running this Jenkins service in Azure, but you are not taking advantage of the preintegration work we have done for you.
> > For more information, see: https://azuremarketplace.microsoft.com/etc.
> > [[Dismiss]]
Yeah, this is where I thought you were going with the original feedback and I
> > auto-configuration tooling?
>
> A general distrust of anything magical and opaque that might or might
> not work depending on factors too complex for users to see or
> understand. If we have a recommended way for Jenkins to run in AWS,
> polish it up and put it on Marketplace, if possible. If we have a
> recommended way for Jenkins to run in a self-managed Kubernetes
> cluster, publish an official Helm chart for it. But keep the basic
> Docker image, i.e., whatever happens when I run something like
>
> docker run --rm -v jh:/var/jenkins jenkins/evergreen
>
> straightforward and predictable.
>
> If you are worried about discoverability of our preconfigurations, you
> can always add an `AdministrativeMonitor` that says something like
>
> > Hey there! From sniffing around, it looks like you are running this Jenkins service in Azure, but you are not taking advantage of the preintegration work we have done for you.
> > For more information, see: https://azuremarketplace.microsoft.com/etc.
> > [[Dismiss]]
agree, it's probably a good idea if for no other reason than to reduce the
opportunity for unintend side effects. I'll look at how I can work this nuance
into the JEP-300, I agree it's important to view different target platforms
(AWS, Azure, Kubernetes) differently and worth capturing that in this overview
document.
Jesse Glick
Feb 14, 2018, 12:00:52 PM2/14/18
to Jenkins Dev
On Tue, Feb 13, 2018 at 7:04 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> I think making a container
> orchestrator a requirement at this point would only inhibit adoption and
> learning. I am instead erring on the side of "simple and fast" from a user
> perspective, the closest we can get (IMHO) to the `java -jar jenkins.war`
> experience, the easier the adoption and experimentation will be
FTR I agree with this approach for the vanilla `jenkins/evergreen`
> I think making a container
> orchestrator a requirement at this point would only inhibit adoption and
> learning. I am instead erring on the side of "simple and fast" from a user
> perspective, the closest we can get (IMHO) to the `java -jar jenkins.war`
> experience, the easier the adoption and experimentation will be
image: having a single, ready-to-run image trumps other
considerations. A `docker-run` command can in fact be _easier_ than
the traditional Jenkins installation experience because you do not
need to even download a WAR and figure out where to put it, install a
compatible JRE, etc.; or deal with native package management issues
such as version conflicts. We are closer to the old JNLP “Run Now!”
button.
If we have distinct packagings for more sophisticated environments, as
I recommended earlier, then we can define those packagings in a way
that more naturally fits those frameworks, for example by keeping the
supervisor and Evergreen client in distinct containers in a pod.
R. Tyler Croy
Feb 14, 2018, 1:26:14 PM2/14/18
to nicolas de loof, jenkin...@googlegroups.com, Surya Gaddipati
(replies inline)
On Wed, 14 Feb 2018, nicolas de loof wrote:
>
> I haven't found any references to supervisord in JEP pull request, neither
> about kubernetes - did I miss something ?
> Anyway I tend to agree with this comment about packaging supervisord with
> jenkins essential within a docker image as a bad practice. I don't think
> this has anything to relate to running this as plain docker or with some
> orchestrator, this just makes the docker image a mutable instance, which
> should be avoided. Without much details on where this discussion comes from
> I can't really tell much about alternatives.
Surya was referring to the second document in this thread which I sent out:
https://github.com/jenkinsci/jep/tree/master/jep/301
I understand the "bad practice" argument against this approach but I haven't
yet seen a counter-proposal which accomplishes what jenkins/evergreen needs to
accomplish, without using supervisord in the "basic" container case, or a
Kubernetes-like thing in the "advanced" case. I'm personally comfortable with
being less dogmatic about what goes into the container, understanding that
there's a future on the horizon filled with Kubernetes :)
As to the concern about the image mutability, the evergreen-client would only
be effectively changing content in the mapped `/var/jenkins_home` which must by
the very nature of Jenkins be mutable. I'll update JEP-301 to make that more
explicit.
Thanks for taking a look-see.
On Wed, 14 Feb 2018, nicolas de loof wrote:
>
> I haven't found any references to supervisord in JEP pull request, neither
> about kubernetes - did I miss something ?
> Anyway I tend to agree with this comment about packaging supervisord with
> jenkins essential within a docker image as a bad practice. I don't think
> this has anything to relate to running this as plain docker or with some
> orchestrator, this just makes the docker image a mutable instance, which
> should be avoided. Without much details on where this discussion comes from
> I can't really tell much about alternatives.
https://github.com/jenkinsci/jep/tree/master/jep/301
I understand the "bad practice" argument against this approach but I haven't
yet seen a counter-proposal which accomplishes what jenkins/evergreen needs to
accomplish, without using supervisord in the "basic" container case, or a
Kubernetes-like thing in the "advanced" case. I'm personally comfortable with
being less dogmatic about what goes into the container, understanding that
there's a future on the horizon filled with Kubernetes :)
As to the concern about the image mutability, the evergreen-client would only
be effectively changing content in the mapped `/var/jenkins_home` which must by
the very nature of Jenkins be mutable. I'll update JEP-301 to make that more
explicit.
Thanks for taking a look-see.
nicolas de loof
Feb 14, 2018, 2:17:42 PM2/14/18
to R. Tyler Croy, jenkin...@googlegroups.com, Surya Gaddipati
Something I don't get here is why you need an external evergreen-client to manage updates, and can't just get this running from within Jenkins.
I also wonder how you manage jenkins.war version here, which is bundled in the docker image vs upgrades. Does this mean the image comes with a default jenkins.war but won't override when used on an existing JENKINS_HOME ? And then upgrading and restarting service won't actually reflect a core upgrade ?
R. Tyler Croy
Feb 14, 2018, 3:06:43 PM2/14/18
to jenkin...@googlegroups.com, Surya Gaddipati
(replies inline)
On Wed, 14 Feb 2018, nicolas de loof wrote:
> Something I don't get here is why you need an external evergreen-client to
> manage updates, and can't just get this running from within Jenkins.
This is covered under "Alternative Approaches" here:
On Wed, 14 Feb 2018, nicolas de loof wrote:
> Something I don't get here is why you need an external evergreen-client to
> manage updates, and can't just get this running from within Jenkins.
https://github.com/jenkinsci/jep/tree/master/jep/301#extending-jenkins-itself
I think that section should help clarify things.
> I also wonder how you manage jenkins.war version here, which is bundled in
> the docker image vs upgrades. Does this mean the image comes with a default
> jenkins.war but won't override when used on an existing JENKINS_HOME ? And
> then upgrading and restarting service won't actually reflect a core upgrade
the resolution which I am incorporating into JEP-301 is that jenkins/evergreen
should be "small" and just have enough code inside the container to reach out
to the Evergreen hosted service layer and ask for the latest (core + plugins)
before starting Jenkins. Succintly put, this would mean jenkins/evergreen would
not ship with a jenkins.war.
Hope that helps
Cheers
nicolas de loof
Feb 14, 2018, 3:17:21 PM2/14/18
to jenkin...@googlegroups.com, Surya Gaddipati
ok, starting to get your point.
But then I wonder why you don't just get evergreen to be the process to start jenkins and control it as a child process, so you don't need extra supervisord
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180214200627.qqocqne3gynh6msx%40blackberry.coupleofllamas.com.
Carlos Sanchez
Feb 15, 2018, 11:36:25 AM2/15/18
to Jenkins Developers
yay! +1
I agree with previous comments that it is not ideal to run multiple processes and that in a kubernetes world this would possibly be a Pod with two containers, but I think it's important to get something out and iteratively improve.
A decent user experience today means just one container and we can always had a Pod definition for Kubernetes users if we find that useful in the future.
Regarding different environment autodetection it would be nice, but having a way to manually select it (system property, env var,...) would be good enough for now
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180207212838.ndyahjf7psu5obk2%40blackberry.coupleofllamas.com.
R. Tyler Croy
Feb 15, 2018, 12:45:34 PM2/15/18
to Carlos Sanchez, Jenkins Developers
(replies inline)
On Thu, 15 Feb 2018, Carlos Sanchez wrote:
> yay! +1
>
> I agree with previous comments that it is not ideal to run multiple
> processes and that in a kubernetes world this would possibly be a Pod with
> two containers, but I think it's important to get something out and
> iteratively improve.
> A decent user experience today means just one container and we can always
> had a Pod definition for Kubernetes users if we find that useful in the
> future.
>
> Regarding different environment autodetection it would be nice, but having
> a way to manually select it (system property, env var,...) would be good
> enough for now
Contemplating jglick's feedback further, I am more in agreement with his point
about providing disparate "flavors" to tailor the experience further for
user-success on cloud environments such as Azure or AWS. To that end I've
updated JEP-300: https://github.com/jenkinsci/jep/pull/58
The detail which makes this important is the suite of plugins necessary to
include "out of the box" to provide the Automatic Sane Defaults will change
depending on the environment. If we can detect AWS, then we can make sure we're
downloading the S3 plugin, for example, but on Azure we wouldn't want to bloat
the user's environment with it. Rather, Azure users should have the Azure VM
Agents and Azure Container Agents plugin included by default.
I will target AWS with the first "flavor" and then we can see where that takes
us.
Thanks for the feedback Carlos!
On Thu, 15 Feb 2018, Carlos Sanchez wrote:
> yay! +1
>
> I agree with previous comments that it is not ideal to run multiple
> processes and that in a kubernetes world this would possibly be a Pod with
> two containers, but I think it's important to get something out and
> iteratively improve.
> A decent user experience today means just one container and we can always
> had a Pod definition for Kubernetes users if we find that useful in the
> future.
>
> Regarding different environment autodetection it would be nice, but having
> a way to manually select it (system property, env var,...) would be good
> enough for now
about providing disparate "flavors" to tailor the experience further for
user-success on cloud environments such as Azure or AWS. To that end I've
updated JEP-300: https://github.com/jenkinsci/jep/pull/58
The detail which makes this important is the suite of plugins necessary to
include "out of the box" to provide the Automatic Sane Defaults will change
depending on the environment. If we can detect AWS, then we can make sure we're
downloading the S3 plugin, for example, but on Azure we wouldn't want to bloat
the user's environment with it. Rather, Azure users should have the Azure VM
Agents and Azure Container Agents plugin included by default.
I will target AWS with the first "flavor" and then we can see where that takes
us.
Thanks for the feedback Carlos!
Baptiste Mathus
Feb 15, 2018, 3:29:34 PM2/15/18
to Jenkins Developers
+1, very exciting to make Jenkins operatable by newcomers in a matter of minutes. Having had very recently to drive a newbie to use Jenkins, even if he was very motivated, I think this is a great initiative. The flexibility of Jenkins is a great strength for advanced users, but a real difficulty for newcomers.
Though I agree using a watchdog tool like supervisord is not the official and best way to provide a Docker image, at the same time I agree this is a powerful way to make the experience very easy OOTB. I did that in the past for SonarQube, and it worked quite well for our usage.
About supervisord specifically though, mostly a NIT/request for clarification: I wonder if the JEP should specify it? I read the reasoning and I understand this can save time and so on to start that work, but for instance if we discover a better alternative now or later, should a part of that JEP which somehow looks like an implementation detail (we mainly need to express we require a watchdog/restart tool don't we). Also/or this is typically a component that could be replaced later if something better surfaces or so?
Also, adding another language dependency (python) when you already plan to have the evergreen client in nodejs might make the image bigger than it could be. But well, back to my previous point, this is something that can anyway be improved/changed over time.
Thanks!
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180206160253.dkz7whf7h74bda64%40blackberry.coupleofllamas.com.
Baptiste Mathus
Feb 15, 2018, 4:06:39 PM2/15/18
to Jenkins Developers
Le 15 févr. 2018 21:29, "Baptiste Mathus" <m...@batmat.net> a écrit :
+1, very exciting to make Jenkins operatable by newcomers in a matter of minutes. Having had very recently to drive a newbie to use Jenkins, even if he was very motivated, I think this is a great initiative. The flexibility of Jenkins is a great strength for advanced users, but a real difficulty for newcomers.Though I agree using a watchdog tool like supervisord is not the official and best way to provide a Docker image, at the same time I agree this is a powerful way to make the experience very easy OOTB. I did that in the past for SonarQube, and it worked quite well for our usage.About supervisord specifically though, mostly a NIT/request for clarification: I wonder if the JEP should specify it? I read the reasoning and I understand this can save time and so on to start that work, but for instance if we discover a better alternative now or later, should a part of that JEP which somehow looks like an implementation detail (we mainly need to express we require a watchdog/restart tool don't we). Also/or this is typically a component that could be replaced later if something better surfaces or so?Also, adding another language dependency (python) when you already plan to have the evergreen client in nodejs might make the image bigger than it could be. But well, back to my previous point, this is something that can anyway be improved/changed over time.
Please ignore that part about the python runtime dependency since it's already been discussed/addressed in the other thread I hadn't read thoroughly yet.
Kohsuke Kawaguchi
Mar 20, 2018, 7:28:47 PM3/20/18
to jenkin...@googlegroups.com
I've assigned Tyler as the BDFL delegate of this JEP-301, given that I consider him the leader of this effort.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180215174522.ewqpuvbmjxchuj3j%40blackberry.coupleofllamas.com.
For more options, visit https://groups.google.com/d/optout.
Kohsuke Kawaguchi
Reply all
Reply to author
Forward
0 new messages