Versioning the entrie system configuration -- not artifacts

[Rinaldo DiGiorgio]

Hi,

  I tried to version my jobs with the Jobs SCM plugin for example and it often gets confused and ends up making my system unusable. Perhaps an entire rewrite is needed and the backend store needs to move to something like redis. I  don't think version control should be done with optional plugins. It should be part of the core system and all configuration data should be in a network store.  

  I can see a solution using docker where one makes a base image.  What happens when you change the configuration however. Do you make a new docker base image?  Some organizations want all the source code to generate an image in some type of SCM and this is the issue.  If you change job configurations or config params you just incurred the hit of a new image generation cycle.  Perhaps I am not looking it in the right way or in the future when everything is pipeline the configuration is pipeline with supporting json and property files.

Rinald

[Michael Lasevich]

Not familiar with "Jobs SCM" and unclear as to what you are trying to do - or how Redis or Docker fits it here. If you have some idea, you are welcome to write a plugin and make it work however you want, which is why the plugin system exists - and answers your question as to why this is done with plugin. But onestly, it sounds like you are making it far more complicated than it needs to be. Generally the setup would be a single, static Jenkins Master server that manages job scheduling and coordination and a number of "slave" nodes" that execute the job. Ideally slave nodes would be pristine and disposable and not have any code/tools installed on them - with master managing any code/tools requires - meaning they do not ever need to be versioned on the slave. 

Modern way of managing your jobs is via Pipelines. You just store your jobs along with your code (I assume your code is in some sort of a SCM to begin with) and for shared libraries use global library repo(s).  Jenkins does not really have a lot of configuration on Jenkins Master side after this, the only thing you really need here is a decent backup strategy - but if you are really interested in tracking individual job changes, there is a "Job Configuration History" plugin that will track each change and who did it, but with most of the job coming with source code, it is not all that critical/necessary.

HTH,

-M

[Victor Martinez]

Give a try job-dsl-plugin
- https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin

Supports 1000+ jenkins plugins, local testing, gradle integration, same Jenkins job paradigm, DRY concept and a bunch of other benefits besides of converting jobs in code and therefore scm oriented.

Cheers

[Michael Lasevich]

Ahh, "Job DSL",  I remember that. It was a good thing when it was the only game in town, but (in my opinion) Pipelines pretty much made it obsolete. Of course it is a matter of opinion, but if you are finding Jobs DSL too complicated, Pipelines may be just right for you - it removes a lot of the complexity, and makes your entire build process far simpler - you no longer need a rabbit-warren of jobs, and with MultiBranch Pipelines + Global Libs + something like Slack notifications, your devs may not even need to login to Jenkins server - just commit code and see notification that the job was created(if needed) and build was complete :-) Join the modern age :-)

-M

[Victor Martinez]

Both of them are good but they have different approaches... Although, IMO, Pipeline is still an incubating feature atm.

I wouldn't say Pipelines are better or worst or even obsolete or modern. If you go for configuration as code, by definition, code should be testable, and I could go further by saying testable locally and automatically, and unfortunately I haven't not seen that feature yet with the Pipelines.

Cheers

[Sean McNamara]

On Mon, Nov 7, 2016 at 3:53 PM, Michael Lasevich <mlas...@gmail.com> wrote:
> Ahh, "Job DSL", I remember that. It was a good thing when it was the only
> game in town, but (in my opinion) Pipelines pretty much made it obsolete. Of
> course it is a matter of opinion, but if you are finding Jobs DSL too
> complicated, Pipelines may be just right for you - it removes a lot of the
> complexity, and makes your entire build process far simpler - you no longer
> need a rabbit-warren of jobs, and with MultiBranch Pipelines + Global Libs +
> something like Slack notifications, your devs may not even need to login to
> Jenkins server - just commit code and see notification that the job was
> created(if needed) and build was complete :-) Join the modern age :-)

The big limitation of Pipelines is that Pipelines' Groovy code isn't
actually Groovy. It's a small-ish subset of Groovy that breaks in ways
that aren't extremely well-documented. You can't just import any
arbitrary Groovy code and expect it to work. A lot of folks who are
currently using the Groovy plugin running system groovy or external
groovy scripts will have to do major rewrites to dumb things down for
the Pipeline plugin.

Until/unless pipelines support all of Groovy, it's difficult to
recommend for scripts larger than a couple of lines. IMO, anyway.

Sean

>
> -M
>
> On Monday, November 7, 2016 at 11:55:20 AM UTC-8, Victor Martinez wrote:
>>
>> Give a try job-dsl-plugin
>> - https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
>>
>> Supports 1000+ jenkins plugins, local testing, gradle integration, same
>> Jenkins job paradigm, DRY concept and a bunch of other benefits besides of
>> converting jobs in code and therefore scm oriented.
>>
>> Cheers
>

> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/5200fb48-2be3-4f51-a26f-08681272a80c%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Michael Lasevich]

I did not mean to start a religious war - like I said, this is only an opinion. I only offered that Pipelines are a bit newer - and while CPS is a bear, it does appear, at least to me, to simplify things drastically - allowing much simpler setups and avoiding situations where it "gets confused and ends up making my system unusable". Perhaps this is the answer to original poster's current frustrations. Or maybe not. 

-M

[Michael Lasevich]

Depends on your perspective - what you call limitations I see as a huge benefit.  I am doing pretty large projects using Pipelines and it is has been rock solid and reliable. And yes, since Global Libs came around, you CAN run just about any Groovy or Java code you want- though you may not want to, as it does comes at a price (you usually lose one or more of convenience, or security, or ability to resume). Of course if you are transferring this code from existing system, you likely did not have those things in the first place. As such, I would consider any mass migration of code from other projects a major red flag and a security/reliability nightmare. I would always start by questioning if this is even necessary to port it in the first place - perhaps a real build tool would be a better choice. Overall the largest mistake I see people make  with Pipelines (and with Jenkins in general) is confuse Jenkins for a build system. It is not. It is a build job orchestration tool, and it orchestrates and executes those complicated things, but you are not meant to do complicated build things with it directly. if you find yourself writing complicated code for that - you are likely doing it very wrong. Nothing like that belongs in Jenkins JVM, not if you want any semblance of stability or security. If you understand that, you can do some majorly advanced things with very little code and if you do not, you end up with a confusing jumbled unstable mess. Thats just my option, I recognize that not everyone may share it, but it works for me.

-M

[Rinaldo DiGiorgio]

Thanks but I was doing CI in 1980 and it is IMO a non started that the core system doesn't implement versioning and uses a local file store instead of a network based store. Given the advances in technology I am seeing less and less value in systems that require more and more plugins for basic things that should be part of the core.