JEP-213: Pluggable Configuration Storage for Jenkins

[Oleg Nenashev]

Hi all,

I have submitted a new JEP-213 for Pluggable Configuration Storage. This JEP is an attempt to document how I would see https://github.com/jenkinsci/jenkins/pull/3393 proposed by Alex Nordlund.

Motivation: Currently Jenkins stores all configurations in the file system within JENKINS_HOME. Such configuration complicates backup management and disaster recovery, because JENKINS_HOME contains lots of other information in addition to sensitive configs. Externalizing them is a also critical task for getting highly-available or disposable Jenkins masters. Currently it is possible to externalize the entire JENKINS_HOME by using shared file storages (e.g. NFS), but there is no engine in Jenkins which would allow moving files outside the filesystem. Plugins like SCM Sync Configuration externalize the configurations, but they duplicate the information and do not act as a main storage for Jenkins initialization.

While working on this story, I have identified several topics which may worth a discussion:

• Configuration Storage driver. Should it be a plugin or a library?
• Is it enough to have XmlFileStorage? Or do we need an abstraction for other configuration file types directly in the core?
• Is there a need to support automatic data migration?

I have added my vision of this topics to the Reasoning section of the JEP. I will appreciate any feedback.

Best regards,

Oleg

[Jesse Glick]

[list owner, please set default reply to be to list]

On Thu, Aug 9, 2018 at 12:52 PM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> Is it enough to have XmlFileStorage? Or do we need an abstraction for other configuration file types directly in the core?

There are a few places in core that use, e.g., `FileBoolean` which
will not be covered by `XmlFile` calls.

> Is there a need to support automatic data migration?

IMO no.

[James Nord]

"because JENKINS_HOME contains lots of other information in addition to sensitive configs"

by default, it is possible to change this.  You move build artifacts to a separate location (already plugable), extract plugins and war (already possible) to a different directory, move workspaces to a different directory (if you even allow builds on master!!)  and what you have left in Jenkins home should be configuration (of which I include build results).

> Is there a need to support automatic data migration? 

Well you need to be able to load data from existing builds and stuff stored in the new place.

You need to be able to load jobs stored in file ssytem and the ones that have been saved since it was updated...

And once externalized if there is no migration path you will never acheive the goal as you will need to replicate the file system.

So IMO you need to be able to support migration, if that is part of Jenkins startup is a different question, and I would say no it should not be automatic as you

1) do not want to block startup of Jenkins whilst all config since the dawn of time is migrated
2) want to be able to test and verify this migration for which purpose it should be non destructive.

[Jesse Glick]

On Fri, Aug 10, 2018 at 11:18 AM James Nord <jn...@cloudbees.com> wrote:
> what you have left in Jenkins home should be configuration

More or less I think. There are likely to be a lot of corner cases,
like update center caches etc. etc., but OTOH we should just fix those
somehow.

BTW be careful not to confuse configuration with mutable state.
`XmlFile` can store both, sometimes in the same file. JEP-201 deals
only with configuration.

> (of which I include build results)

Build results are not suitable for storage in anything like this IMO.

> you need to be able to load data from existing builds and stuff stored in the new place.

I do not think so—we can just say that this applies to new instances only.

[Baptiste Mathus]

Le ven. 10 août 2018 à 19:57, Jesse Glick <jgl...@cloudbees.com> a écrit :

On Fri, Aug 10, 2018 at 11:18 AM James Nord <jn...@cloudbees.com> wrote:
> what you have left in Jenkins home should be configuration

More or less I think. There are likely to be a lot of corner cases,
like update center caches etc. etc., but OTOH we should just fix those
somehow.

I agree with the "more or less". It's true that it's mostly feasible to segregate important data and variable/transient one, but it requires a fair bit of config, as we did for Essentials + that basically no plugin deviates or makes false assumptions (example JENKINS-52123).

BTW be careful not to confuse configuration with mutable state.
`XmlFile` can store both, sometimes in the same file. JEP-201 deals
only with configuration.

> (of which I include build results)

Build results are not suitable for storage in anything like this IMO.

> you need to be able to load data from existing builds and stuff stored in the new place.

I do not think so—we can just say that this applies to new instances only.

--
You received this message because you are subscribed to the Google Groups "Jenkins Cloud Native SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-cloud-nati...@googlegroups.com.
To post to this group, send email to jenkins-clou...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkins-cloud-native-sig/CANfRfr0xte5ZbnW3g-BSLM8cgen1CxRxDCarENQMfC_gLZLT%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.