[Essentials] Using non default locations for builds and workspaces (was: [JENKINS-49406] Evergreen snapshotting data safety system pre-JEP: feedback welcome)

[Baptiste Mathus]

Hey everyone, 

I thought I'd start a dedicated email here about that part after some feedback I got in https://github.com/batmat/jep/pull/1.

I know people are very busy so maybe an email is easier for many than looking at the PR.

So the one the ideas I have for Essentials' data snapshotting system is to use the configuration parameters to put builds and workspaces under a single and non default root. 

Concretely, with this setting, (simplified) we'd have under JENKINS_HOME (instead of everything under jobs as it is usually):

var/

└── the_job

    ├── builds

    │   ├── 1

    │   │   ├── build.xml

    │   │   ├── changelog.xml

    │   │   └── log

    └── workspace

jobs/

└── the_job

    ├── config.xml

Jesse and James are even saying we should move the var directory somewhere else *not* under JENKINS_HOME.

Also, saying that we should not even (try to) put the builds (and workspace I assume) in the Git repository, since it is not scalable.

I'm in mind to adjust the JEP with that proposal, though it somehow conflicts with that statement in the JEP-301 that "all mutable state and data will be written and stored in JENKINS_HOME".

Any opinion about this?

Any good or bad experience using those parameters to move build records and/or workspaces somewhere else on the disk?

Thanks!

2018-03-14 18:55 GMT+01:00 R. Tyler Croy <ty...@monkeypox.org>:

(replies inline)

On Wed, 14 Mar 2018, Baptiste Mathus wrote:

> Hello everyone,
>
> For Jenkins Essentials
> <https://github.com/jenkinsci/jep/tree/master/jep/300>, one critical
> requirement is to be able to upgrade, and hence rollback in an automated
> manner.
> So, as we are committed to an open design
> <https://github.com/jenkins-infra/evergreen#open-design> process, I have
> written a first draft of the associated Jenkins Enhancement Proposal.
>
> It is up for review at https://github.com/batmat/jep/pull/1
>
> I am very eager for any kind of feedback there.
> I am especially interested in catching & clarifying (more or less) glaring
> holes in that design.


Thanks for taking the time to send this out Ba(p)tiste! Now that I've had a
chance to take a look, I think the one thing that's missing from this document
is a bit more explanation of the problem which requires this solution.

My take on this problem space is that core and plugin upgrades can result in
modification of config.xml and other object-serialized-files on disk when an
upgrade occurs. As these files are serialized from objects in memory, when an
internal API changes within a plugin/core, it will necessarily result in
changes to files on disk. These changes may not be safe to "rollback" from,
i.e. Plugin A v0 cannot load a file generated by Plugin A v1.

This means an upgrade of Jenkins Essentials has a very real potential to cause
irreversible modifications to files on disk which prevent a safe rollback.


So that type background/context is (IMHO) missing a bit from the JEP document.

I think the Motivation section should also explain a bit more explicitly that
"bricking" a Jenkins Essentials instance is a severe failure for the project,
and thus we need to prevent against irreversible modifications to files causing
runtime failures for the Jenkins Essentials installation.


Overall, I think this looks quite reasonable. I look forward to seeing the
implementation and tests we get to write to support it :)


Cheers
- R. Tyler Croy

------------------------------------------------------
     Code: <https://github.com/rtyler>
  Chatter: <https://twitter.com/agentdero>
     xmpp: rty...@jabber.org

  % gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180314175527.yeofyld6a2d2p4ro%40blackberry.coupleofllamas.com.
For more options, visit https://groups.google.com/d/optout.

[R. Tyler Croy]

(replies inline)

On Thu, 15 Mar 2018, Baptiste Mathus wrote:

> I thought I'd start a dedicated email here about that part after some
> feedback I got in https://github.com/batmat/jep/pull/1.
> I know people are very busy so maybe an email is easier for many than
> looking at the PR.

Thank you for bringing this discussion back to the jenkinsci-dev@ mailing list,
I think this will give more people to an opportunity to weigh in and
consolidate our discussions into threads for easy linking in the JEP
"References" section.

More below..

> So the one the ideas I have for Essentials' data snapshotting system is to
> use the configuration parameters to put builds and workspaces under a
> single and non default root

> <https://github.com/batmat/jep/blob/03d3404aba079b2f9b67284f6fc9940bf2ccfacd/jep/302/README.adoc#segregate-job-configuration-and-build-data>
> .
>
> Concretely, with this setting, (simplified) we'd have under *JENKINS_HOME*
> (instead of everything under *jobs* as it is usually):
> var/
> ????????? the_job
> ????????? builds
> ??? ????????? 1
> ??? ??? ????????? build.xml
> ??? ??? ????????? changelog.xml
> ??? ??? ????????? log
> ????????? workspace
> jobs/
> ????????? the_job
> ????????? config.xml
>
> Jesse <https://github.com/batmat/jep/pull/1#discussion_r174508332> and James
> <https://github.com/batmat/jep/pull/1#discussion_r174472740> are even

> saying we should move the var directory somewhere else *not* under
> JENKINS_HOME.

I cannot say I have ever intentionally moved /jobs/ around in this manner, I
assume that it "just works" and there's no quality concerns we should have from
a non-standard placement of data on disk?

Otherwise I have no problems with this.

> Also, saying that we should not even (try to) put the builds (and workspace
> I assume) in the Git repository, since it is not scalable.

From my understanding of the potential failure scenarios with a plugin or core
upgrade is that unreadable build.xmls won't present a problem, or be mutated by
a new plugin update, correct?

If that's the case then I don't see a problem with leaving them out of the
snapshotting system. Generally speaking, I believe we should be mostly tracking
data which will be mutated on upgrade, or other potentially high importance
data.

> I'm in mind to adjust the JEP with that proposal, though it somehow
> conflicts with that statement in the JEP-301 that "all mutable state and
> data will be written and stored in JENKINS_HOME

> <https://github.com/jenkinsci/jep/tree/master/jep/301#mutable-data>".


When I wrote `JENKINS_HOME` I was actually just using that as a short-hand for
what is commonly understood as `/var/lib/jenkins` to most people. Basically,
mutable state should not be scattered around inside the container's temporary
filesystem, but rather contained in the single mounted volume, e.g.:

docker run -v /srv/jenkins:/var/jenkins_home jenkins/evergreen


So long as we're mounting everything through a single volume mount for the
container, my concerns/requirements for JEP-301 are met. Does that make sense
to you Ba(p)tiste?

[Jesse Glick]

On Thu, Mar 15, 2018 at 12:27 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> I
> assume that it "just works" and there's no quality concerns we should have from
> a non-standard placement of data on disk?

Any obvious problems would have been reported long ago. (I remember
some fixes involving folder renames or something like that.) By
forcing Essentials users to run with this mode, we would quickly flush
out remaining corner cases, I would think.

> From my understanding of the potential failure scenarios with a plugin or core
> upgrade is that unreadable build.xmls won't present a problem

Not that I am aware of.

> or be mutated by
> a new plugin update

It is possible for a plugin to mutate the `build.xml` of a completed
build, but unusual, and probably even less likely that such an update
would occur during Jenkins startup.

> mutable state should not be scattered around inside the container's temporary
> filesystem, but rather contained in the single mounted volume, e.g.:
>
> docker run -v /srv/jenkins:/var/jenkins_home jenkins/evergreen

So we could have a volume which contains `$JENKINS_HOME` as a
subdirectory, which would be the root of a Git repository; plus some
other subdirectories for other unversioned data. That makes more sense
to me than what was written in JEP-301.

[Baptiste Mathus]

@Jesse The alternative I had considered too was to just have multiple volumes. But I agree having only one is also more inlined with Essentials philosophy, having only one volume to actually backup (because yes, users should still backup, JENKINS-49406 is *not* a backup system) is simpler.

@James I agree with your comment on logs. 

• For Jenkins logs: unsolved yet, but we will have to dig into it very soon for JENKINS-49805, which is another critical part for the success of the project. I think if we go the way surfacing above, i.e. adding another level under $JENKINS_HOME to separate what is snapshotted, and what is not, we would kind of have a good place for those anyway. 
  
• For access logs: good point, though same, not enabled yet. I think we could also decide to use a reverse proxy for this purpose, since anyway we're likely to need to introduce that component so that users connected to Jenkins via their browsers see a breakage when Jenkins is restarting. 

Will adjust the proposal very soon with all this.

Thanks a lot for your feedback, I feel we're progressively reaching a more decent thing together, which is always an intent for reviews. So that's great.

Cheers

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3RUuod%3D5aqV2F_Hwu%3D1K%3DnVmM9a9VhvySVnccL-_mvpw%40mail.gmail.com.

[Jesse Glick]

On Fri, Mar 16, 2018 at 4:20 PM, Baptiste Mathus <m...@batmat.net> wrote:
> yes, users should still backup, JENKINS-49406 is *not* a backup system

True, though users will try to treat it as one.