Anything wrong with storing expanded plugin data in var cache?

[Sam Gleske]

I normally store expanded plugin metadata within /var/cache/jenkins/plugins similar to how WAR filre metadata is stored in /var/cache/jenkins/war.

Is there any particular reason the Jenkins packages don't do this? Are there any drawbacks? I'm curious if others have any opinions on this.

I've been running Jenkins quite a while like this.

[1]: https://github.com/samrocketman/jenkins-bootstrap-shared/blob/1a409cad89007f56ed36342427b767dd4e88fbd9/packaging/docker/run.sh.in#L38

[Baptiste Mathus]

For context, I asked Sam to come and discuss this here after his comment on the JEP about Essentials Evergreen snapshotting system, in the part where we explain how we plan to aggressively segregate data.

So, my take is going to be two-fold:

• On Essentials, I think we are free to do more things because we have basically no existing setup. So I'm leaning towards doing this.
  I'm just slightly worried about:
  
• Bad performance (though heavily depending on the storage driver) writing things in a layer that could grow pretty big outside the $EVERGREEN_HOME volume. 
• At the same time, this would also be a good thing® since by definition this would be cleaned up if someone starts with a fresh image version (though it's not *really* the use case of Essentials, as the content will auto-update)
• we could also have a second volume dedicated to writing caches, but I'm not very keen on doing that since I feel this would go against the evergreen idea to be very simple to use. And users would probably have to manage the data written in that second volume instead of a single one (which I suppose people are more used to see).
  
  
• For Jenkins packages, I think this would be a good thing to do to, it's just that it's clearly more complex to do since there are existing setups out there. So we'll want to make sure we don't break people instances when upgrading after such a change. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/05beb1e5-a824-428c-b980-07a4e343acdc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen Connolly]

I see nothing wrong with it... but I would say that having added the --pluginroot option myself iirc ;-)

[Samuel Van Oort]

I think it's a genuinely great proposal and it sounds like most of the things I'd be concerned about have already been thoroughly considered and addressed effectively.  There are two remaining points to consider:

I think it might be desirable to give plugins some mechanism to interact with the Gitignore (or an equivalent) -- at least for cases where there's a non-standard storage mechanism that needs to be included in the SNAPSHOT (or excluded).  Extra flexibility will also help with addressing any edge-case issues that might crop up, and enables the project to push some of the pain onto plugin maintainers if they're (ab)using the Filesystem in non-idomatic (for Jenkins) ways that are hard to support.  I say that in the nicest way, hoping that I'm not a guilty party.

Also, I do worry a bit that we might encounter compatibility issues when separating the job config from their builds -- probably plugins doing something relative to the ITEM directory (which is wrong, but could still happen).  I know some of our senior Jenkins engineers don't see any issues with breaking plugins that are Doing It Wrong (tm), but generally I'd like to see some testing across different plugins and a healthy beta period with healthy in-the-wild-use just to ensure positive user experiences.  Relocating build records isn't done as often aside from expert installations, and less experienced users won't be able to distinguish between "Evergreen broke me" and "some random plugin broke because that wasn't respecting the build record directory."  Also this burn-in-testing period will help us identify and build out the rules for what to include/exclude in snapshots.

Bonuses: I like a lot that we're limiting the scope to just upgrade/downgrade and not trying to use this as a general-purpose backup.  It might even be desirable to limit the Git history in some fashion to prevent it becoming bloated.  

In general this is a well-considered and valuable addition to the Jenkins ecosystem.  Should generally help with performance -- especially if we end up splitting into a couple volumes down the road which can be relocated to faster/slower performance as needed.

On Monday, March 26, 2018 at 8:49:17 AM UTC-4, Stephen Connolly wrote:

I see nothing wrong with it... but I would say that having added the --pluginroot option myself iirc ;-)

On 24 March 2018 at 00:29, Sam Gleske <sam.m...@gmail.com> wrote:

I normally store expanded plugin metadata within /var/cache/jenkins/plugins similar to how WAR filre metadata is stored in /var/cache/jenkins/war.

Is there any particular reason the Jenkins packages don't do this?  Are there any drawbacks?  I'm curious if others have any opinions on this.

I've been running Jenkins quite a while like this.

[1]: https://github.com/samrocketman/jenkins-bootstrap-shared/blob/1a409cad89007f56ed36342427b767dd4e88fbd9/packaging/docker/run.sh.in#L38

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

[Damien Duportal]

Hello,

As a Jenkins User, I've run instances in production or punctually (workshops) with the "plugins" directory into a RAMDISK without issues.
When the persistence was required, I use to migrate this to a "machine local directory" (generally hypervisor local disk) to avoid having it stored on the NFS or the SAN used for Jenkins_home.

@Baptiste, I understand the concern about Evergreen. Obviously a trade-off will be in this specific case. Let's say you declare it as a separate volume in the initial Docker image, what would be the impact for the end user? My point here is to use the Docker "VOLUME" instruction to give a good default implementation, but without having to bother the end user: Docker ill take care of creating a dedicated data volume for this. It will ensure a fairly good default performance (if Docker's data volumes are slow, then the plugins directory will be the least of your concern :) ), and will work everywhere (Docker will take care of creating the volume at initial startup). Also, advanced Docker users would be able to tune it if they have specific needs. But it will still be a different VOLUME in the Image's metadata, expressing the different persistence properties. What do you think?

Le lundi 26 mars 2018 11:19:52 UTC+2, Baptiste Mathus a écrit :

For context, I asked Sam to come and discuss this here after his comment on the JEP about Essentials Evergreen snapshotting system, in the part where we explain how we plan to aggressively segregate data.

So, my take is going to be two-fold:

• On Essentials, I think we are free to do more things because we have basically no existing setup. So I'm leaning towards doing this.
  I'm just slightly worried about:
  
• Bad performance (though heavily depending on the storage driver) writing things in a layer that could grow pretty big outside the $EVERGREEN_HOME volume. 
• At the same time, this would also be a good thing® since by definition this would be cleaned up if someone starts with a fresh image version (though it's not *really* the use case of Essentials, as the content will auto-update)
• we could also have a second volume dedicated to writing caches, but I'm not very keen on doing that since I feel this would go against the evergreen idea to be very simple to use. And users would probably have to manage the data written in that second volume instead of a single one (which I suppose people are more used to see).
  
  
• For Jenkins packages, I think this would be a good thing to do to, it's just that it's clearly more complex to do since there are existing setups out there. So we'll want to make sure we don't break people instances when upgrading after such a change. 

2018-03-24 1:29 GMT+01:00 Sam Gleske <sam.m...@gmail.com>:

I normally store expanded plugin metadata within /var/cache/jenkins/plugins similar to how WAR filre metadata is stored in /var/cache/jenkins/war.

Is there any particular reason the Jenkins packages don't do this?  Are there any drawbacks?  I'm curious if others have any opinions on this.

I've been running Jenkins quite a while like this.

[1]: https://github.com/samrocketman/jenkins-bootstrap-shared/blob/1a409cad89007f56ed36342427b767dd4e88fbd9/packaging/docker/run.sh.in#L38

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.