Jenkins + Kubernetes - Cache .m2 for jobs

[Enguerrand A]

Hello everyone,

I use the Kubernetes plugin for Jenkins to launch all Jenkins jobs in a Kubernetes cluster. For better performance, I want to use a persistent disk for the cache of different job Maven with insulation of it, ie a disk space per job (to separate the cache)

Do you have ideas for automatic solutions with the Kubernetes plugin to have cache on the .m2 (or another type of cache) with a different disk by jobs ?

Thank you in advance !

[Carlos Sanchez]

you can mount a PV or PVC but depending on what backend you use that will prevent concurrent executions

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Enguerrand A]

The problem is that I would like to automate the installation of the PV or PVC without needing to specify it in the Jenkinsfile but at least I can automate the creation of PVC or PV for each job to avoid manual operation.

I saw that automatically a PVC or PV could be mounted for the workspace Jenkins, but it raises concerns if 2 build on the same job runs on the same shared workspace (build can inflict another by removing files for example)

[Tristan FAURE]

Hello !

we have the same issue

what we would like is the same behavior as : https://docs.gitlab.com/ee/ci/caching/

I have seen this example : https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/maven-with-cache.groovy

but if several jobs run in parallel I don't want them to modify the same volumes and I would like to not create volumes before the jobs

I have seen the dynamic claim options for workspace but data are common for a run and not a job does it ?

our idea currently is creating a volume claim with a name ${jobid} or {folderId} and set a K8S cron job that delete frequently but we don't find a way to set this by default (we tried in raw yaml option but it seems it only support kind : Pod)

Regards

[Liejun Tao]

This is my solution for a little different purpose: use a .m2 cache for common libs.

https://github.com/liejuntao001/jenkins-k8sagent-lib/blob/a6ced3975e37fe559c0ae8a918ed01b6920f1efa/resources/podtemplates/base.yaml#L24

The idea is to make a docker image with the most used artifacts(external dependencies) in .m2 folder.

At the launch of the Agent, copy them to workspace's .m2 folder from the image.

This should be adapted to fit your needs.

I assume you are not going to run concurrent builds for a job to avoid concurrent write, if you want a job has a dedicated cache.

At the end of a build, create a docker image based on the existing .m2 content and push back for next time use.

If you are going to run concurrently builds, this idea of a sharing cache is invalid, I think.

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,

Baibai

[Tristan FAURE]

Hello

Thank you for the hint ! I think it could help some projects !

but it will make some docker images very heavy and I would be interested to have jenkins team opinion about getting the equivalent of https://docs.gitlab.com/ee/ci/caching/

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/X61BK83LHLU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAMRm5jbRhhtF5S4bzzT%3Du7La4sggNW2c-kEAjVMaqBeBK6cHPg%40mail.gmail.com.

[Tristan FAURE]

seems like there is an issue about it:  https://issues.jenkins-ci.org/browse/JENKINS-42422

[James Nord]

you can massively speed up the build (even without caching) by having a maven repository proxy/cache nexus etc) in the same cluster (or at least on a high bandwidth low latency link). even when you have things local to the container the remote maven repo will still be contacted and depending on your setup can still slow the build.