On Wed, Nov 30, 2016 at 10:18 AM, Peter Hayes <
pete...@gmail.com> wrote:
> each time you run a job, you
> start with a fresh container without any previously cached dependencies (we
> use gradle generally). This increases the length of the build and adds
> network traffic to our Artifactory instance. I looked around for existing
> plugins but didn't find any so I have started a plugin[1] based on
> SimpleBuildWrapper that stores a configured set of files on the master at
> the end of the build and then on the next build downloads them to master in
> the original location.
This seems like a poor approach; rather than overloading Artifactory,
you will be overloading the Jenkins master. Archiving artifacts via
the Remoting channel can already wreck performance; you are talking
about potentially orders of magnitude more traffic than that.
There are two basic approaches to this kind of problem. One, which
assumes that the agents reuse workspaces between builds, is to set the
local repository/cache location to a workspace location. The
`docker-workflow` demo does this:
https://github.com/jenkinsci/docker-workflow-plugin/blob/46432bbe36af17dac93cfedcc93ffa51beba1343/demo/repo/flow.groovy#L20-L22
The other approach is to mount a volume containing the cache, letting
the Docker daemon handle the storage, which the
`parallel-test-executor` demo does:
https://github.com/jenkinsci/parallel-test-executor-plugin/blob/3961df3784045df1f6f285bc2b685ead4bc8593b/demo/Makefile#L3-L27
The volume-based approach is probably the more scalable, though there
are two points to beware: at least Maven’s `install:install` will dump
locally built artifacts into the repository alongside downloaded
releases (probably Gradle does something similar); and Maven’s Aether
repository manager is by default not thread-safe (Takari fixes this).
Maven 5 may allow the cache to be properly separated (again I am not
sure how Gradle fares here); in the meantime you may need to ensure
that there is a distinct volume for every potentially concurrent
build, for example keyed by `${JOB_NAME}/${EXECUTOR_NUMBER}`.
At any rate the exact solution chosen is going to depend on details of
how agents are provisioned and workspaces managed, so at root this
might simply be an RFE for CJP-PSE.