Sharing files between master and slave (dropbox like)

[Jerome Lacoste]

I am trying to centralize on my master the list of files required for my various slaves.

My current solution(s) including drawbacks are described below. Feel free to comment if you have a better alternative.

Because of security reasons, I want to keep a minimal number of connections between my slaves and my master. My slaves are in the cloud and I treat them as less trusted than my LAN master.

Ideally I would have liked some kind of dropbox like feature in jenkins. A folder I can drop files on my master or slave and have them automatically available on all machines (pull or push).

I don't have that so I've used a 2 way process:

* sharing files from Master to slave(s)

I've placed a Software/ catalog under the JENKINS_HOME/userContent/.

I pull the files when I need them.

DONE

* sharing files from slave to Master

Solution #1

Using the copy to slave plugin.

I created a simple free style job called infra_share_files that has a post build action that copies **/* from the slave's workspace to the master

On the server I also ran from JENKINS_HOME the following command:

  ln -s ~/jobs/infra_share_files/workspace/ userContent/slavesContent 

Usage:

1. I edit my job to run on the specific slave. 

2. [if first time for that particular slave] run the job

3. copy the relevant files on my slave under the job's workspace (I create a directory per slave)

4. run the job

5. after completion go to master file system and organize the files found on the master's workspace for the job into Jenkin's userContent directory

Drawbacks:

* things aren't fully automated

* this blocks my job queue... 

* the copy operation isn't very verbose

* the copy operation doesn't do a smart copying (like rsync would for example)

Alternative using a matrix job.

I created a similar matrix job called infra_share_files that has the same post build action as above. Select nodes per label and select all slaves but the master

On the server I also ran from JENKINS_HOME the following command:

ln -s ~/jobs/infra_share_files2/configurations/axis-label/ userContent/slavesContent

Changes from previous:

* advantage: no need to edit the job to choose which slave to run on

* drawback: given the copy to slave post build isn't smart, it takes a longer time (I have few slaves so it's OK)

* drawback the userContent/slavesContent/ navigation is uglier (not just the workspace is shared) (would be nice if on the server the matrix job was creating node specific links  under the workspace/ directory.

I can live with that as I don't do that everyday. Yet maybe there is a better solution ?

Possible "simple" implementations:

* a directory on the slave that automatically gets synced with the master on a particular location e.g. JENKINS_HOME/slaveContent/$slaveName/

* a file upload mechanism from Jenkin's web GUI ? (where I could upload files into userContent - or a perSlave directory)

[Les Mikesell]

On Thu, Sep 6, 2012 at 8:50 AM, Jerome Lacoste <jerome....@gmail.com> wrote:
> I am trying to centralize on my master the list of files required for my
> various slaves.
>

[...]

> Possible "simple" implementations:
> * a directory on the slave that automatically gets synced with the master on
> a particular location e.g. JENKINS_HOME/slaveContent/$slaveName/
> * a file upload mechanism from Jenkin's web GUI ? (where I could upload
> files into userContent - or a perSlave directory)

We just use a common nfs/samba read-only file share for the tools and
common libraries needed on the slaves. It's not particularly secure
against reading (just IP based restrictions), but files can only be
changed on the master which happens to be the host. If network speed
is an issue for runtime access, you could rsync copies locally in a
job that could be scheduled or manual. For our own library
components, we use svn externals to pull the right versions into the
right places which works the same whether it is a manual checkout or a
jenkins build.

There shouldn't be anything 'new' on a slave unless jenkins builds it,
in which case you can archive the build artifacts as part of the job
so they end up versioned back on the master.

--
Les Mikesell
lesmi...@gmail.com

[Adrian Nicolau]

Hi there, I also have a question regarding this topic, sharing files between the master and the slaves. Is there anyway to edit the userContent files via the provided web interface? My master is a remote machine and I don't want to connect to it every time I need to edit some file. My jobs require a certain amount of configuration files to be copied to slaves before the build step, so I would like to find an easy way to edit them.

Thanks!

[Jason Swager]

Have you considered using the Custom Tool plugin (https://wiki.jenkins-ci.org/display/JENKINS/Custom+Tools+Plugin)?  We also used to use the approach of copying files from /userContent to the slaves on a per-build basis. 

But Custom Tools made it rather easier.  First, the tool would only be copied to the slave once, and updated only if the tool source was updated.  If someone removed the tool from the slave, it would be automatically installed the next build.  The source of the tools is flexible - any URL can be the source - including Jenkins /userContent or anywhere Jenkins can reach.  Each tool requested by the job updates the PATH (convenient when the tool is an executable) and a separate ENV variable (convenient when you need to copy from those files).

[Les Mikesell]

Why don't you put everything you need for the builds in your version
control system so it will automatically be checked out on the build
slave? And you can update it the same way as the source.

--
Les Mikesell
lesmi...@gmail.com