Use Repository Sharing - advice please

[David Brown]

Here is my scenario. I have a number of Jenkins projects which build from the same set of Mercurial repositories, compile source code and generate executable files, etc.  I’m used to being able to run any combination of these jobs in parallel. Each build runs for at least 3+ hours, but the first third of the build time is spent cloning the repositories and preparing an individual workspace. I have restricted disc space so the jobs are setup to delete their workspace at completion.

 

I am looking to reduce my build times.

 

When I discovered the “Use Repository Sharing” option I was hoping it would enable me to shave the repository cloning time off the start of each build. However I have not been able to find much in the way of documentation or advice on the use of this feature, perhaps I’ve not been looking in the right places or else I’ve not looked hard enough.

 

Here’s what I’ve discovered so far, partly through trial and error:

• Need to also assign a specific Directory in the “Use custom workspace” option within Advanced Project Options (a per Project setting)
• I set the same shared workspace folder against two jobs
• When I attempted to run both jobs in parallel one job failed with message “The process cannot access the file because it is being used by another process.”, suggesting that I have to wait for the build of Project A to finish before I can start a build of Project B

 

Am I naive in expecting this feature to allow jobs to be run in parallel? If I can’t do this then I don’t think this feature will be of any benefit to me.

 

Any tips, guidance or advice would be gratefully received.

 

 

 

---

Confidentiality notice: This message may contain confidential information. It is intended only for the person to whom it is addressed. If you are not that person, you should not use this message. We request that you notify us by replying to this message, and then delete all copies including any contained in your reply. Thank you.

[Maciej Jaros]

David Brown (2015-07-28 22:51):

Here is my scenario. I have a number of Jenkins projects which build from the same set of Mercurial repositories, compile source code and generate executable files, etc.  I’m used to being able to run any combination of these jobs in parallel. Each build runs for at least 3+ hours, but the first third of the build time is spent cloning the repositories and preparing an individual workspace. I have restricted disc space so the jobs are setup to delete their workspace at completion.

 

I am looking to reduce my build times.

 

When I discovered the “Use Repository Sharing” option I was hoping it would enable me to shave the repository cloning time off the start of each build. However I have not been able to find much in the way of documentation or advice on the use of this feature, perhaps I’ve not been looking in the right places or else I’ve not looked hard enough.

 

Here’s what I’ve discovered so far, partly through trial and error:

• Need to also assign a specific Directory in the “Use custom workspace” option within Advanced Project Options (a per Project setting)
• I set the same shared workspace folder against two jobs
• When I attempted to run both jobs in parallel one job failed with message “The process cannot access the file because it is being used by another process.”, suggesting that I have to wait for the build of Project A to finish before I can start a build of Project B

 

Am I naive in expecting this feature to allow jobs to be run in parallel? If I can’t do this then I don’t think this feature will be of any benefit to me.

 

Any tips, guidance or advice would be gratefully received.

I think you can approach this problem differently.

1. Create a separate job (e.g. `load-repo`) which will only load current repository to some folder (this can be just workspace).
2. In the build/test jobs simply use the folder (you can use it directly or add symbolic link to it or copy if you must - depending on your build process).
3. Create a separate job that just runs above jobs. In this job you run `load-repo` as a blocking job and then run other jobs in parallel.

I don't think you can do it with just repository sharing as you would always have a problem when one job is already running (using repository) and other job wants to update repository.

Regards,
Nux.

[Ginga, Dick]

I will chip in with my thoughts… I am not surprised that your builds are getting that error. You can’t necessarily run parallel builds against the same source folders.

 

First, you build process must put built artifacts (both temporary and final) into some unique folder structures on a per build job basis. They can’t go into the source folder structure and this must be settable at build time.

Second, your OS/build process must allow multiple processes to “read” files because your different builds might be compiling the same file at the same time (again, to a different destination)

Third, the source folder must be re-usable. You say “cloning the repositories and preparing an individual workspace”. So the build processes can’t corrupt or change anything in that folder structure

 

Then, this could work. As Maciej says below, you would have a Load-Repo job followed parallel executions of build jobs. This can be orchestrated using Build Flow (which is being deprecated but I like it) or Work Flow. You would have to be sure that no builds were still running when you run the load-repo job.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/55B87867.6050008%40mol.com.pl.
For more options, visit https://groups.google.com/d/optout.

[David Brown]

Thank you for the replies.  It seems that "Use Repository Sharing" is not for me! I shall reconsider my approach to my problem and investigate your suggestions. Thank you again.