Managing Build dependencies

[Jeff Payne]

Hello all,

I'm struggling a bit with the concept of managing build dependencies in Jenkins - perhaps someone could suggest some best practices to get me headed down the right track.

For starters, I understand that Jenkins is not a replacement for make.  However, Jenkins does allow for the incremental development / build of libraries and other resources that higher level projects may depend upon.  I've found that you can trigger dependent builds in two ways, but both have issues:

1) Top Down: you can trigger the build of a dependency as a build step, which in turn could trigger other builds.  The biggest problem I have with this is that with any level of complexity and a fairly small number of executors, it is pretty trivial to deadlock your build - Top level builds waiting for lower level builds to complete, which cannot complete because there are no free executors.  The other issue with this is that dependencies build regardless of whether or not they are up to date, and can end up being built multiple times for a large project.  (Our check-outs are incremental, but the objects have to be built clean because we don't trust C-language header dependency management)

2) Bottom Up: This way we can poll the SCM to trigger builds, and trigger the build of dependent modules as the libraries change.  The problem here is that you can end up with a maintenance headache where every low level library has to know all the projects that depend on it.  Drop one high level project, and you have to go through and change the build of each dependency.

Is there any way to specify the dependencies from the top-down, yet trigger builds from the bottom-up?  Of course I could integrate the whole thing back to one make build, but that doesn't really take advantage of Jenkins' strengths.

Thanks,
Jeff

[Didier Durand]

Hi,

Do you use Subversion as SCM ?

if yes, svn:externals are a very convenient way of specifying
dependencies among projects and have jenkins schedule builds according
to those dependencies

In the /lib of each project, you don't copy "physically" the jars of
yours that it depends on but you make an svn:externals link to them
where they are stored in svn.

Then, each time the low-level lib is changed, Jenkins detects it via
the externals, re-import the new version in its workspace and starts
buidling.

By doing, so you don't have duplicate definitions in Jenkins versus
SVN: everything is in 1 single place: the externals of svn

regards

didier

[Jeff Payne]

Thanks for the reply,

An interesting approach - I am using SVN, so this is an option, although it sort of defeats the purpose of saving / copying artifacts in Jenkins.  We don't usually archive our libraries (.a, since we are developing in C) but I'll give it some thought.

Thanks again,
Jeff

[Leo Przybylski]

C is pretty platform dependent, so just because you store libs in scm doesn't mean you are hitting all your targets.

I would probably recommend using ivy. Ivy will give you the ability to use transitive dependency management over your c libs. You can contextualize by platform. This will give you flexibility to build cleanly for different platforms or even just different versions of libraries or whatever.

You can keep the libs out of your vcs.

You don't even need to build with it. You can just use it to handle your dependencies, but i recommend not.

Leo

[Kamal Ahmed]

on the UI, i get:

Status Code: null

Exception:
Stacktrace:

(none)

---

Generated by Winstone Servlet Engine v0.9.10 at Tue Dec 13 11:02:59 EST 2011

and on jenkins log:

[Winstone 2011/12/13 11:02:59] - Untrapped Error in Servlet
javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-1.443.jar!/jenkins/model/Jenkins/configure.jelly:58:84: <st:include> No page found 'config.jelly' for class org.jenkinsci.plugins.slave_setup.SetupConfig
    at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:100)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:563)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:477)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:159)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
    at winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:157)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:131)
    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:83)
    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:61)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
    at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
    at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
    at java.lang.Thread.run(Thread.java:619)

[Jesse Farinacci]

Greetings,

On Tue, Dec 13, 2011 at 11:08 AM, Kamal Ahmed <kamal22...@yahoo.com> wrote:
> [Winstone 2011/12/13 11:02:59] - Untrapped Error in Servlet
> javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException:
> jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-1.443.jar!/jenkins/model/Jenkins/configure.jelly:58:84:
> <st:include> No page found 'config.jelly' for class
> org.jenkinsci.plugins.slave_setup.SetupConfig

Disable the slave setup plugin, restart. Then look for an existing
JIRA that captures this problem, or open a new one.

-Jesse

--
There are 10 types of people in this world, those
that can read binary and those that can not.

[Kamal Ahmed]

Thanks Jesse,

Disabeling the Slave Setup Plugin, resolved the issue

---

From: Jesse Farinacci <jie...@gmail.com>
To: jenkins...@googlegroups.com
Sent: Tuesday, December 13, 2011 11:13 AM
Subject: Re: Cannot access "Manage Jenkins" 1.443