Multiple master with shared slave pool
100 views
Skip to first unread message
Jonathan K
May 30, 2011, 6:24:41 PM5/30/11
to Jenkins Users
My team is currently in the process of designing a fairly large CI
solution with Jenkins. We are providing masters for 5 separate teams
but would like to utilise a common pool of slave machines.
With the base Jenkins, we know that we can run multiple slaves on a
single "slave machine" but there is no shared knowledge between the
masters about each others utilisation, so if we configure the 5
masters with a slave that has 4 executors then there could be 20 jobs
running (too many) but a maximum of 4 per master (too few).
Is anyone able to suggest a better/different solution so that we can
fully utilise our hardware?
Jonathan
solution with Jenkins. We are providing masters for 5 separate teams
but would like to utilise a common pool of slave machines.
With the base Jenkins, we know that we can run multiple slaves on a
single "slave machine" but there is no shared knowledge between the
masters about each others utilisation, so if we configure the 5
masters with a slave that has 4 executors then there could be 20 jobs
running (too many) but a maximum of 4 per master (too few).
Is anyone able to suggest a better/different solution so that we can
fully utilise our hardware?
Jonathan
David Nellans
May 30, 2011, 6:34:49 PM5/30/11
to jenkins...@googlegroups.com, Jenkins Users
Why do they have to be on separate masters? Jenkins supports access control to sandbox groups/jobs from each other on a single master. Global resource management is always the best option when possible.
Confidentiality Notice: This e-mail message, its contents and any attachments to it are confidential to the intended recipient, and may contain information that is privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the original e-mail message and any attachments (and any copies that may have been made) from your system or otherwise. Any unauthorized use, copying, disclosure or distribution of this information is strictly prohibited.
Jonathan K
May 30, 2011, 8:06:15 PM5/30/11
to Jenkins Users
Can you elaborate more on the sandbox groups/jobs? We have a high
turnover of developers who pretty much do whatever they want so there
is no opportunity to categorise jobs or implement security on a per
job basis.
The separation is for a few of reasons:
* Political - The teams are not required to follow the same processes,
etc.
* Political - We don't want the decisions of one team (e.g. which
plugins to use) to affect the other teams
* Technical - We're concerned about configuring 1000 or more jobs on
the same master
* Technical - With jobs created free-form the categorisation of jobs
is a concern for us
Our thinking is that by having multiple masters we can absorb some of
the shortcomings of the business.
Jonathan
On May 31, 8:34 am, David Nellans <DNell...@fusionio.com> wrote:
> Why do they have to be on separate masters? Jenkins supports access control to sandbox groups/jobs from each other on a single master. Global resource management is always the best option when possible.
>
turnover of developers who pretty much do whatever they want so there
is no opportunity to categorise jobs or implement security on a per
job basis.
The separation is for a few of reasons:
* Political - The teams are not required to follow the same processes,
etc.
* Political - We don't want the decisions of one team (e.g. which
plugins to use) to affect the other teams
* Technical - We're concerned about configuring 1000 or more jobs on
the same master
* Technical - With jobs created free-form the categorisation of jobs
is a concern for us
Our thinking is that by having multiple masters we can absorb some of
the shortcomings of the business.
Jonathan
On May 31, 8:34 am, David Nellans <DNell...@fusionio.com> wrote:
> Why do they have to be on separate masters? Jenkins supports access control to sandbox groups/jobs from each other on a single master. Global resource management is always the best option when possible.
>
David Nellans
May 31, 2011, 9:35:40 AM5/31/11
to jenkins...@googlegroups.com
If you can't implement access control on a job basis nor even categorize
jobs then you're pretty much SoL for access limitations.
jobs then you're pretty much SoL for access limitations.
Plugins can be installed without having to be used globally - they're
only in use if people choose to use them generally.
Les Mikesell
May 31, 2011, 11:32:44 AM5/31/11
to jenkins...@googlegroups.com
On 5/30/2011 7:06 PM, Jonathan K wrote:
> Can you elaborate more on the sandbox groups/jobs? We have a high
> turnover of developers who pretty much do whatever they want so there
> is no opportunity to categorise jobs or implement security on a per
> job basis.
>
> The separation is for a few of reasons:
> * Political - The teams are not required to follow the same processes,
> etc.
> * Political - We don't want the decisions of one team (e.g. which
> plugins to use) to affect the other teams
> Can you elaborate more on the sandbox groups/jobs? We have a high
> turnover of developers who pretty much do whatever they want so there
> is no opportunity to categorise jobs or implement security on a per
> job basis.
>
> The separation is for a few of reasons:
> * Political - The teams are not required to follow the same processes,
> etc.
> * Political - We don't want the decisions of one team (e.g. which
> plugins to use) to affect the other teams
If the groups want to maintain political independence, they should have
their own budget constraints. Why not make each group that doesn't want
to cooperate with the others supply their own build slaves?
> * Technical - We're concerned about configuring 1000 or more jobs on
> the same master
> * Technical - With jobs created free-form the categorisation of jobs
> is a concern for us
>
> Our thinking is that by having multiple masters we can absorb some of
> the shortcomings of the business.
You could compromise by using virtual machines for the slaves, either
sharing host resources or not. That gives you the technical separation
in terms of isolating the configurations and a way to share the physical
resources (or not, as you prefer), but it still won't automatically
optimize the use of your resources at any particular time. Someone
might still need to manually balance the number of active VM's to match
the different group's activities - or each group might fire up a few of
their own if the shared resources weren't sufficient.
--
Les Mikesell
lesmi...@gmail.com
Kamal Ahmed
May 31, 2011, 11:42:25 AM5/31/11
to jenkins...@googlegroups.com
Jenkins version, 1.413
after installing plugins , during a fresh install on ubuntu 10.10, i get:
Status Code: 500
Exception:
Stacktrace:
Generated by Winstone Servlet Engine v0.9.10 at Tue May 31 11:33:00 EDT 2011
after installing plugins , during a fresh install on ubuntu 10.10, i get:
Status Code: 500
Exception:
Stacktrace:
org.apache.commons.jelly.JellyTagException: jar:file:/var/run/jenkins/war/WEB-INF/lib/jenkins-core-1.413.jar!/lib/layout/layout.jelly:93:54: java.util.ConcurrentModificationException
at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:709)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:282)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:63)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
at org.kohsuke.stapler.jelly.JellyRequestDispatcher.forward(JellyRequestDispatcher.java:55)
at hudson.util.HudsonIsLoading.doDynamic(HudsonIsLoading.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:282)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:149)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:88)
at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:352)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:477)
at org.kohsuke.stapler.Stapler.service(Stapler.java:159)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
at winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:94)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:86)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:162)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:782)
at java.util.ArrayList$Itr.next(ArrayList.java:754)
at hudson.PluginManager$UberClassLoader.findResource(PluginManager.java:652)
at java.lang.ClassLoader.getResource(ClassLoader.java:977)
at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:1159)
at org.kohsuke.stapler.framework.adjunct.Adjunct.parseOne(Adjunct.java:149)
at org.kohsuke.stapler.framework.adjunct.Adjunct.(Adjunct.java:110)
at org.kohsuke.stapler.framework.adjunct.AdjunctManager.get(AdjunctManager.java:103)
at org.kohsuke.stapler.framework.adjunct.AdjunctsInPage.findNeeded(AdjunctsInPage.java:134)
at org.kohsuke.stapler.framework.adjunct.AdjunctsInPage.assumeIncluded(AdjunctsInPage.java:104)
at org.kohsuke.stapler.jelly.AdjunctTag.doTag(AdjunctTag.java:81)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
... 47 more
Generated by Winstone Servlet Engine v0.9.10 at Tue May 31 11:33:00 EDT 2011
Jonathan K
May 31, 2011, 11:18:54 PM5/31/11
to Jenkins Users
What stops us from implementing anything on a per job basis, is that
we have 100+ developers who are responsible for adding their own jobs,
so I don't see a way for us to implement anything on a per job basis,
the job is whatever each person decides to do. Even with documentation
or job templates, because each user can freely do their own thing
means that they will.
I can think of having a cron job which runs over jobs and looks for
potential problems, can you think of anything else? Making a job name
match a regex could make our life easier, then jobs could be
automatically categorised.
On May 31, 11:35 pm, David Nellans <dnell...@fusionio.com> wrote:
> If you can't implement access control on a job basis nor even categorize
> jobs then you're pretty much SoL for access limitations.
>
> Plugins can be installed without having to be used globally - they're
> only in use if people choose to use them generally.
>
>
>
>
>
>
>
> On Mon, 2011-05-30 at 18:06 -0600,JonathanK wrote:
> > Can you elaborate more on the sandbox groups/jobs? We have a high
> > turnover of developers who pretty much do whatever they want so there
> > is no opportunity to categorise jobs or implement security on a per
> > job basis.
>
> > The separation is for a few of reasons:
> > * Political - The teams are not required to follow the same processes,
> > etc.
> > * Political - We don't want the decisions of one team (e.g. which
> > plugins to use) to affect the other teams
> > * Technical - We're concerned about configuring 1000 or more jobs on
> > the same master
> > * Technical - With jobs created free-form the categorisation of jobs
> > is a concern for us
>
> > Our thinking is that by having multiple masters we can absorb some of
> > the shortcomings of the business.
>
> >Jonathan
>
> > On May 31, 8:34 am, David Nellans <DNell...@fusionio.com> wrote:
> > > Why do they have to be on separate masters? Jenkins supports access control to sandbox groups/jobs from each other on a single master. Global resource management is always the best option when possible.
>
we have 100+ developers who are responsible for adding their own jobs,
so I don't see a way for us to implement anything on a per job basis,
the job is whatever each person decides to do. Even with documentation
or job templates, because each user can freely do their own thing
means that they will.
I can think of having a cron job which runs over jobs and looks for
potential problems, can you think of anything else? Making a job name
match a regex could make our life easier, then jobs could be
automatically categorised.
On May 31, 11:35 pm, David Nellans <dnell...@fusionio.com> wrote:
> If you can't implement access control on a job basis nor even categorize
> jobs then you're pretty much SoL for access limitations.
>
> Plugins can be installed without having to be used globally - they're
> only in use if people choose to use them generally.
>
>
>
>
>
>
>
> On Mon, 2011-05-30 at 18:06 -0600,JonathanK wrote:
> > Can you elaborate more on the sandbox groups/jobs? We have a high
> > turnover of developers who pretty much do whatever they want so there
> > is no opportunity to categorise jobs or implement security on a per
> > job basis.
>
> > The separation is for a few of reasons:
> > * Political - The teams are not required to follow the same processes,
> > etc.
> > * Political - We don't want the decisions of one team (e.g. which
> > plugins to use) to affect the other teams
> > * Technical - We're concerned about configuring 1000 or more jobs on
> > the same master
> > * Technical - With jobs created free-form the categorisation of jobs
> > is a concern for us
>
> > Our thinking is that by having multiple masters we can absorb some of
> > the shortcomings of the business.
>
> >Jonathan
>
> > On May 31, 8:34 am, David Nellans <DNell...@fusionio.com> wrote:
> > > Why do they have to be on separate masters? Jenkins supports access control to sandbox groups/jobs from each other on a single master. Global resource management is always the best option when possible.
>
Reply all
Reply to author
Forward
0 new messages