True High Availability/load balancing for Jenkins. Its possible!!

[Surya Gaddipati]

You need the following ingredients 

 1. DbBacked builds ( https://github.com/groupon/DotCi ) , i.e no in memory build data.  

 2. NFS mount for 'jobs' folder to share console output, artifacts ect. 

 3. Message Passing between various masters ( eg: build queuing, build abort, queue abort ect ) . https://github.com/suryagaddipati/HaJenkins

 4. No shared agents. You could use any cloud plugin , although I like instant provisioning ones eg: https://github.com/suryagaddipati/jenkins-docker-swarm-plugin

Setup: 

 1. Install and configure plugins mentioned above

 2. Start multiple Jenkins instances with share mounted/symlinked  'jobs' folder . 

 3. Done. 

Method: 

 Build Create: When a build comes into the queue, it gets saved into redis queue ( I had to hack override Queue Implementation to achieve this).  Build gets picked up any of the masters that are watching the queue. 

 Build Abort:  Aborting a 'ha build' puts a message into redis queue , which gets processed by all masters and the master running the actual build aborts it. 

 Queued item Abort: Same mechanism as above ^. 

 Build Execution: Build execution happens in a dynamically created agent.  Build info is written to db via DotCi, and build logs are synced to all masters via NFS. 

 Build Delete: Build gets deleted in DotCi database. 

I've done some testing and this setup seems to work fine and is totally transparent to end user.  But I haven't done any extensive testing for edge cases.

I am curious to know what experts here think about this. What are the things that could go wrong with a setup like this? 

Surya

[Michael Neale]

Very nice Surya. 

I am guessing by looking that most of the magic in: https://github.com/suryagaddipati/HaJenkins

I know it is a common pattern to use redis as a queuing mechanism (but some point it wasn't really intended) - do you run redis itself in some HA configuration? 

[Surya Gaddipati]

HI Michael, 

I haven't actually tried putting it to production in any capacity. I think we have HA redis that I can use if this turns out to be a viable option. 

Its impossible to do anything 'production grade' with current Jenkins model of single point of failure.  This I think is the biggest downfall of Jenkins that's stopping it from turning into a serious CD tool.  


--Surya