It sounds nice (I have not tried it). Some technical questions / comments:
On Wed, Jan 23, 2019 at 9:16 AM Tomasz Sęk <
tomasz...@gmail.com> wrote:
> every single plugin and it's depended plugins have to be installed with specific version
Good, I have long fought against plugin management scripts/tools which
try to be helpful by picking the latest plugins from the update
center, or adding dependencies automatically! A complete and explicit
list of plugins is safer, but you then run the risk of leaving lots of
unused plugins behind after configuration changes. Thoughts:
https://issues.jenkins-ci.org/browse/JENKINS-53506?focusedCommentId=348904&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-348904
(And it would be great if we could get your system synchronized with
Evergreen, at least optionally.)
> use Kubernetes plugin to build jobs (every new job spins up new pod in Kubernetes
With master executor count set to zero, I hope.
> make backup of jobs history before Jenkins master pod deletion and restore it in new pod
You do not just set a custom build record directory (IIRC this is a
system property) and use a persistent volume claim? See Evergreen for
example. (Beware however that there is some more true state besides
build history; it is safest to persist the entire `$JENKINS_HOME`, and
let JCasC + `job-dsl` overwrite old configurations.)
> improve JNLP agent to handle Jenkins master pod restarts
Not sure where you are going with this. An agent cannot connect to two
different master sessions; it is just not possible without completely
rewriting Remoting. Pipeline builds do however attempt to continue
running, even in the middle of a `sh` step, across master restarts so
long as the actual machine / VM / container running a “JNLP” agent
survives: the agent will try to reconnect to the same master URL at
intervals and start a new Remoting channel, at which point the
Pipeline logic detects a connected agent with the same name as the
build was using before and the same filesystem layout and proceeds.
https://github.com/jenkinsci/kubernetes-plugin/blob/ef177b3b1297a928b22292644afabb1556b5da68/src/test/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/RestartPipelineTest.java#L179-L195
Similarly, a build using an agent launched from the master can be
resumed so long as the new master session can reconnect to the same
container somehow.
> In current solution only one Jenkins master can be run at the same time, the old goes down then the new one can be created.
> I thought we can spin up new Jenkins master pod when the old one have to be killed and then switch traffic to the new one.
This is a very hard problem in Jenkins generally, since many features
in core & plugins assume that the Jenkins service is stateful, the
`$JENKINS_HOME` directory is that state, and only one process accesses
it. While workarounds are certainly possible if you accept
restrictions on the features in use, providing a satisfactory and
general solution is one of the goals of the SIG:
https://jenkins.io/sigs/cloud-native/#cloud-native-jenkins