ci.jenkins.io configuration as code

[Damien Duportal]

[Cross-posted on jenkins-infra and jenkins-dev]

Hello dear users of ci.jenkins.io ,

As part of the public roadmap (https://www.jenkins.io/project/roadmap/, filter on "Infrastructure and Services"), one of the task that we (the infrastructure team) is starting to plan and work on, is the migration of the configuration of ci.jenkins.io to "Configuration as Code". 

Using a full configuration as code would benefits everyone:

• Contributors (core, plugins, any project built in this instance) so they could tests plugins in early versions and have a better service (improved MTTR,better  performances, less cost)
• Infra team, as every change could be traced, versioned, followed, approved and done through a GitHub repository
• Community, as it would be a "drink your own beverage" of Jenkins CasC on a public and real life instance
• Security people, as it would allow easier traceability, credential rotation
• The list can go on…

Such a change comes with risk (as all changes) that could threaten the stability and the security of this instance: we need to plan.

Therefore I’m opening the discussion here (as a started) around the following proposal:

• Writing a JEP to describe this subject, and get to a transparent and open process, in a shared and written manner:
• Goals
• Risk Analysis
• Planning
• Operations
• Updating the roadmap to point to this JEP for the element "ci.jenkins.io" 

For technical mapping, ci.jenkins.io is running as a Docker container, inside a virtual machine managed by the Puppet system (see https://github.com/jenkins-infra/jenkins-infra).

It’s data is stored on a persistent volume, and there are groovy scripts ensuring that the default security and minimalistic configuration is applied.

Of course, this change is not gonna be applied on 1 shot: the challenge is to find the correct areas where it would bring enough values.

Food for though, there are different areas to be treated:

• Managing Agents / Clouds
• Managing Jobs (JobDSL?)
• Managing plugins, core version
• Managing Security (AutN. , AuthZ, API)
• Managing Credentials (both Security and Job management)

My initial proposal (that have to be materialized in a document, ideally the JEP mentioned earlier) TL;DR is to start by only focusing on the agent management for the first iteration:

• No existing legacy tooling: This configuration area is not covered anywhere as for today, everything is done manually through the UI (meeh)
• Low risk: Rollback is easy and fast in case of any issue
• Definition of the area is easy: JCasC already provides us an initial export of the existing configuration
• High value: Cost management, continuous update, new architectures/platform support easily

While we discuss (or not) this subject and the plan, I’m starting to work on a subject that is needed whatever the direction (even NOT switching to ci.jenkins.io): 

allowing any contributor to reproduce a ci.jenkins.io instance locally, at least a "almost close to production) by reusing the existing Vagrant + Puppet setup and focusing on simulating a local LDAP as close as possible as the real one.

If you have any insights, tip, recommendation, advise or any red flag, please let me know by answering this email :)

Cheers,

Damien DUPORTAL