Jenkins Operator (aka kubernetes-operator) Community Project's Governance: Request for Revision

[aben...@redhat.com]

Background

In mid-october 2019, Jenkins on OpenShift developers team from Red Hat started to engage in the jenkinsci/kubernetes-operator (aka Jenkins Operator) as contributors and committers. The initiators of the project are contributors from VirtusLab company.

 After a few commits related to documentation and minor bug fixes, the team, backed by product management, expressed their wish to engage more in the community and the project.

 To facilitate communication and collaboration, several rituals has been put in place by Red Hat team members:

• Subscribing to the slack channel on virtuslab's instance and providing best effort community support to the users who were asking.

• Scheduling a developer call each week to discuss developer related issues and ways to improve the software. Several additional stakeholders from Red Hat, who were strongly interested in extending Jenkins features were able to join (to name a few, J.F, R. Crisp, S. Bose, ....)

• A weekly community call to allow users to talk about their use of the plugin and features they would like to see. If not enough users were joining, which was often the case, this call was used to discuss architecture and development topics.

 After months, discrepancies arose, especially regarding architecture and design. At the same time, contributing features was harder both for technical reasons (monolithic operator) and for governance reasons (Red Hat developers didn't have commit permissions).

Despite these difficulties, Red Hat management insisted in remaining engaged in a community based operator and to continue the contributions, and also engage in larger refactoring tasks. Red Hat committers asked several times to have commit permissions. Initially, these requests have been rejected by invoking the will to keep control on the code quality. After several months of discussions, and difficult collaboration, Red Hat team members engagement was constant but commit permissions were still refused.

 Then, during a period of around 2 months and related to personal reasons, the main committer was not able to validate PRs, nor letting the other contributors be able to do it. The project was and is almost frozen.

 Red Hat is escalating the discussion to the Jenkins CI board to see how to solve this issue. As it was discussed as there is no existing process in place for transferring permissions when a component maintainer explicitly rejects that. Red Hat is asking to set up a governance that complies with the Jenkins Code of Conduct and allows Open Source friendly collaboration.

Governance revision proposal

The proposed governance has a main objective  to ensure permanently that a distribution of the commit rights across the active stakeholders of the project always exists. The definition of "active stakeholder" can be defined more accurately by the project board. But, as an initial proposal, an active stakeholder can be defined as:

"An entity contributing to the project in any form within the last 3 months would be by submitting code, committing code, proposing PRs and documentation or participating actively on a regular basis in the community calls".

We can add to this group any active user who can submit a minimum amount of documented and reproducible issues.

Initial working group leaders are defined using the following requirements:

• Initial leadership group consists of 3 or 5 members

• Founding parties: Virtus Lab, Red Hat, Jenkins Governance Board

• Each party nominates a representative

• Domination prevention: there should be no party having more than 50% of working group members.

 

The governance charter needs to define a few engagements and entities:

• Scope of rights and responsibilities explicitly held by the committee

• Committee structure that meets the requirements above

• Election process, including:

  • special elections in the case someone resigns or is impeached

  • who is eligible to nominate candidates and how

  • who is eligible to run as a candidate

  • Voter registration and requirements

  • election mechanics such as

    • committee company representation quotas

    • Limits on electioneering

    • Responses to election fraud

  • How are changes to the charter enacted, and by what process

  • How are meetings conducted

    • Recorded or not, and if not, how is the information shared

    • How is work tracked? Example steering project board

    • Is there a member note taker, or is there a neutral facilitator role that exists outside of the committee?

    • Frequency, duration, and required consistency

  • Committee decision-making process, and specifically those areas of action that require more/less consensus, e.g. modifications the charter

Contribution Process

• The active stakeholders of the project engage in getting members from the contributor/user community to form a working group to drive the direction of the project.

• Explains to potential contributors how/if they can add code to the repository/repositories

• Documents Workflow and management of pull requests

• Identifies who is authorized to commit or revert code

• Identifies automation is required for normal operations

• Defines how release decisions are made

  • Who is authorized to release and when.

  • Frequency limits

• Defines the documentation process

• Defines what CLA process is required and how it is programmatically enforced before code is merged.

Project Communication Channels

• The working group is added as an official entity of the Jenkins community. E.g. as a sub-project until the jenkins.io terminology is updated: http://jenkins.io/projects 

• The working group initially includes two roles: leader and member

• Jenkins Operator project meetings are considered to be the main communication and decision making channel

  • The decisions are made by a consensus among contributors

  • If a consensus cannot be reached, a voting between the working group leadership takes place. The decision is achieved by plain majority.

  • If a decision cannot be reached on the working group level, the Jenkins Project decision making process is used: https://www.jenkins.io/project/governance/#meeting

Permissions and access

TBD

Issues and questions

The following questions and issues are not answered yet and have to be answered before the establishing the governance chart:

• Can a contributor lose its status? Permanently or temporarily ?

• How can newcomers be promoted to leaders or contributors roles and with which frequency ?

• What happens if an organization gets disengaged from the community ?

• What if no quorum nor consensus or majority can be reached for some decisions ?

• What are the permissions and accesses given to a maintainer and what are the different roles maintainers can have. Eg: CI Cop, Release Cop etc.

[slide]

FYI, I made contact with some folks at VirtusLabs. They are hoping to respond to this discussion thread in the next little while. I will keep on top of it an communicate with to keep the ball rolling. 

[Paweł Dolega]

Hi all

Quick introduction - I am CTO of VirtusLab, original company behind Jenkins Kubernetes Operator.

------

First of all, apologies for prolonged time it took us to get back to this. 

To the point - it is definitely useful and desirable to have well understood governance rules (and thanks Akram for submitting this draft proposal 🙏 ). Details might need to be discussed but this is definitely good starting point.

To give you a little bit of background:

Out step back from the project (in fact not a step back, rather rethinking first principles) resulted due to our internal work related very much to Jenkins Kubernetes Operator.  We wanted to put some public work on hold until we have this sorted out. We were hoping for the work to be concluded in Oct, but (as is often the case with software development) we got some delays and currently aiming at having work concluded next week.

We definitely see a case for cooperation and making Jenkins Kubernetes Operator more popular e.g. in environments like Open Shift.  (which I believe is main point of interest for Red Hat). That said I think it is very much justified to say that the overwhelming majority of work done on this project (original author and overwhelmingly major developer efforts including majority of bug fixes ) comes from our organisation and we felt we have a little bit more right to steer the direction of the evolution of the operator project. We definitely see that doing work that would help Jenkins Operator be more usable e.g. in Open Shift is valuable, in the same time however this cannot be done by sacrificing the overall direction (we have more cases than only this one!). 

We do however understand that to have Jenkins Operator being more collaborative project we need to communicate our intention better to make sure everyone is on the same page. To that end we are going to work in following areas in upcoming weeks:

• public roadmap for the project
  
• contribution / governance model (which can be very much based on your suggestion)
  
• separation of specific area - e.g. Open Shift is much more important for Red Hat than us and we should be able to separate this area a little bit to make Red Hat contributor's work more streamlined.
  

As for the timeframes:

• we are working on these things right now
  
• we aim at being ready some time in December (given holidays etc - it would need to be 1st part of the month).
  

Once again - I really appreciate the governance proposal you suggested 🙏  I think it is very much in line with our thinking and I have zero doubts we would be able to come up with version that would feel satisfactory for all the sides (+future contributors). 

I hope this sounds reasonable (if not - don't hesitate to let me know, happy to discuss more details!)

Email correspondence is considered personal data processing. Check out our Privacy Policy for details about the controller of your data and your rights according to GDPR/RODO.

[Oleg Nenashev]

Hello,

Thanks a lot to Virtus Lab for the response! I hope we could revive this discussion, all interested parties are welcome to participate.

For the record, last week there was a hosting request from Red hat for starting the new operator: https://groups.google.com/g/jenkinsci-dev/c/oAWCXDbgcG4 . This request has been approved by the community, hence right now we may end up in a situation when there are 2 operators and a community split between them.

Best regards,

Oleg

[Paweł Dolega]

HI @Oleg 

Thanks for your quick reply!

So we have just announced Jenkins Operator Service for Azure here: https://virtuslab.com/jenkins-operator-service-on-azure/

This is just the beginning of the road for Jenkins Operator Service for major cloud providers and platforms. Jenkins Kubernetes Operator is be a commercial offering, backed by freely available OSS version.

Now we have a week or two of breathing space and we are getting back to planning future interactions and community engagement. 

Here are key points:

0. Jenkins Kubernetes Operator is a base for business initiative for VirtusLab, therefore we are committed to building proper full time team working on it.

1. OSS version is key part of our strategy and we don't intend to change that - our business model is based on managed marketplace offerings and support model (value added comparing to OSS that you can grab and install yourself).

2. We do agree that current Jenkins Kubernetes Operator is complicated and not properly modulerized. It worked for our initial version, however for our future plans this cannot stay. Redesigning / decoupling architecture is a critical step forward.

3. Given 2, we are now focusing on preparing proper roadmap, governance model and planning.

Indeed point 2 will cause weird situation where there is simple-jenkins-operator targeting mostly OpenShift (not sure if this is the intention of @Akram's team, but my impression was that previous contributions were mostly targeted in this area) and jenkins-kubernetes-operator targeting wider audience (with eventually simplified architecture).

I think it would be best discuss further cooperation going forward and we are happy to do that. 

[Oleg Nenashev]

Hi Pawel,

I am looking forward to see the maintenance and development to be revived in the repository. One thing that requires your immediate attention is the Jenkins trademark usage in your product name. The name "Jenkins" is a registered trademark in the USA (#4664929, held by Software in the Public Interest, Inc.) in order to protect the project and users from confusing use of the term. To reduce the process overhead and uphold our open-source spirit, we adopt the Linux kernel policy on trademark usage. You need to apply for a sublicense if you are using the term "Jenkins" as part of your own trademark or brand identifier for Jenkins-based software goods or services. You can find more information about the trademark policy here: https://www.jenkins.io/project/trademark/ 

"Jenkins Operator Service on Azure" does not fit the pre-approved trademark usage patterns defined in the Linux Foundation guidelines, and hence your company needs to go through the trademark approval process to be able to use "Jenkins" in such product name.

Best regards,

Oleg Nenashev

[Paweł Dolega]

Going to take a look at this right away. Apologies for the misuse of the name, we did analyse the trademarks but for some reason apparently completely missed that.

On it now, thank you 🙏

[Bartłomiej Antoniak]

Hi,

First, let me introduce myself. I’m Cloud Engineering Manager at VirtusLab, I was involved in development and design of Jenkins Operator in its early days. Currently, I’m responsible for the cloud-native landscape within the company - which includes our involvement with Jenkins Operator. I’d love to be more active as part of this initiative and community engagement in general.

In terms of the future of Jenkins Operator and involvement from our side, in addition what Pawel was writing about earlier, I’m getting back to you with the following documents:

• Governance Model proposal - still early version, expect next update by end of 2020
• Contribution Model - will issue a new pull request with this soon
• Jenkins Operator Roadmap and Vision

(feel free to add some comments to these documents)

It would be great if we could discuss these and introduce the team early next year.

For us, this is now a strategic project, with dedicated team and long-term commitment. We are happy to partner with you and invest more of our engineering effort to make sure the project brings value in the both Kubernetes and Jenkins ecosystem, as well as to actively take part in community engagement aspects.

I believe that further cooperation with RedHat and Jenkins Board will result in solid support for Jenkins in cloud-native ecosystem. 

Apart from that, I’d like to wish you good time with your families during the Christmas season.

Regards,

Bartek

[Oleg Nenashev]

Hi Bartek,

Thanks for assembling the proposal! The proposal is definitely a good start. I'd guess the key topic is about the "Founder leader" role: "Founder leader (VirtusLab): establishes project vision, controls all permissions to merge code into it, and assumes the right to speak for it in public". It is more or less aligned with the community practices for plugin maintenance, plugin maintainers have ultimate powers in their repositories until they decide to step down. This practice has its own pros and cons though. Would be nice to get more feedback from others.

I also suggest cross-posting it in the Jenkins Operator community channels and GitHub Issues so that we get some feedback flowing in.

Contribution model is definitely +1. https://github.com/jenkinsci/jep may be considered for proposals involving multiple repositories.

Best regards,

Oleg Nenashev

[Bartłomiej Antoniak]

Hi Oleg & Jenkins board,

It terms of the "Founder leader" we'd like to ensure the initial re-architecting phase goes smoothly and "representatives" actively contribute according to quality standards. Long-term we can transfer ownership of individual components to other founding leaders using model similar to Kubernetes SIGs.

The https://github.com/jenkinsci/jep makes a lot of sense, especially to align Kubernetes Operator with Jenkins development.

I'm happy to set up a dedicated meeting to walk through this document and address all concerns if needed. Also, feel free to collaborate offline and add comments directly in the documents for further discussion.

Regards,

Bartek

[Oleg Nenashev]

Hi all,

We plan to have a discussion about the Jenkins Operator and community alignment at the next Cloud Native SIG meeting on Feb 05.

Please feel free to join if you are interested: https://docs.google.com/document/d/13zeaKgtud5jZ5RqZEh1lrwjDXJRm7j31scPymlrMpfo/edit#heading=h.wydyxiol9ok9

Best regards,

Oleg