Creating a new Gitlab SCM Plugin with Multibranch Pipeline support

[Parichay Barpanda]

Hi all,

I am preparing a proposal to add Multibranch Pipeline support to the Gitlab plugin. Existing Gitlab plugin does not support Multibranch pipeline builds in a way that it enables build triggers but cannot configure the variables (basically multibranch pipeline support is non-existent) - the API doesn't support it. But there are a lot of existing users that use the GitLab plugin at the moment and I fear API changes might break binary compatibility. 

My suggestions is to develop 2 new plugins: a Gitlab API plugin and a Gitlab SCM Plugin.

1) Gitlab API plugin which, very similar to Github API plugin, wraps the Gitlab Java API.

2) Gitlab SCM plugin which will be a major design overhaul version of existing Gitlab Plugin to accomodate both pipeline and mulitbranch pipeline jobs along with other type of job configurations.

I have 2 ways to implement this:

Method 1:
1) I am thinking freestyle jobs will be deprecated in the future in favor of pipeline jobs. Gitlab plugin supports freestyle builds, so as long as freestyle builds are favoured the existing Gitlab plugin will support it. 

2) Focusing on just pipeline will ease the task of designing API and handling the complexity due to which all the SCM plugins are divided into two i.e. <scm> plugin and <scm>-branch-source plugin.

Method 2:
1) If freestyle jobs are important and cannot be compromised then modify the Gitlab plugin to add multibranch pipeline support and find a way to take out Gitlab API and wrap it in a separate plugin. I haven't been able to figure out how much security risks and backwards compatibility will be involved in this method. Need someone with experience tell me about this.

Main Objective of this proposal: Just have one SCM plugin which does all type of jobs and remove users' confusion of having 2 separate SCM plugins and code duplication.

Need your feedbacks so that I can finalise which method to carry forward and start working on this proposal.

Thanks.

Regards,

Parichay (baymac)

[Matt Sicker]

I've been under the impression that freestyle jobs don't really intend
to go anywhere anytime soon. A lot of pipeline functionality is driven
by the same plugin code that freestyle jobs utilize.

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/a872ab3c-d180-4275-81ed-35418805bae2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Matt Sicker
Senior Software Engineer, Jenkins Security, CloudBees

[Parichay Barpanda]

Hi Matt,

Yeah I lately realised freestyle jobs are important as of now. Moreover in the Gitlab plugin freestyle jobs will be of a concern as much as support of pipeline job. I think I'll explore the 2nd option more and figure out if the API can be driven out of the existing GitLab plugin without breaking anything. 

Thanks.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4ozOu5T7RpgO60fKhTkdur1HTRWQ7GEsqEqS9t-Y1EVe9Q%40mail.gmail.com.

[Rick]

Hi,

As said at the title. I think that the freestyle job is not in this scope.

Second, we need a multi-branch API plugin, according to the project proposal https://docs.google.com/document/d/1Gqz4LyU5sw6I50OdAVsQSW_WPNDlvXg4Ic9NdcYj4Ts/edit#heading=h.pr5cab6hhyg.

Good to see your proposal here. But we have another mail list to a discussion about GSoC. https://groups.google.com/forum/#!forum/jenkinsci-gsoc-all-public

And I saw many proposals there from other students.

If you can write your proposal in a Goole Doucment that would be easy to comment and discuss.

Best regards,

Rick

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAD0DWAPqBEkn-EVwfuS7C-1U8dT7sZCLnx6pD5C6e-V%3DCYB7Tg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

https://github.com/LinuxSuRen

[Parichay Barpanda]

Hi Rick,

I was waiting for your comment on this. My main motive was to make this decision on what I have described is whether to develop a new plugin or improvise the existing one. I'm a bit confused if we intend to take out the API from the GitLab plugin will it be able to support the existing applications. I think I need to explore a little more and come up with a clearer idea to discuss with community.

My proposal is developing in a Google doc like the rest of the community. Working on for some changes then I will be drop it in the GSoC mailing list. Hopefully today. 

Regards, 

Parichay (baymac)

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTF7DJfHQvYas09HqR-kaF_S7hsb2SO0YCX9zugg1dcH0Q%40mail.gmail.com.

[Owen B. Mehegan]

I maintain the GitLab plugin, but unfortunately in the last few months I have had no time to dedicate to it. I have been hoping that a capable and motivated person would offer to take over maintaining it, but so far that has not happened. I have also lobbied CloudBees a little bit to invest some resources in this specific feature area, but although they have been positive about the idea, nothing has been committed. So I'm glad to hear that you're interested in at least working on this feature area, it is one I have wanted to see added for a long time.

You say that "basically multibranch pipeline support is non-existent," but IMO this is a bit of an exaggeration. The existing multibranch support is admittedly imperfect because all it does is trigger branch indexing and let Jenkins figure out which branches have changes and should be built. This is not ideal because branch indexing is a somewhat expensive operation, and also as you point out none of the environment variables sent by the webhook are accessible in the job. Having said that, this support has been in the plugin for a few years now and people are definitely using it. 

In my opinion, the path to improving this support is to implement Branch Source support for GitLab. In addition to making Multibranch jobs work better, this would also let Jenkins support the same "org folder" functionality that can already be done with GitHub and Bitbucket. There was someone working on this, but the work has stalled and the person has gone silent. The original plan was that we were going to build this functionality into the existing GitLab plugin, but in recent conversations I have had with other folks, I have come to the conclusion that that would be a bad design, one that goes against the established paradigm of the GitHub and Bitbucket integrations. Both of those have separate trigger, API, and branch source plugins (github, github-api, github-branch-source). https://github.com/jenkinsci/gitlab-plugin/issues/499 discusses this whole proposal, and has links to the incomplete gitlab-branch-source plugin code. I've just seen your comments there (sorry again that I have been unavailable recently), so I'm happy to continue the conversation in that forum. I just wanted to share more widely what the state of the plugin and past proposals for this feature work are.

Thanks again for your interest in contributing!

[Parichay Barpanda]

Hi Owen,

I am really glad this thread got your attention. Firstly, I would like to apologise for being critical about the multi-branch pipeline support in Gitlab Plugin. I acknowledge a lot of hard work and time must have been dedicated to develop the Gitlab plugin. I have been spending time with the codebase learning how it could be improved, I find it very well written and readable. Lots of stuffs in there to learn from.

Secondly, I would like to explain my idea for adding the multi-branch support. I understand the convention for SCM plugins is to implement 2 different plugins (<SCM> and <SCM-Branch-Source>), but the community also understands this is an accident of history. From my discussion with one of the project mentors, Joseph Peterson (@casz) we came to a conclusion that the branch source functionality (folder org type multi-branch pipeline support) can be implemented inside the Gitlab Plugin. So, in order to remove confusion from user point of view we want to implement everything in one plugin. My conviction is this wouldn't cause much disruption of existing conventions because we are only making things simpler. And if we can have a proper documentation of the history of the plugin development then there shouldn't be any issue. Just one plugin to solve all your GitLab related continuous integrations.

Our aim is also to separate the API from GitLab Plugin and create a new API plugin like GitHub API plugin. Now the question is, if we do it, will it break GitLab plugin's existing support? This area needs a little time for research and discussion. I'll come up with a more concrete proposal containing the possible tradeoffs. I'm working on it currently.

Other areas of improvement for GitLab plugin is implementation of new SCM API filters (or traits) rather than having filters inside GitLab plugin. This way other plugins can take the advantage of extending the Gitlab plugin and create their custom filters using the same SCM API.

Btw this idea is one of the GSoC ideas. 

You can find the Mentor's proposal here- 

https://jenkins.io/projects/gsoc/2019/project-ideas/gitlab-support-for-multibranch-pipeline

The progress of my detailed proposal can be found here- 

https://docs.google.com/document/d/1YpuCC129U8KPXAwiXRXQ_4XWuLursPGl3rzQjz43-CY/edit?usp=sharing

Please feel free to drop your comments and suggestions in the Google doc.

Thanks.

Regards,

Parichay (baymac)

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/1ef3514a-7a2f-44aa-b7ba-86be03a2d061%40googlegroups.com.

[Rick]

Hi Owen and Parichay,

Thanks for your time. I'm interested in the GitLab plugin. That's why I created this project proposal.

From my view, there're some similar logic and codes. If I understand correctly, this is also Parichay said

Other areas of improvement for GitLab plugin is implementation of new SCM API filters (or traits) rather than having filters inside GitLab plugin. This way other plugins can take the advantage of extending the Gitlab plugin and create their custom filters using the same SCM API.

We need to consider that other people could integrate multi-branch support in their system. 

Anyway, let's make this better.

Best regards,

Rick

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAD0DWAOCPPL_bxd%3DaRNcr5kC5g6NY9F9Qa6nnmBTa4R8h46MHQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

https://github.com/LinuxSuRen