Investigating Debian source packages repository SCM plugin idea

[Jean-Baptiste Boric]

Hi all,

A bit of context first: we're in the process of migrating/consolidating the various CIs of several software development teams into a single, modern Jenkins instance. One of the projects to migrate is a Debian-based product with the following (highly simplified) CI:

1. Build source code repositories, publish source packages on Debian repository
2. Build Debian source package, publish binary packages on Debian repository
3. Create product artifacts from Debian binary packages

The existing CI is performing step 2 with an old, customized Open Build System (OBS) instance and we're currently evaluating our options for moving forward, with a strong emphasis on simplifying the whole process. One of the options considered would be to use Jenkins for building these Debian source packages so that we’d have only one automation server to worry about. However, we have ~60 Debian source packages across multiple branches to build, so creating ad-hoc jobs for each wouldn’t scale. Therefore, I have emitted the idea of using the Debian source packages of a Debian repository as a SCM provider and let the MultiBranch plugin + external Jenkinsfile take care of business.

Which brings me to the point (and the choice of the developers mailing list instead of the users for this post): as far as I can tell, it appears this SCM plugin does not exist and no one seems to have considered this approach before. Am I correct in this assessment?

Regarding the SCM plugin itself, it would expose for a given distribution in a Debian repository each Debian source package as a SCMHead, masking source packages whose build dependencies are not met. To properly handle the Debian package lifecycle, SCMRevision would encode not only the source package version, but also the latest binary package versions of its immediate build dependencies for a given architecture as well. We are not concerned so far about repository GPG signing, we're fine with basic SCM polling for this particular task and we'll probably make the source package masking and binary dependency versions tracking optional to provide some flexibility.

I did give it a fair amount of thought and note scribbling beyond this very succinct abstract and I believe this to be a workable design, as the normal Jenkins workflow combined with the SCMHead/SCMRevision tricks as described above should be enough to iteratively keep the binary packages of a Debian repository for a given architecture/distribution pair up to date. The feedback from colleagues so far is positive once they wrapped their heads around it, but I haven’t briefed our local Debian expert about this yet since he’s currently on vacation. Have I missed something obvious?

To be clear, we’re not looking to recreate something as performant and feature-complete as Open Build System or buildd that can handle the entire Debian repository without breaking a sweat. We merely want to build our own private collection of Debian source packages, preferably on the same automation server as the rest of our CI. The only missing piece would be this hypothetical SCM plugin, which we’re currently evaluating whether we should go ahead and write it, the rest being standard Jenkins setup/configuration stuff.

In short :

• Does this Debian source packages repository SCM plugin sounds like a good idea on a technical level?
• Has it been done before and have I somehow missed it?
• Have I missed something obvious that would prevent this from working?

Best regards,

Jean-Baptiste Boric