[DISCUSS] Switching to CentOS for Jenkins Docker base image

222 views
Skip to first unread message

Baptiste Mathus

unread,
Feb 26, 2019, 4:23:24 PM2/26/19
to Jenkins Developers
Hey everyone,

We had a discussion at the regular Platform SIG meeting, and we thought we would socialize this idea/subject with a larger group of people.

Many events or issues have recently collided that led many us to question the Docker image variants that the Jenkins Project provides, especially the base images that these are based on. 
Currently, we provide 3 variants, all are based on openjdk's provided images
* Debian based
* Alpine based

Given a few recent events around Debian or Ubuntu JDK [1], we started thinking it might be good to switch to a different base image. 
(The situation around the JDK support is BTW even worse for Alpine. Which led to switching Evergreen already [2]).

So we have started thinking CentOS would seem like a good alternative choice, given how central Java is to this platform in general.
In this regard, going to Fedora could probably be a possibility too.

We are really interested in hearing your opinions around this. Please voice your concerns, if any. 
If you think this sounds like a sensible move, please don't hesitate to let us know either :).

Thanks!

-- Baptiste

[2] The ticket for switching Evergreen to CentOS, already done https://issues.jenkins-ci.org/browse/JENKINS-55547



Matt Sicker

unread,
Feb 26, 2019, 4:34:07 PM2/26/19
to jenkin...@googlegroups.com
Based on the details regarding Debian and Ubuntu's poor maintenance of JDK packages, I'd support using a different distro like CentOS. That certainly pushes myself away from defaulting to ubuntu or debian for Java Docker images.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7kdR1cqVaKsU7A4wOvyGWYyUrUKjAbtMCP6QC5oHpEjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
Matt Sicker
Software Engineer, CloudBees

R. Tyler Croy

unread,
Feb 26, 2019, 5:31:46 PM2/26/19
to jenkin...@googlegroups.com
(replies inline)

On Tue, 26 Feb 2019, Matt Sicker wrote:

> Based on the details regarding Debian and Ubuntu's poor maintenance of JDK
> packages, I'd support using a different distro like CentOS. That certainly
> pushes myself away from defaulting to ubuntu or debian for Java Docker
> images.


Agreed! I think this is worthwhile to do. The first responsibility IMO of our
containers is to provide the most stable and secure Jenkins environment for end
users.

We already maintain an Alpine image, to where if there are people depending on
a Debian-based image, making that available as another set of tags is also
always an option.


+1

--
GitHub: https://github.com/rtyler

GPG Key ID: 0F2298A980EE31ACCA0A7825E5C92681BEF6CEA2

Mark Waite

unread,
Feb 26, 2019, 5:52:27 PM2/26/19
to jenkinsci-dev
On Tue, Feb 26, 2019 at 3:31 PM R. Tyler Croy <ty...@monkeypox.org> wrote:
(replies inline)

On Tue, 26 Feb 2019, Matt Sicker wrote:

> Based on the details regarding Debian and Ubuntu's poor maintenance of JDK
> packages, I'd support using a different distro like CentOS. That certainly
> pushes myself away from defaulting to ubuntu or debian for Java Docker
> images.


Agreed! I think this is worthwhile to do. The first responsibility IMO of our
containers is to provide the most stable and secure Jenkins environment for end
users.

We already maintain an Alpine image, to where if there are people depending on
a Debian-based image, making that available as another set of tags is also
always an option.


Adding tags for CentOS images seems least likely to break existing users.

Since the base Jenkins docker image does not include certain critical programs
(like "git-lfs"), I needed to extend the image by installing additional packages with
the operating system package management system.  It works well and lets me
test interesting cases faster.

If we add a CentOS image, I'm very much in favor of it.  +1 if so

If we replace the Debian image with a CentOS image, that will be disruptive
to me and those like me that needed additional operating system specific
extensions in the image.  I can adapt, but there will be others that will
grumble about adapting.

Is the proposal to add CentOS or to replace Debian with CentOS?

Mark Waite
 

+1

--
GitHub:  https://github.com/rtyler

GPG Key ID: 0F2298A980EE31ACCA0A7825E5C92681BEF6CEA2

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

Richard Bywater

unread,
Feb 26, 2019, 6:49:24 PM2/26/19
to jenkin...@googlegroups.com
If the concern is regarding packaging, would it be worthwhile switching the images to drag in something like a AdoptOpenJDK archive and use that instead? (Presumably would need to create symlinks etc. for binaries). I took a look at the CentOS OpenJDK11 package and it was hard to tell how up-to-date it was given it seems to be from November 2018 but AdoptOpenJDK has a build from late January.

Richard.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtHk38Oi7zcKarTLZ_6L%2By12nmAa1o9-7VaAxn4MR2612g%40mail.gmail.com.

Slide

unread,
Feb 26, 2019, 7:17:16 PM2/26/19
to Jenkins Developer List
Looks like AdoptOpenJDK had some different options as well for base Docker images. 


To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAAy0hwczub-i9bouUoRyu%2BsSD7FcvBXu9yuEQVxHsa%3DFdo%3DsUw%40mail.gmail.com.

Richard Bywater

unread,
Feb 26, 2019, 7:19:55 PM2/26/19
to jenkin...@googlegroups.com
Ah - hadn't realised they also did Docker image. Thanks for pointing that out.

Richard.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPiUgVcnJ7N%2B3iTQS9n3S4zJ7N4zHvWOBayoqXwK38Nm0kU93g%40mail.gmail.com.

Olblak

unread,
Feb 27, 2019, 3:46:31 AM2/27/19
to 'Gavin Mogan' via Jenkins Developers
I agree with Mark, I am also not a big fan of changing suddenly from Debian to Centos for the base image as it means additional unplanned works for all people who rely on jenkins/jenkins as their base image.
Unless we also provide a new 'debian' tag for those who don't have the time to update their image at the moment

So they can just change 'FROM jenkins/jenkins:latest' to 'From jenkins/jenkins:debian-latest' until they have the time to properly work on their Dockerfile.

But I am wondering, instead of going with Centos why not using this PPA with ubuntu?
This would imply a smaller breaking change

---
-> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
---


Matt Sicker

unread,
Feb 27, 2019, 10:39:33 AM2/27/19
to jenkin...@googlegroups.com
On Wed, Feb 27, 2019 at 2:46 AM Olblak <m...@olblak.com> wrote:
> But I am wondering, instead of going with Centos why not using this PPA with ubuntu?
> This would imply a smaller breaking change

I'd support that as much as CentOS provided that they're serious about
packaging openjdk and not some weird mish-mash of security backports
developed in house.

This also reminds me that if we were to use another image similar to
openjdk instead of a clean base image, do note that they tend to come
with a bunch of useless programs. For example, openjdk:latest
typically bundles Mercurial for some reason.

Whatever approach we take, I'd prefer for us to install the minimal
set of packages required to run Jenkins and allow administrators to
install additional packages afterwards. Or even a system similar to
JFR for creating custom Docker images using a Jenkins config or
similar.

R. Tyler Croy

unread,
Feb 27, 2019, 11:29:20 AM2/27/19
to jenkin...@googlegroups.com
(replies inline)

On Wed, 27 Feb 2019, Olblak wrote:

> But I am wondering, instead of going with Centos why not using this PPA <https://launchpad.net/~openjdk-r/+archive/ubuntu/ppa> with ubuntu?
> This would imply a smaller breaking change

I do not believe that Jenkins should rely on any PPA (Personal Package
Archive), they have a tendency of growing stale unlike mainstream official
packages.

Oleg Nenashev

unread,
May 10, 2019, 4:09:56 AM5/10/19
to Jenkins Developers
FYI there is a pull request for CentOS image in Jenkins Docker packages

Scott McCarty

unread,
May 15, 2019, 5:14:40 PM5/15/19
to Jenkins Developers
All,
    I saw this thread a while back, but couldn't respond until after we launched UBI publicly. UBI follows the RHEL lifecycle, but has the added bonus that 1. new versions come out before CentOS and 2. receives RHEL updates (exact same RPMS). You can build on think of it as CentOS+ when ran anywhere, with the added bonus that it can be run on RHEL/OpenShift and be fully supported by Red Hat. It's distributed under a different EULA than other Red Hat which does allow redistribution of Red Hat trademarks in the content set (YUM/RPMS, images, etc). Also, we will likely add packages in the future, but will never remove them. Feel free to ping me if you have any questions (smcc...@redhat.com) or this email...
Message has been deleted

Fred Blaise

unread,
Jun 17, 2019, 1:01:48 PM6/17/19
to Jenkins Developers
Hi Scott,

What do you think of the export restrictions in the EULA? (some ref: https://www.law.cornell.edu/cfr/text/15/740.17)

Any chance you could confirm internally with Redhat that UBI is 100% fit for Jenkins open-source?

Thank you.
Best,
fred

Oleg Nenashev

unread,
Jun 18, 2019, 10:00:32 AM6/18/19
to Jenkins Developers
FTR https://github.com/jenkinsci/docker/pull/826 for CentOS.

Regarding UBI, I have the same concern as Fred. We have no tools to enforce the Export limitations on DockerHub. I am also not sure that restricting specific countries according to US laws is compliant with how the Jenkins open-source project operates. IIRC we used to have contributors from the countries restricted by US.

Best regards,
Oleg

Scott McCarty

unread,
Jun 18, 2019, 2:42:00 PM6/18/19
to Jenkins Developers
Oleg & Fred,
     Very good question. I am actually not sure myself, exactly what these restrictions mean. I am going to run it by one of our lawyers and get back to you. I will try and get more clarity...

Best Regards
Scott M

Scott McCarty

unread,
Jun 19, 2019, 3:45:50 PM6/19/19
to Jenkins Developers
So, I went and did some research on this. Disclaimer, I am not a lawyer, and Red Hat can't give specific legal advice. That said, these export restrictions are in place and applicable no matter which base image you choose/use (Alpine, CentOS, Debian, Ubuntu, etc). Essentially, the law is the same no matter what, and can extend to non-US citizens as well (I remember this from our yearly legal training) which I dread in December :-/

The difference here is that the UBI EULA is basically making people pay attention to the problem now. Obviously, Red Hat is not going to be the entity suing you if you break export compliance, it would be the US government.  Apparently, the whole world is "doing this wrong" today and the world hasn't ended. I totally understand your nervousness with seeing this in writing now.

I tried to check the DockerHub FAQ [1], but it "looks" like they may only be enforcing export compliance for their own products (they are an entity that might be targeted). We are doing the same thing for quay.io and I could talk to the quay people to have this turned on if you wanted to distribute there (aka then quay.io would block those countries for you). Quay.io has a roadmap item to give people a "check box" to turn this on, but it doesn't exist yet and appears delayed. The short term solution is "ask quay.io to turn it on behind the scenes" - sub optimal, but still good that it's available.

Best Regards
Scott M

Fred Blaise

unread,
Jun 19, 2019, 6:16:53 PM6/19/19
to jenkin...@googlegroups.com
Thank you Scott for going the extra mile. 

Your answer is what I expected it to be, and I would somehow concur on the fact that it's been around forever, but noone ever really cared.

CentOS has them too, as you mentioned: https://www.centos.org/legal/

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6c0842d2-7e1c-4e00-97a0-3fea4eac979f%40googlegroups.com.

Scott McCarty

unread,
Jun 19, 2019, 7:56:48 PM6/19/19
to jenkin...@googlegroups.com
Fred,
    Yeah I just googled for Debian [1], Suse [2], Ubuntu [3]. They all pretty much say the same thing. US Law, blah, blah, blag. Like I said, I am now 99% sure the whole world is just "doing it wrong" and most entities (projects, people, companies) are too small for the government to notice.

I also scheduled a call with our legal export team to better understand this myself. I need to be able to, at least describe it to communities, partners and customers...

Like I said, quay.io can manually turn on an enforcing feature if you need it. I suspect DockerHub can too...


Best Regards 
Scott M

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPNh5Ty7%3DoO%2BJCXaokqyPrthE4xKLZvG981b4dT%2BoH85f4XBaQ%40mail.gmail.com.

Oleg Nenashev

unread,
Jun 20, 2019, 4:53:07 AM6/20/19
to Jenkins Developers
Yeah, it looks like to be the state of the affairs anyway. I am not against adding UBI to the distribution, but I think it is fine if we keep both CentOS and UBI.

Regarding quay.io, it worth discussion in a separate thread.
We can always update our release scripts to deploy to this platform, it is rather a question of the implementation cost and further maintenance overhead.

Best regards,
Oleg
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages