Proposal: Cloud Native Special Interest Group

90 views
Skip to first unread message

Carlos Sanchez

unread,
Jul 4, 2018, 12:16:26 PM7/4/18
to Jenkins Developers
Hi there,

I have submitted a JEP draft for a SIG around "Cloud Native", to foster collaboration around improving Jenkins to run on Cloud environments as a "Cloud Native" application.
Stories covered here include cloud artifact storage, external logging, external database for configuration,...


I have had talks with some cloud providers and there is a lot of interest on making Jenkins run more integrated in cloud environments, so looking forward to have a common group where we can collaborate.

Thanks

Tracy Miranda

unread,
Jul 5, 2018, 5:42:39 AM7/5/18
to jenkin...@googlegroups.com
+1. Great and much needed initiative. 

I confirm I will be participating. 

Regards,
Tracy

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALHFn6PmTj2N%3Dz_KzTaD%3DjQ0ue8O8%3DSGigzBDgF2V%2BtC4d0Eug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

nicolas de loof

unread,
Jul 5, 2018, 8:22:20 AM7/5/18
to jenkin...@googlegroups.com
+1 as well. Confirm I'd like to be in and contribute, at least for Configuration-as-Code support, but not limited to this scope as long as I can be helpful. 

Le jeu. 5 juil. 2018 à 11:42, Tracy Miranda <tmir...@cloudbees.com> a écrit :
+1. Great and much needed initiative. 

I confirm I will be participating. 

Regards,
Tracy
On Wed, Jul 4, 2018 at 5:16 PM, Carlos Sanchez <car...@apache.org> wrote:
Hi there,

I have submitted a JEP draft for a SIG around "Cloud Native", to foster collaboration around improving Jenkins to run on Cloud environments as a "Cloud Native" application.
Stories covered here include cloud artifact storage, external logging, external database for configuration,...


I have had talks with some cloud providers and there is a lot of interest on making Jenkins run more integrated in cloud environments, so looking forward to have a common group where we can collaborate.

Thanks

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CACTaz6rY5QekJUn7qwG1iTwhk0QXV-14SDnZqMW1pU%3DEMZtc5w%40mail.gmail.com.

Carlos Sanchez

unread,
Jul 5, 2018, 9:16:30 AM7/5/18
to Jenkins Developers
Given that JEP is not needed I'll add here the content for discussion

== Abstract

We see more and more users running Jenkins in the Cloud and containerized environments, and that number is not going down.
However, we often see Jenkins running as a "legacy" app that does not make use of the advantages provided by a cloud environment.

== Specification

The Cloud Native group of contributors and collaborators focuses on improving Jenkins to run on Cloud environments as a "Cloud Native" application.

=== Target Audience

The improvements are targeted at both existing and new Jenkins users that use, or would prefer to use, Jenkins deployed in one of the cloud providers, or using cloud services for their operation.

=== Meetings

There will be regular SiG meetings scheduled.
Initially the meetings will be conducted once per month, then they may be adapted according to the activity.

Meetings will be conducted and recorded via Jenkins Hangouts-on-Air.

=== Chairs

* Carlos Sanchez

=== Areas for Improvement

There are several clear areas open for improvement, which are summarized here and will be detailed in future documents.

A mayor pain point is the usage of local disk as all-purpose storage, which causes issues running on containerized or distributed environments, requiring highly performant filesystems, and all the configuration pain like initial sizing and resizing with downtime.

We believe that by using cloud provided services the overall usability, performance and scalability can be improved while enabling new demanded functionality.

By converting Jenkins to a "stateless" application long awaited features such as high availability and replication would help to operate Jenkins more efficiently, with zero downtime and reducing risk in upgrade operations by using canary or rolling deployments.

==== Artifact Storage

The current approach of storing artifacts in the filesystem prevents users from taking advantage of cloud provided blob stores (ie. AWS S3), which provide a lot more features for this use case.

Sizing becomes a problem as disk usage needs to be accounted for all the future artifacts stored or forces the need for resizing, which causes downtime.
Typical use cases such as versioning, expiration or pay per use are impossible to implement today.

While currently it is possible to use specific plugins to store artifacts in blob stores such as the https://plugins.jenkins.io/s3[S3 plugin],
this forces explicit configuration in jobs and makes configuration harder.

==== Log Storage

Logs disk usage causes the same issues previously mentioned for artifacts.
Plus a external focused log storage such as https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html[AWS CloudWatch Logs] allows demanded features as centralized log management, log retention policies, advanced querying, ...

There are already options to externally ship the logs to a backend, or plugins that would do that like the  https://github.com/jenkinsci/aws-cloudwatch-logs-publisher-plugin[aws-cloudwatch-logs-publisher-plugin], but there is no integrated way to both send and display logs from external log storage.

==== Configuration Storage

While xml files do not typically take a big amount of disk space, they make it really hard to query information about the system.
This configuration is better suited in a cloud provided database in combination with https://github.com/jenkinsci/configuration-as-code-plugin Configuration as Code

==== Replication

If all the data currently stored in the filesystem is moved to external storage a replicated Jenkins service becomes possible, and the next steps are true High Availability, rolling or canary upgrades and zero downtime.


== Motivation

The current default approach of storing everything into the filesystem is the main reason why Jenkins does not fit the "Cloud Native" model, with features like HA, zero downtime, or pay per use.

While there are plenty of plugins that implement parts of this vision, this becomes cumbersome to configure and a usability nightmare for users, as the https://github.com/jenkinsci/jep/tree/master/jep/300 [Essentials JEP] has pointed out.
Going towards a model where cloud services are consumed where it makes sense, the overall complexity on operating Jenkins in a cloud or containerized environment is greatly reduced.

Other related projects include https://github.com/jenkinsci/jep/tree/master/jep/400 [Jenkins X] which would greatly benefit from a Cloud Native Jenkins inside Kubernetes.

== Reasoning

Later JEPs in this series will will address design and implementation.

This document just lays down the ideas behind a "Cloud Native" Jenkins to build consensus around them.

== Backwards Compatibility

Backwards compatibility of the specific implementations will be discussed in future documents.
It is likely that new versions of core will be needed to enable such features, but nothing proposed as part of this goal would prevent the usage of Jenkins in non cloud environments.

== Infrastructure Requirements

Although there are no infrastructure requirements to discuss in this document, additional documents which propose new cloud functionality will likely request the ability to test that functionality in the target cloud environments.

It is not expected that the Jenkins project would provide all resources needed for such environmental tests.

Olblak

unread,
Jul 6, 2018, 4:40:01 AM7/6/18
to jenkin...@googlegroups.com
+1 I am also interested by this topic

---
-> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
---


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Samuel Van Oort

unread,
Jul 6, 2018, 8:17:27 AM7/6/18
to Jenkins Developers
I would also be interested in participating -- I'm a reviewer for some of the related work and some of the things I'm doing will overlap with this space. 

Oleg Nenashev

unread,
Jul 6, 2018, 12:32:36 PM7/6/18
to Jenkins Developers
I'm in

linuxsuren

unread,
Jul 7, 2018, 10:01:03 PM7/7/18
to jenkin...@googlegroups.com, jenkin...@googlegroups.com
+1 I am glad to follow


素人派
生命不息,折腾不止,技术改变生活!


https://github.com/LinuxSuRen

Leonardo Rossetti

unread,
Jul 9, 2018, 9:36:11 AM7/9/18
to jenkin...@googlegroups.com
I would like to join/contribute as well.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/FD8E11DB-ACFB-4441-8241-488C9E48C070%40gmail.com.

Baptiste Mathus

unread,
Jul 9, 2018, 9:58:45 AM7/9/18
to Jenkins Developers
+1. Having worked on this quite recently, I'm interested to participate :).

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CACHG7gxRbKqZDDAiU2qCvAbsOV2G7WUx2_dy1wfNQUkwd%2B2DYg%40mail.gmail.com.

Vic Iglesias

unread,
Jul 9, 2018, 4:41:40 PM7/9/18
to Jenkins Developers
+1 Looking forward to engaging in this SIG.


On Monday, July 9, 2018 at 6:58:45 AM UTC-7, Baptiste Mathus wrote:
+1. Having worked on this quite recently, I'm interested to participate :).

Le lun. 9 juil. 2018 à 15:36, Leonardo Rossetti <m...@lrossetti.com> a écrit :
I would like to join/contribute as well.
2018-07-07 22:58 GMT-03:00 linuxsuren <linux...@gmail.com>:
+1 I am glad to follow


素人派
生命不息,折腾不止,技术改变生活!


https://github.com/LinuxSuRen

Jesse Glick

unread,
Jul 9, 2018, 5:54:32 PM7/9/18
to Jenkins Dev
On Thu, Jul 5, 2018 at 9:16 AM Carlos Sanchez <car...@apache.org> wrote:
> Given that JEP is not needed I'll add here the content for discussion

Ideally we would have a “vision JEP”¹ which lays out the general
requirements which particular topics like JEP-202 can refer to. This
is what Essentials did with JEP-300, and Jenkins-X with JEP-400.
Ideally we would have our own 5xx series or something, but then
JEP-202 would better be renamed JEP-501.

I agree we do not need JEPs for SIGs, but this is not just a SIG like
others proposed (e.g., Jenkins on special platforms)—it is an actual
proposed series of changes, each of which would be a regular JEP.


¹JEP-300 and JEP-400 are listed as “Standards” but this fails to
capture the distinction from regular feature JEPs. I suggest that

https://github.com/jenkinsci/jep/blob/master/jep/1/README.adoc#jep-types

be amended with a new category.

martinda

unread,
Jul 9, 2018, 6:53:07 PM7/9/18
to Jenkins Developers
Hello Carlos,

Interesting initiative. Did you know that the External Workspace Manager Plugin was designed with Cloud Storage in mind? Perhaps that might be useful.
I am interested in knowing where this SIG is headed, but I can make no commitment.

Martin

Evan Brown

unread,
Jul 11, 2018, 9:27:32 AM7/11/18
to Jenkins Developers
+1 SIGn me up!

Liam Newman

unread,
Jul 12, 2018, 1:48:15 PM7/12/18
to Jenkins Developers
I'm creating the SIG now, following the steps JEP-4 with the PR from Oleg. 
sig-id: cloud-native

Jesse,
Your suggestion re JEP-1 is interesting and accurately points out a rough edge in the current JEP process.
I think this particular structure has come out of the size of some of these JEPs. 
Please start a thread on here and maybe file a PR with suggested changes.  

Oleg Nenashev

unread,
Jul 12, 2018, 1:50:09 PM7/12/18
to JenkinsCI Developers
Hi Liam,

I have draft for the SIG so I can submit it to save your time.

BR, Oleg

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/CEeQwqQaGLo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/e721762c-9f6f-4458-86ec-65f5795a1710%40googlegroups.com.

Liam Newman

unread,
Jul 12, 2018, 2:31:12 PM7/12/18
to Jenkins Developers
Google Group: https://groups.google.com/forum/#!forum/jenkins-cloud-native-sig

Oleg will submit a PR shortly with the sig page. 

Oleg Nenashev

unread,
Jul 12, 2018, 2:59:21 PM7/12/18
to JenkinsCI Developers

@Carlos, do you want to have a GitHub Team created? If yes, we will need GitHub IDs of all participants.
In such case please create an INFRA ticket and assign it to me.


To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/28de647c-ce58-4990-8fd7-aea3232bfb2f%40googlegroups.com.

Daniel Beck

unread,
Jul 12, 2018, 5:15:54 PM7/12/18
to jenkin...@googlegroups.com

> On 12. Jul 2018, at 20:59, Oleg Nenashev <o.v.ne...@gmail.com> wrote:
>
> @Carlos, do you want to have a GitHub Team created? If yes, we will need GitHub IDs of all participants.
> In such case please create an INFRA ticket and assign it to me.
>

Already done for https://issues.jenkins-ci.org/browse/INFRA-1706:

https://github.com/orgs/jenkinsci/teams/cloud-native-sig

Carlos is a team maintainer, so can add and remove people.

Carlos Sanchez

unread,
Jul 13, 2018, 3:40:31 AM7/13/18
to Jenkins Developers
Thanks all!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/62A64574-6189-4F4A-877D-13E87C375A2E%40beckweb.net.

Oleg Nenashev

unread,
Jul 13, 2018, 5:29:43 AM7/13/18
to JenkinsCI Developers
OK, SIG and all its resources are live: https://jenkins.io/sigs/cloud-native/
The next step is to schedule some meetings :)

In 2016 we were working on the External Build Log storage, and I would like to recover the discussion.

Best regards,
Oleg


On Fri, Jul 13, 2018 at 10:40 AM, Carlos Sanchez <car...@apache.org> wrote:
Thanks all!

On Thu, Jul 12, 2018 at 11:16 PM Daniel Beck <m...@beckweb.net> wrote:

> On 12. Jul 2018, at 20:59, Oleg Nenashev <o.v.ne...@gmail.com> wrote:
>
> @Carlos, do you want to have a GitHub Team created? If yes, we will need GitHub IDs of all participants.
> In such case please create an INFRA ticket and assign it to me.
>

Already done for https://issues.jenkins-ci.org/browse/INFRA-1706:

https://github.com/orgs/jenkinsci/teams/cloud-native-sig

Carlos is a team maintainer, so can add and remove people.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/CEeQwqQaGLo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

Tracy Miranda

unread,
Jul 16, 2018, 4:01:31 PM7/16/18
to jenkin...@googlegroups.com
Hi all,

Thanks all for getting all the parts in place. 


Thanks all to those for your public show of support so far - the response has been great.
To build momentum I would like to add all the folks on this thread as participating members on that page (just names, no emails)
If you would prefer not to be included please let me know on or off list. 

We will also be listing participating organizations so if you would like your organization to be listed please let me know (this can be in addition or instead of individual's names).

Regards,
Tracy


Ewelina Wilkosz

unread,
Jul 18, 2018, 2:48:19 AM7/18/18
to Jenkins Developers
can I still join? :)
Thanks all!

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/CEeQwqQaGLo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Oleg Nenashev

unread,
Jul 18, 2018, 3:49:39 AM7/18/18
to JenkinsCI Developers
Sure. Just join the mailing list :)


To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/16743eae-5dfd-4d9f-9319-115c7c08667d%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages