Building and deploying helm charts with CI/CD

[Michael Venezia]

Hey there

Premise

I've run into a need to build and deploy helm charts similar to container images.  By this I mean we want to be able to version the release, test it, deploy it to a central location, consume it on various kubernetes clusters, and in most cases, share the release.

Helm charts are generally useful for deploying an application, but maintenance and testing on the charts creeps up.  For example an older nginx ingress chart may need to be updated to comply with more current RBAC implementations.  Similarly, a person might forget that "networking/v1" for implementing network policies is only available in 1.7 and not realize that their updated chart no longer works with an older kubernetes environment.

I wanted to describe what I've done to solve some of this, comment about the rough edges that I've found, and see if anyone in the community has done a similar thing or would like to work on a better way to do it.

What I Did

Repository per Chart
I realized that while github.com/kubernetes/charts has some great charts, it does not have all the charts.  And regardless even if it did, I might need to make changes for my needs.  My organization has long had a repository of helm charts, but it was not great for having older versions of the charts or even easily seeing in source control when releases of individual charts were made.  Because of that I opted to move to a repository per chart model.  This way tagging, branching and forking of this chart would be a bit more bite sized and easier to manage. 

You can see an example of it by going to https://github.com/samsung-cnct/chart-mongodb-replicaset

Use quay.io to store charts
I'm not nearly as awesome as others, so the likelihood of getting all of my charts included into kubernetes/charts is very low.  I also figured it did not make sense to maintain my own website that had my charts for a couple of reasons.  One, that was maintenance I did not want to have to maintain.  Two, I would like to share my charts with the community.  Docker hub has been great for sharing container images, github is great for sharing source code, and while I should be allowed to run my own server to store helm charts, I'd prefer to be able to discoverable by others without using Google.

Because of that, I decided to publish my charts to quay.io using the helm appregistry plugin.  It works and solves my problems of having auditing, versioning, and I'm not maintaining it.

You can see an example application repository by looking at https://quay.io/application/samsung_cnct/mongodb-replicaset

Use Jenkins to build and test charts
Once I had a repository for my chart and a place to store it once it was tested and built, I needed something to make the magic happened.  I turned to Jenkins (running in Kubernetes I might add :) ) Multibranch Pipelines.

The logic right now is relatively simple:  When you're ready to do a release tag it in github and push it up.  Take the tag, insert its value into Chart.yaml for the version, do a test (currently just linting, but will do a lot more later), and then, assuming things pass, publish it up to quay.

Jenkins was chosen in part because it is free (like everything else chosen above this), but I'm sure other CI/CD systems could be used.

You can see this by going to https://common-jenkins.kubeme.io/job/chart-mongodb-replicaset/

Rough Edges

Build / Jenkins settings
Helm does have a concept of tests - https://github.com/kubernetes/helm/blob/master/docs/chart_tests.md - but it probably needs a bit of help to enhance it from "Test a specific installation" to "Test a matrix of installations installed on a series of kubernetes environments"  This might have been out of scope for the original tests, or I'm just missing it

Furthermore, I'm not sure where artifacts like "Jenkinsfile" belongs within helm charts.  Currently (and for reasons covered in the next section) I put the chart in a subfolder and create a folder called 'build' that had build information in it - but not everyone will want to use Jenkins and it might be too specific (targeting different kubernetes installations, etc.)

Hard to fork and contribute back
Don't get me wrong - I like that there is a kubernetes/chart repo.  But if you move to a "one repo per chart" model, it is hard to simply fork an existing chart in kubernetes/chart, improve it and submit the changes back to the author.  This may seem frivolous, but I think the easier it is for someone to make an improvement to an existing chart the better it is for the entire community.

Has anyone done something similar or better?

I doubt I'm the only one that has wanted versioned helm chart releases or that wanted to have a build server build them.  But in my limited ability to effectively google, I have not seen a similar set up (centralized helm chart repository, build server, etc.) so that's why I'm posting here.  If someone else has a similar set of needs and has solved it either radically different or similar, perhaps we can work together to help others who will inevitably run into a similar situation.

An additional link that could be helpful
https://github.com/samsung-cnct/template-chart/tree/master/docs - instructions I've given on how to configure a new chart to do what I've been doing.

Thanks

Mike

[Antoine Legrand]

Repository per Chart

 I prefer that model one-repo:one-app too or have the resources located with the application when possible.

Is moving and splitting charts to kubernetes-charts/* repo an option ? 

 

I'd prefer to be able to discoverable by others without using Google.

Because of that, I decided to publish my charts to quay.io using the helm appregistry plugin.  It works and solves my problems of having auditing, versioning, and I'm not maintaining it.

As a maintainer of the app-registry and Quay, thanks for sharing your experience using it. 

You pointed out most of our goals with appr, we want to make it easier to discover, build CI/CD, share and organize charts/kubernetes resources.  

I think, kubernetes/charts would benefit too of using the registry to publish and organize their releases, maybe we can discuss that idea during a sig-apps meeting ? 

 

The logic right now is relatively simple:  When you're ready to do a release tag it in github and push it up.  Take the tag, insert its value into Chart.yaml for the version, do a test (currently just linting, but will do a lot more later), and then, assuming things pass, publish it up to quay.

Has anyone done something similar or better?

 

Jimmy did a Blog post and a short demo/video with a very similar workflow using helm and the app-registry: improving-kubernets-build-and-deployment-pipelines

​

In his example the Chart is located within the application source-code: (ant31/cookieapp). 

 

An additional link that could be helpful

Thanks again, it's very interesting to learn how the community/orgs are doing CI/CD using kubernetes and get more real examples/docs about it. 

--

Antoine

[Brian Grant]

On Tue, Jul 11, 2017 at 2:57 AM, Antoine Legrand <antoine...@coreos.com> wrote:

Repository per Chart

 I prefer that model one-repo:one-app too or have the resources located with the application when possible.

Is moving and splitting charts to kubernetes-charts/* repo an option ? 

FYI, we grabbed the org for this purpose already:

https://github.com/orgs/kubernetes-charts

 

 

I'd prefer to be able to discoverable by others without using Google.

Because of that, I decided to publish my charts to quay.io using the helm appregistry plugin.  It works and solves my problems of having auditing, versioning, and I'm not maintaining it.

As a maintainer of the app-registry and Quay, thanks for sharing your experience using it. 

You pointed out most of our goals with appr, we want to make it easier to discover, build CI/CD, share and organize charts/kubernetes resources.  

I think, kubernetes/charts would benefit too of using the registry to publish and organize their releases, maybe we can discuss that idea during a sig-apps meeting ? 

 

The logic right now is relatively simple:  When you're ready to do a release tag it in github and push it up.  Take the tag, insert its value into Chart.yaml for the version, do a test (currently just linting, but will do a lot more later), and then, assuming things pass, publish it up to quay.
Has anyone done something similar or better?

 

Jimmy did a Blog post and a short demo/video with a very similar workflow using helm and the app-registry: improving-kubernets-build-and-deployment-pipelines

​

In his example the Chart is located within the application source-code: (ant31/cookieapp). 

 

An additional link that could be helpful

Thanks again, it's very interesting to learn how the community/orgs are doing CI/CD using kubernetes and get more real examples/docs about it. 

--

Antoine

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-apps@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/CADkr4xm_nSLuQMccWZfXVr%3Di7yAFmEGdt9tJw0h7GgBcnk53UQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Brian Grant]

On Tue, Jul 11, 2017 at 2:57 AM, Antoine Legrand <antoine...@coreos.com> wrote:

Repository per Chart

 I prefer that model one-repo:one-app too or have the resources located with the application when possible.

Is moving and splitting charts to kubernetes-charts/* repo an option ? 

 

I'd prefer to be able to discoverable by others without using Google.

Because of that, I decided to publish my charts to quay.io using the helm appregistry plugin.  It works and solves my problems of having auditing, versioning, and I'm not maintaining it.

As a maintainer of the app-registry and Quay, thanks for sharing your experience using it. 

You pointed out most of our goals with appr, we want to make it easier to discover, build CI/CD, share and organize charts/kubernetes resources.  

BTW, how is the DAC format specified? I didn't see that in the spec. 

Do you expect a configuration/deployment tool to know to request packages implemented using a compatible format?

Is there a mechanism to map format types to images implementing compatible tools (e.g., helm, kompose, kubectl)?

 

I think, kubernetes/charts would benefit too of using the registry to publish and organize their releases, maybe we can discuss that idea during a sig-apps meeting ? 

 

The logic right now is relatively simple:  When you're ready to do a release tag it in github and push it up.  Take the tag, insert its value into Chart.yaml for the version, do a test (currently just linting, but will do a lot more later), and then, assuming things pass, publish it up to quay.
Has anyone done something similar or better?

 

Jimmy did a Blog post and a short demo/video with a very similar workflow using helm and the app-registry: improving-kubernets-build-and-deployment-pipelines

​

In his example the Chart is located within the application source-code: (ant31/cookieapp). 

 

An additional link that could be helpful

Thanks again, it's very interesting to learn how the community/orgs are doing CI/CD using kubernetes and get more real examples/docs about it. 

--

Antoine

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-apps@googlegroups.com.

[Jimmy Zelinskie]

Hi Brian,

As of the time of this post, the best place to understand the high level goals of App Registry is our talk from CoreOS Fest 2017 and whereas the fine-grained details have to be found in our Python implementation. We prioritized getting something into the hands of users before working on a well defined specification; we’d be grateful to work with anyone interested in fleshing out the spec to define a stable version of the API.

The crux of App Registry is an opaque tarball of configuration data (whatever that may be) and a JSON manifest containing metadata. One piece of metadata is a mimetype that’s used to determine the format of the tarball’s contents -- this is exactly the same mechanisms used by the Docker & OCI image specs to differentiate versions.

The standalone `appr` binary supports different formats, but the user must specify metadata like this via CLI flags. The Helm Registry Plugin is a wrapper of appr that introspects the local Chart to intuit the values you'd normally have to specify manually. Here’s an example usage with just `appr` and a Docker Compose file:

    mkdir mytestcompose && cd mytestcompose

    touch docker-compose.yaml

    appr push --media-type docker-compose quay.io/jzelinskie/mytestcompose:1.0.0

    curl -L https://quay.io/cnr/api/v1/packages/jzelinskie/mytestcompose | jq

    [

      {

        "content": {

          "digest": "2eca5017d4bada1f843722d23c89e9cfb2c59329caf2b4690cb51bec880715bc",

          "mediaType": "application/vnd.cnr.package.docker-compose.v0.tar+gzip",

          "size": 192,

          "urls": []

        },

        "created_at": "2017-07-11T18:12:01",

        "digest": "sha256:ba6e02ca536aff2afcddaf3d0278e75aa9e3c363aacc6d62ee4eef33ba972fb3",

        "mediaType": "application/vnd.cnr.package-manifest.docker-compose.v0.json",

        "metadata": null,

        "package": "jzelinskie/mytestcompose",

        "release": "1.0.0"

      }

    ]

To post to this group, send email to kubernete...@googlegroups.com.

[Antoine Legrand]

BTW, how is the DAC format specified? I didn't see that in the spec. 

App Registry is built to consider a variety of DAC and not solely Helm Charts. As a result, we consider it core to easily specify common formats: docker-compose, ksonnet, *.yaml, Charts….

As described by Jimmy, the protocol is voluntary simple: it’s an opaque tarball and a media-type.
Without restricting any content from using the API for basic primitives such as authenticated storage, our long-term goal with App Registry is to create a specialized behavior for well-defined DAC that manages their metadata/data.

For example, we could:
-  index fields from the Chart.yaml and create a search by authors
-  index by kubernetes-version compat
-  expand templates on the registry/server-side to display a diff between releases
-  create dependency graphs between apps and images by parsing pod specs

In order to provide more dedicated features, we are limiting our focus on DAC formats that resolve to Kubernetes resources. This means that in the common case, Kubernetes users won’t have their applications treated as opaque tarballs in App Registry.

Do you expect a configuration/deployment tool to know to request packages implemented

Yes. We’re working to make the API as simple to implement as possible and in many cases (such has kubectl or Helm), pre-existing plugin models can leverage our existing code.

 using a compatible format?

 

If I’m understanding the question correctly, the goal isn’t to enforce any particular format on the configuration package. In the future, maybe it will make sense to standardize across tools a means of providing application metadata (e.g. Chart.yaml).

Is there a mechanism to map format types to images implementing compatible tools (e.g., helm, kompose, kubectl)?

As said, each format is mapped to a versioned media-type string and tools can leverage this information to request and manage their own packages. Official mappings between popular tools and media-type may live in the written specification, but developers should still be able to create custom mime-types without official support.

One result I’d personally like to see is the workflow management detached from the templating/package format. As a user, I shouldn’t care if the resources are packaged using opencompose, ksonnet, helm, or ‘plain’ kubernetes resources. I would like to manage/deploy/upgrade my applications in a consistent way regardless of the templating format, using my favorite tool (e.g. kubectl, Helm-Tiller, an operator, kpm) and not having to switch between them depending on the app maintainer preferences.

One of our goals for the next cycle is to create a kubectl plugin, part of which can explore this idea:

$ kubectl appr deploy quay.io/ant31/cookieapp

  -> deploying from a 'docker-compose.yml'  (kompose)
$ kubectl appr deploy quay.io/charts/gitlab

  -> deploying from 'helm-chart' package

$ kubectl appr deploy quay.io/ksonnet/redis

  -> deploying from 'ksonnet' package
$ kubectl appr deploy quay.io/k8s/kubernetes-dashboard

  -> deploying from 'plain kubernetes resources’

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsubscribe...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/CADkr4xm_nSLuQMccWZfXVr%3Di7yAFmEGdt9tJw0h7GgBcnk53UQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

To post to this group, send email to kubernetes-sig-apps@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/e19331f7-9337-49d6-9cf3-e47e207f511a%40googlegroups.com.

[salisbur...@gmail.com]

If you're interested, I wrote up how we deploy at Giant Swarm today (see: https://blog.giantswarm.io/giantswarm-continuous-deployment/) - there are some notes on how we deploy with helm and the like there.

Feel free to buzz me if you have any questions.

Cheers,

Joe

[johnny...@gmail.com]

Thank you for a great write-up. This closely resembles how we are doing it at the moment, and you also point out some familiar pain points.

I'd very much like to see the app-registry style of pushing charts to a central repository (e.g. quay.io) for helm. This would simplify the CI/CD workflow.

E.g. we are using Landscaper for keeping the platform apps (deploy daemon, monitoring, metrics etc) in sync across our k8s clusters, but we cannot use quay.io since landscaper does not currently support the app-registry/cnr spec. Instead of being able to simply push the tarball via the cli we now have to clone a repo, move files and regenerate the index file. 

Hopefully Helm will move towards making this the default core functionality and not a plugin once the spec stabilizes. This would provide those building tooling on top of this with a more stable interface.

[salisbur...@gmail.com]

As a note, our opensource tool draughtsman (https://github.com/giantswarm/draughtsman) should be extendable past using Quay / Helm. Not certain how much work is required to integrate with more tools, but the core engine should support it.

In case it helps :D

[Brandon Philips]

I tried reading the draughtsman readme just now and struggling to understand the use case. Could you try to expand the readme and maybe include a concrete example.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/387e1060-8267-45cc-bc1b-f2441d53446d%40googlegroups.com.

[Josh Dolitsky]

Hi, we've just released ChartMuseum which may help you with your chart release process: https://github.com/chartmuseum/chartmuseum

[curlis...@gmail.com]

Hi

Been using these charts in my cluster ...

https://kubernetes-charts.storage.googleapis.com/

Regards

Clark