How to share common configuration across multiple application instances

[John Pugliesi]

Hello,

I have been trying out kpt for the past couple weeks, and appreciate the overall direction it provides towards publishing/consuming k8s config.

I am in the process of exploring how to 

1. create a package of k8s config, and 

2. deploy that package across many different kubernetes clusters, with only slight changes across the configuration (i.e. cluster_name: "my-cluster", environment: "dev", etc.).

Of course, kpt helps solve the first part of packaging up k8s config, but it's unclear how it can be leveraged to achieve my second goal (though I have a feeling it can be used, I'm just not thinking of it in the right way yet).

Here's an example of what I'm trying to achieve:

Say I have two k8s applications, each of which has a config variable for an S3 bucket. I'd like to deploy these two applications in many different k8s clusters. Each k8s cluster has its own S3 bucket. When both applications are deployed in the same cluster, they should be configured with the same S3 bucket that belongs to that cluster. But instances of the applications in different clusters will of course be configured with different S3 buckets.

It's unclear to me how I can achieve this with kpt and kustomize. Specifically, I dont know how to avoid copy/pasting the S3 bucket name across many different application and/or kustomization.yaml manifests (which of course is error prone). Ideally, I'd define the cluster's s3 bucket config value in one place, and then use that config value in many kustomized application instances within that cluster.

I'm wondering if:

1. Theres a way to structure the kustomize folders/overlays to achieve this, which I haven't yet considered, and/or

2. If kpt can help solve this problem in tandem with kustomize (via setters/substitutions, or functions....?)

I really appreciate any help, thoughts, and guidance you can provide in addressing this use case!

Thanks,

John

[Phillip Wittrock]

1. How are you configuring the app -> cluster -- Is it through the `--context` flag, or something else?  How would I know which cluster a resource is being deployed to based on the resource itself?
2. Do you want to have a separate set of config / target per-cluster, or a single set of config that is transformed as it is applied to a cluster?

From your message, I am guessing you want a separate config for each app+cluster+env pair, and want values such as the S3 bucket to "just be correct" based on the cluster value.

Do you need the layer of indirection: cluster -> bucket and package -> cluster, or is package -> bucket sufficient?  If it is simply package -> bucket, then using kustomize should be sufficient -- e.g. use a configmap generator or patch to set environment variables for each Deployment/StatefulSet/etc, or something of this sort.  You would have a separate kustomization file for each app/cluster/env pair.

If you really need the layer of indirection so that cluster -> bucket is defined in only 1 place, and packages define their cluster only (not the bucket), then it is a bit less obvious how you would do this.  The question then is how do you mix-in a cross-cutting value that is derived from some other value

I can think of a couple options:

• Option 1: If you are declaring the cluster as part of the resource configuration (e.g. as an annotation on each resource or in the cluster metadata field), write a function that for each resources, updates the S3 buckets based on the cluster annotation and some lookup table of cluster -> S3 bucket -- this could even update the kustomization file or patch files.
• The cluster could also be derived from the directory structure or filenames for a convention over configuration approach
• Option 2: Set the S3 bucket values at deploy time based off which cluster it is being deployed to -- this could be implemented using either setters or a function.
• Option 3: If it is the same field in each app that is set (e.g. the path to the field is always the same), then you could use a set of kustomize patches which target based on an annotation or label (in the cluster).  https://github.com/kubernetes-sigs/kustomize/blob/master/examples/patchMultipleObjects.md

Options 2 and 3 are relatively simple to implement.  Option 1 is more difficult, but provides a lot of control over how the resources look.

We are working on ways to reduce the overhead of writing and running functions to make the first option simple, but haven't gotten there yet so expect a bit of work.

If you like typescript, the functions sdk may be a good way to write the function: https://github.com/GoogleContainerTools/kpt-functions-sdk

- Phil

--
You received this message because you are subscribed to the Google Groups "kpt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kpt-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kpt-users/96301eed-dec3-4aa9-91f4-9b48a4b4b86e%40googlegroups.com.