k8s app and helm chart CI with skaffold

[Andrea Frittoli]

Dear all,

first of all thanks for creating Skaffold! 

After I started working with Helm charts, I tried setting up decent dev and test workflows, but I was missing something that would build my docker images and inject them into the helm charts, which is what skaffold does very nicely.

I'd like to setup a CI system to test changes both to the application and docker files, as well as to the helm charts and since I'm not the first one doing this I wanted to ask if you had any recommendation / best practices or even just experiences to share. 

I started looking at drone.io, there's an helm plugin that can be used to deploy a chart, however there's an overlap with skaffold, since drone itself provides facilities to build test images.

Any suggestion would be greatly appreciated!

thanks

Andrea Frittoli

[Kim Lewandowski]

Hi Andrea,

Check out Container Structure Tests for a testing framework. Is this what you're looking for?

[Andrea Frittoli]

On Tuesday, April 17, 2018 at 5:13:31 AM UTC+1, Kim Lewandowski wrote:

Hi Andrea,

Check out Container Structure Tests for a testing framework. Is this what you're looking for?

Thank you Kim,

that is interesting to know about for container testing.

What I'm looking for recommendations for a CI system to test a whole k8s application end to end, which includes building the images, deploying the app, running end to end tests.

I could build docker images on VMs and from there deploy against a k8s test cluster using skaffold, and coordinate the whole process using something like travis.

Or I could use something like kaniko (https://github.com/GoogleCloudPlatform/kaniko) to build the images from within a container. What would work well with drone.io and the drone helm plugin.

Since I'm rather new in this space I was wondering if there is any proven set-up.

thanks!

Andrea Frittoli

[Vic Iglesias]

Hey Andrea!

Can you give an example of the pipeline you'd want to run?

You could do something like this from Drone or any other container based CI system:

1. Checkout code
2. Run unit tests
3. Create namespace $BUILD_ID
4. Use skaffold run to build/push/deploy the app in namespace $BUILD_ID
5. Run integration tests which can be configured run via "helm test $RELEASE_NAME"
   
1. Here are some examples from the charts repo
1. Spinnaker
2. Jenkins test config, Jenkins test runner 
6. Delete namespace $BUILD_ID

Let me know if this maps to what you were thinking.

Thanks,

Vic

[Andrea Frittoli]

On Thursday, April 19, 2018 at 11:54:07 PM UTC+1, Vic Iglesias wrote:

Hey Andrea!

Hi Vic!

 

Can you give an example of the pipeline you'd want to run?

You could do something like this from Drone or any other container based CI system:

1. Checkout code
2. Run unit tests
3. Create namespace $BUILD_ID
4. Use skaffold run to build/push/deploy the app in namespace $BUILD_ID
5. Run integration tests which can be configured run via "helm test $RELEASE_NAME"
   
1. Here are some examples from the charts repo
1. Spinnaker
2. Jenkins test config, Jenkins test runner 
6. Delete namespace $BUILD_ID

Thanks for this and for the example helm charts, it helps!

When using a native container CI system like drone I have the advantage that I can easily create custom test images, and use them to do what I want.

However that has drawbacks too: 

1. I cannot build docker images in a container using docker. I could use something like kaniko but that's not integrated with docker. Is there any plan to have a pluggable build mechanism in skaffold? 

2. I cannot run minikube in a container and run helm against it. I could still deploy against a dedicated k8s test cluster though.

I will go back to the idea of using VM test nodes for now, so I can run docker and minikube on them.

Andrea Frittoli