Keycloak container - stop and start

3,541 views
Skip to first unread message

Sebastian Łaskawiec

unread,
Jun 23, 2020, 3:53:32 AM6/23/20
to Keycloak Dev
Hey,

Over the last few months we received a few requests to implement Container stop and start. A few weeks back we received a Pull Request with an implementation proposal [1], however, I'm not convinced we should support this at all!

When executing `docker stop`, the Docker Daemon sends SIGTERM and SIGKILL signals to the root process inside the container (PID = 1). Starting a stopped container (`docker start`) executes the ENTRYPOINT, however all files that had been modified before stopping the container remains untouched.

This is a bit problematic in case of both Keycloak and SSO containers. During the bootstrap process we modify standalone-ha.xml (for Keycloak) and standalone-openshift.xml (for SSO) based on environment variables and other mounted files. Those modifications are permanent between stop and start. So, when we start a stopped container, our ENTRYPOINT scripts try to modify files for the second time - leading to a crash.

There are two ways of solving this in my opinion:
- make all the CLI script idempotent. This means, we'd need to introduce an "if-exists" check to all the things we add to the configuration. If something already exists, we'd need to skip it.
- make all shell script conditional. If we discover that we're restarting a container, we'd need to back off.

However, when thinking about this feature a bit more, I'm not sure if we should support it at all. This opens up Pandora's Box and may lead to slipping many unwanted features in. Just a few examples:
- (as Iavael suggested in [1]): When a TLS key/cert changes between stopping and starting the container, we should probably pick up the changes.
- The same applies for TrustStore
- What if the environment variables change?
- What if we operate within a cluster and its settings changed?

I used to think about containers as immutable things. Whenever you'd like to update your configuration (like mount a new TLS key/cert), you always need a restart. This is true in Kubernetes world (e.g. if you'd like to mount new values from an updated ConfigMap you need to restart the container). Unless someone convinces me otherwise, I'd vote for leaving the support for stop/start out.

What you guys think about this?

Thanks,
Sebastian


Schuster Sebastian (IOC/PDL22)

unread,
Jun 23, 2020, 4:20:08 AM6/23/20
to Sebastian Łaskawiec, Keycloak Dev

I agree. Keeping things immutable, throwing them away and recreating them if something changes and not pampering some configuration with all its problems over time (configuration drift etc.) is the way to go IMHO.

 

Best regards,

Sebastian

 

Mit freundlichen Grüßen / Best regards

Dr.-Ing. Sebastian Schuster

Project Delivery Berlin 22 (IOC/PDL22)
Bosch.IO GmbH | Ullsteinstr. 128 | 12109 Berlin | GERMANY | www.bosch.io
Tel. +49 30 726112-485 | Mobil +49 152 02177668 | Telefax +49 30 726112-100 | Sebastian...@bosch.io

Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling

--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/18aa9216-c4fd-47d9-85a6-ad92f790c8b7o%40googlegroups.com.

Stian Thorgersen

unread,
Jul 6, 2020, 5:35:06 AM7/6/20
to Sebastian Łaskawiec, Keycloak Dev
+1 I don't see a real practical use-cases for stop/start of containers.

Tom Mewett

unread,
Aug 18, 2020, 6:38:09 AM8/18/20
to Keycloak Dev
Hi, we've run into this issue internally, and would love to be able to restart containers after a reboot. We'd like to understand more about the decision to not support it.


> What if the environment variables change?

Is it possible to change the environment on a created container? I'm not sure this is supported by Docker. Even so I think it would be reasonable to not support env var changes; I think this is tangential to the issue of starting and stopping containers.

> I used to think about containers as immutable things. Whenever you'd like to update your configuration (like mount a new TLS key/cert), you always need a [re-run].

I also think this is tangential, and I think it would be helpful to isolate these issues in discussion. There are two separate questions here:

1. How the container responds to changed configuration (where it is possible to change, e.g. mounted volumes)
2. Whether the container can be restarted, with the same configuration

We only want (2), and most of the work in the linked PR has gone into (2), so I'm not sure it should be dismissed on the basis of (1).

The way I see it, the current behaviour of the start-up scripts is actually contrary to the idea that the container is immutable after creation. Making all the configuration scripts execute only once would support start/stop without introducing any issues with regard to changing configuration. Unless I misunderstand something, I think this would be an uncontroversial change and would solve (2). What do you think?

The other point is about configuration by mounted volumes, e.g. certs; I think you're right that changes to these should be picked up. Is it possible to make these scripts, which copy/reformat the mounted files, execute every time? That would solve (1). But again, I think this could be broken out into a separate issue.

Sebastian Łaskawiec

unread,
Aug 19, 2020, 2:28:10 AM8/19/20
to Tom Mewett, Keycloak Dev
On Tue, 18 Aug 2020 at 12:38, Tom Mewett <tom.m...@codethink.co.uk> wrote:
Hi, we've run into this issue internally, and would love to be able to restart containers after a reboot. We'd like to understand more about the decision to not support it.

> What if the environment variables change?

Is it possible to change the environment on a created container? I'm not sure this is supported by Docker. Even so I think it would be reasonable to not support env var changes; I think this is tangential to the issue of starting and stopping containers.

The short answer is no.
 

> I used to think about containers as immutable things. Whenever you'd like to update your configuration (like mount a new TLS key/cert), you always need a [re-run].

That's correct.
 

I also think this is tangential, and I think it would be helpful to isolate these issues in discussion. There are two separate questions here:

1. How the container responds to changed configuration (where it is possible to change, e.g. mounted volumes)

This is the problem with stop and start. It essentially breaks the immutable promise and let's you change the environment variables between stop and start.
 
2. Whether the container can be restarted, with the same configuration

We only want (2), and most of the work in the linked PR has gone into (2), so I'm not sure it should be dismissed on the basis of (1).

I'm not sure if we should do even a step in that direction. The whole stop and start concept breaks container immutability promise. Therefore, I'd prefer to stay away from it. Note, that in most of the cases the same result might be achieved by restarting a container (with killing it). The only problematic environment is Docker Compose. I think the right approach there is to use a proper restart policy: https://docs.docker.com/compose/compose-file/#restart
 

The way I see it, the current behaviour of the start-up scripts is actually contrary to the idea that the container is immutable after creation. Making all the configuration scripts execute only once would support start/stop without introducing any issues with regard to changing configuration. Unless I misunderstand something, I think this would be an uncontroversial change and would solve (2). What do you think?

The other point is about configuration by mounted volumes, e.g. certs; I think you're right that changes to these should be picked up. Is it possible to make these scripts, which copy/reformat the mounted files, execute every time? That would solve (1). But again, I think this could be broken out into a separate issue.

That's yet another argument to not support stop and start.
 

--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.

Artur Z

unread,
Aug 26, 2020, 10:47:24 PM8/26/20
to Keycloak Dev

Hello, I would very much support the idea of just allowing stop/start of the container possible.

Perhaps in the discussion here we try to handle to many edge cases. 
I bet that 99.9% use case for this feature would be just to stop/start the container with assumption that no other environmental change happened.

This ability is very useful in various docker orchestrators that are already out there in use.
For example: Rancher 1.6 where start/stop is supported, orchestrator can perform that for the whole batches of containerised services.
This helps eg during maintenance windows, where we can selectively disable certain services.

Currently it is a quite a nasty surprise seeing that Keycloak container is unable to come back up, along with all the other containers.

Thank you,
Artur Znamierowski

Sebastian Łaskawiec

unread,
Aug 27, 2020, 2:41:35 AM8/27/20
to Artur Z, Keycloak Dev
On Thu, 27 Aug 2020 at 04:47, Artur Z <azna...@gmail.com> wrote:

Hello, I would very much support the idea of just allowing stop/start of the container possible.

Perhaps in the discussion here we try to handle to many edge cases. 
I bet that 99.9% use case for this feature would be just to stop/start the container with assumption that no other environmental change happened.

This ability is very useful in various docker orchestrators that are already out there in use.
For example: Rancher 1.6 where start/stop is supported, orchestrator can perform that for the whole batches of containerised services.
This helps eg during maintenance windows, where we can selectively disable certain services.

In vanilla Kubernetes, upgrading a node (which I assume is one of the main reasons for scheduling a maintenance window) requires draining [1] it (asking Kubernetes to remove all the running Pods out of it and schedule them somewhere else).

Rancher seems to handle stop and start in a special way [2]:
>  [...] The only things you can edit are things that we store that aren’t really part of the Docker container. This includes restarting, it’s still the same container if you stop and start it [...]
As you pointed out, it seems to prevent you from changing the environment. But still, if you have access to Docker Daemon, you can do whatever you like - it's just a stopped container. So I'm still not convinced it's worth the risk.

I wonder, what does the stop and start buy you? Why do you think it's better than just to restart a container? In Keycloak's case the stop and start goes through the same bootstrap sequence and even if patch it up, it will take the same amount of time (basically Wildfly bootstrap time plus deployment time). 

On a side note, I believe Rancher should let you define restart policy somehow (I haven't used Rancher, I'm just guessing that's an option every container orchestrator should have). Perhaps you should let Keycloak start (from the stopped state) and let it crash. Once Rancher restarts it, it will boot up. It might seem scary but container orchestrators are designed to keep containers alive, so handling crashes is part of their job.


Stian Thorgersen

unread,
Aug 27, 2020, 5:57:04 AM8/27/20
to Artur Z, Keycloak Dev
I think you are correct here. Allowing a stop/start assuming that it should just "continue" from where it was stopped should be relatively easy. We can just add a file/marker that init has already been taken care of so just skip all the config/init parts and run the startup script. Then we can consider specific cases on demand later if needed.

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/d4d8cb2f-c07d-4fb9-8320-f666ab8faadbn%40googlegroups.com.

Tom Mewett

unread,
Aug 27, 2020, 6:53:29 AM8/27/20
to Keycloak Dev
On Thursday, 27 August 2020 at 07:41:35 UTC+1 sebastian...@gmail.com wrote:

As you pointed out, it seems to prevent you from changing the environment. But still, if you have access to Docker Daemon, you can do whatever you like - it's just a stopped container. So I'm still not convinced it's worth the risk.

Could you quickly point me to how and to what extent it's possible to modify a stopped container? I looked into environment variables and couldn't see how it was possible. Changing the contents of a volume is possible, but as they're bind mounts I think that's possible during the running of the container anyway? (I'm not particularly an adept Docker user :)

I'm not completely sure that my original email was fully understood. Even if it is possible to change the environment variables of a stopped container, we're not asking for those changes to do anything at all. I don't understand how allowing the container to be merely stopped and started, without respecting any configuration change, is an argument against the mutability of the container or its configuration?

I hope I'm not missing the point too badly.

Thanks,
Tom

Sebastian Łaskawiec

unread,
Aug 28, 2020, 6:27:00 AM8/28/20
to Tom Mewett, Keycloak Dev
On Thu, 27 Aug 2020 at 12:53, Tom Mewett <tom.m...@codethink.co.uk> wrote:
On Thursday, 27 August 2020 at 07:41:35 UTC+1 sebastian...@gmail.com wrote:

As you pointed out, it seems to prevent you from changing the environment. But still, if you have access to Docker Daemon, you can do whatever you like - it's just a stopped container. So I'm still not convinced it's worth the risk.

Could you quickly point me to how and to what extent it's possible to modify a stopped container? I looked into environment variables and couldn't see how it was possible. Changing the contents of a volume is possible, but as they're bind mounts I think that's possible during the running of the container anyway? (I'm not particularly an adept Docker user :)

Yes, I was referring to the situation you described. You stop a container, modify its volume (e.g. change the TLS key/cert) and start it again. The question arises - should it pick up the new cert? If yes, what if its password has changed?

But you have a good point - if you have access to a Docker Daemon, you can do anything to the container. No matter if it's stopped or not.
 

I'm not completely sure that my original email was fully understood. Even if it is possible to change the environment variables of a stopped container, we're not asking for those changes to do anything at all. I don't understand how allowing the container to be merely stopped and started, without respecting any configuration change, is an argument against the mutability of the container or its configuration?

As I see there's a lot of interest in this feature, I guess we could get it in. However, we'd need a few assumptions:
- We'd need to change the text I wrote a few days ago [1] and make it crystal clear - if there were any changes made to the container between stop and start, those changes will not be picked up. In other words - no configuration changes allowed between stop and start.
- I agree that we should probably add a flag indicating that Keycloak has already been bootstrapped. If this flag (or a marker file) is present, we'd skip all configuration [2].

If you guys agree to this plan, I'd need a volunteer to create a JIRA for this and contribute the implementation.

 

I hope I'm not missing the point too badly.

Thanks,
Tom

--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages