Setting restart policy on individual container (not pod)

[yathindra dev]

Hi, 

I'm trying to use authz plugin for docker on kubernetes. The authz plugin is started inside a docker container. The latest version of docker requires a restart of docker daemon in order to talk to authz plugin. Once docker daemon restarts, it restarts all the containers with restartpolicy set on them. As a result, the authz plugin running in the container comes up and docker daemon can talk to the plugin. 

When I tried this on kubernetes, docker is not able to talk to authz plugin since I'm not able to set restartpolicy on a container.

Question is does kubernetes allow setting restart policy on container?

[Eric Tune]

Are you running the authz plugin in a Pod, or running it directly via Docker?  You might need to do the latter in this case.

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/d9812a2f-c07b-4094-a0a0-304d75623a94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[yathindra dev]

Does kubernetes have any mechanism (any API's) to launch containers directly via Docker?

I'm running the authz plugin inside a pod. When I restart docker daemon, it fails to connect to the plugin. At this point I think it's because the pods didn't get restarted as part of docker restart.

It works fine on single host non-kubernetes environment (Ubuntu 15.10).

[Tim Hockin]

I don't follow. After docker restarts, is your plugin container
running? It should be. We have pod-level restart policy which
applies to all containers.

> https://groups.google.com/d/msgid/kubernetes-dev/CADz9--jDN%2BQFPwL1%2B3DPSiRpeWXxYX_me-m_ohVivpBAhja9Ng%40mail.gmail.com.

[yathindra dev]

After docker restarts, it tries to do a handshake with the plugin socket. The problem seems to be that the container running the plugin does not come up and as a result docker times out trying to connect to the plugin socket.

Here's some of the logs after I restart docker,

Jul  8 22:08:19 ip-172-31-0-50 systemd[1]: Starting Docker Socket for the API.

Jul  8 22:08:19 ip-172-31-0-50 systemd[1]: Listening on Docker Socket for the API.

Jul  8 22:08:19 ip-172-31-0-50 systemd[1]: Starting Docker Application Container Engine...

....

....

Jul  8 22:08:20 ip-172-31-0-50 docker[22169]: time="2016-07-08T22:08:20.981869238Z" level=error msg="Failed to load container 02570582f552f331bd520c9f5f57134b6233133404f40f68a03d7a5259cd9f7a: open /var/lib/docker/containers/02570582f552f331bd520c9f5f57134b6233133404f40f68a03d7a5259cd9f7a/config.v2.json: no such file or directory"

Jul  8 22:08:21 ip-172-31-0-50 docker[22169]: time="2016-07-08T22:08:21.021619795Z" level=warning msg="error locating sandbox id 3be90f2b547bdc14f6d37630cb34a6ad468e6e4c70061349418aaed3307e8747: sandbox 3be90f2b547bdc14f6d37630cb34a6ad468e6e4c70061349418aaed3307e8747 not found"

Jul  8 22:08:21 ip-172-31-0-50 docker[22169]: time="2016-07-08T22:08:21.022369397Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/ec926bd9e7475b743a14885d5c55d521710ed247e17f5e58cbb0917524506a11/shm: invalid argument"

Jul  8 22:08:21 ip-172-31-0-50 docker[22169]: time="2016-07-08T22:08:21.022747615Z" level=warning msg="error locating sandbox id b168db14a7d47acd0e79be80d150ca717d4665261562fe3722b1ab368cfc4d03: sandbox b168db14a7d47acd0e79be80d150ca717d4665261562fe3722b1ab368cfc4d03 not found"

....

....

Jul  8 22:08:21 ip-172-31-0-50 systemd[1]: Started Docker Application Container Engine.

Jul  8 22:08:21 ip-172-31-0-50 docker[22169]: time="2016-07-08T22:08:21.046035463Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/auth.sock:/Plugin.Activate, retrying in 1s"

Jul  8 22:08:21 ip-172-31-0-50 docker[22169]: time="2016-07-08T22:08:21.046041273Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/auth.sock:/Plugin.Activate, retrying in 1s"

Jul  8 22:08:21 ip-172-31-0-50 kernel: [ 8585.597603] cbr0: port 2(vethd4be99f) entered disabled state

Jul  8 22:08:21 ip-172-31-0-50 kernel: [ 8585.603778] device vethd4be99f left promiscuous mode

Jul  8 22:08:21 ip-172-31-0-50 kernel: [ 8585.605863] cbr0: port 2(vethd4be99f) entered disabled state

Jul  8 22:08:22 ip-172-31-0-50 docker[22169]: time="2016-07-08T22:08:22.046390488Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/auth.sock:/Plugin.Activate, retrying in 2s"

Jul  8 22:08:22 ip-172-31-0-50 docker[22169]: time="2016-07-08T22:08:22.046390733Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/auth.sock:/Plugin.Activate, retrying in 2s"

.....

.....

[Tim Hockin]

Can you confirm whether your plugin is or is not running after docker
restarts? Did you set the restart policy to something other than
Always (which is the default)?

I don't really know how Docker's API regsitration works, but it used
to be "drop a UNIX socket in this directory" - is that still true?

[yathindra dev]

When I run the plugin inside a pod, after restart, the plugin did not start. I did not change the restart policy on the pods (its default).

Docker still uses a "drop a UNIX socket in this directory" method. 

[yathindra dev]

I performed this experiment and it works but not ideal. 

1. Ran the plugin container using docker manually and set the restartpolicy to always.

2. Restarted docker 

3. Now, docker is able to connect to the plugin container.

[Tim Hockin]

Can you boil this down to a reproducible case? Restarting docker
should ALWAYS result in your pod containers coming back, and we do
have end-to-end tests for this,..

[yathindra dev]

If I don't specify "--authorization-plugin=<plugin>" with docker options then restarting docker does result in pod containers coming back. The problem happens only when I restart docker with "--authorization-plugin=<plugin>".

Is there a ordering on which pod gets restarted first?

[Tim Hockin]

there is no ordering. Why is that container failing to start? I
assume kubelet is retrying it periodically. Does `docker ps -a` show
it? What are its logs?

[Yifan Gu]

How did you restart your docker daemon? Did you sigkill or sigterm it?

If you restart the daemon gracefully, it will stop the plugin container
first, right? Then when the daemon restarts, it cannot find the auth
plugin container and get stuck.

Is that the case?

- Yifan

[yathindra dev]

Yifan, Even with an unclean shutdown of docker daemon I see the same result.

Tim,

This is how authz plugin works on docker: https://docs.docker.com/engine/extend/plugins_authorization/

As you mentioned, docker uses a drop-in file to override docker options to enable authz plugin: https://docs.docker.com/engine/admin/systemd/

Is this supported in kubernetes?

I see there is a special way kubernetes handles network plugins: http://kubernetes.io/docs/admin/network-plugins/#network-plugin-requirements

You received this message because you are subscribed to a topic in the Google Groups "kubernetes-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-dev/Hqy3TmJaULw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/57804008.5090102%40coreos.com.

[Tim Hockin]

I haven't heard anything should not work.  He suspicious part is where you say the container er did not restart.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CADz9--gYv8uwLognRWSrygjMwTnMq4Q2VSJeZZDXFfjZpLc8WA%40mail.gmail.com.

[yathindra dev]

That's correct. As I mentioned before, when I manually start the authz plugin container (with restart policy set to always) using docker and restart docker (with plugin option), docker manages to restart the container and connect to the plugin.

When I run the same container using pods (pods have restartpolicy set to always by default) and restart docker (with plugin option), it is not able to connect with the authz plugin. This leads to believe that the pod containing plugin is somehow not getting started and therefore docker engine is not able to connect with it. 

Please note that when I restart docker (without plugin option), the pods are restarted successfully. It is only in the case when docker is restarted with plugin option that pods are not getting restarted as expected.

In order to restart all the pods successfully, does docker need to start kubelet? Is there any such dependencies?

[Tim Hockin]

I;'m asking you to verify and investigate. You said "This leads to
believe that the pod ... is somehow not getting started". Don't
believe, please - prove. Can you see it in docker ps? docker ps -a?
Does it have logs when it fails to start?

There's only so much debug I can do for you over email. I didn't hear
anything that should not work, so something is not working as
intended. Please help us figure out what.

Docker does not (usually) run kubelet.