Help with hot-restart with running envoy within docker container

[manik....@gmail.com]

Hi all, 

We have just started evaluating envoy and were trying to test the hot-restart functionality as per : https://www.envoyproxy.io/docs/envoy/latest/operations/hot_restarter. 

Our start_envoy.sh looks like this :

#!/bin/bash

echo "STARTING CONTAINER"

docker run -i --ipc="host" --net="host" envoy.v1 /usr/local/bin/envoy -c /etc/envoy.yaml --restart-epoch $RESTART_EPOCH --drain-time-s 2 --v2-config-only

We are using the hot_restarter.py script as is to launch the envoy process 

>>python2.7 hot-restarter.py start_envoy.sh

However, when send a SIGHUP to the hot-restart process, we see that the newly launched envoy process dies with the following messages :

[2018-04-02 11:07:35.249][1][critical][assert] source/server/hot_restart_impl.cc:57] panic: cannot open shared memory region /envoy_shared_memory_0 check user permissions

[2018-04-02 11:07:35.250][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Aborted, suspect faulting address 0x1

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:90] Backtrace obj</lib/x86_64-linux-gnu/libc.so.6> thr<0> (use tools/stack_decode.py):

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #0 0x7ff5dc958428

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #1 0x7ff5dc95a029

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</usr/local/bin/envoy>

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #2 0x52b234

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #3 0x52b6a1

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #4 0x46450f

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #5 0x46478f

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #6 0x4156bb

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</lib/x86_64-linux-gnu/libc.so.6>

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #7 0x7ff5dc94382f

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</usr/local/bin/envoy>

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #8 0x45a868

[2018-04-02 11:07:35.251][1][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0

We are running this on CentOS 6.   Appreciate if someone can help with this issue.

Cheers,

Manik

[fay...@gmail.com]

I had the same problem. You should mount host's dev directory(-v /dev:/dev).

try this:

#!/bin/bash

echo "STARTING CONTAINER"

docker run -i --ipc="host" --net="host" -v /dev:/dev envoy.v1 /usr/local/bin/envoy -c /etc/envoy.yaml --restart-epoch $RESTART_EPOCH --drain-time-s 2 --v2-config-only