Custom notifications to systemd from within a docker container

[James Sharp]

Hi guys

I've been looking at ways of notifying systemd when a container is ready (e.g. when a web server is all configured and listening) so that it doesn't register as 'started' until this point. It seems like the Type=notify directive is exactly what I need (in conjunction with sd_notify - docs here), however I can't work out a way of getting hold of the notification socket within the container (its an abstract namespace socket, not an fs one so I don't think a bind/mount will work).

So a couple of questions…

1. Does this sound like a good method or should I be looking at some other mechanism?

2. Is there any way of getting hold of the socket from within the container?

Thanks in advance for any help. And great work with CoreOS!

James

[Alex Polvi]

James, using Type=notify is exactly the right way to do this. Ideally docker would give us this after it has opened the port and started the service. systemd-nspawn handles this correctly, so you can use that directly as well.

In your ideal world, what would this look like? Maybe we can find a way to build it.

-Alex

[Darren Shepherd]

To my understanding, nspawn will call sd_notify right before it clones the process.  I think what is needed is a way for the running application in the container to be able to call sd_notify themselves so that the application can determine if the execution was good or bad.

It seems to me that docker should provide its own notification mechanism to be portable and not systemd specific.  It would be nice to be able to call /.dockerinit --notify READY=1 and then that would somehow call sd_notify if it is running under systemd.  Not sure of the technical detail of how that would work, but it does seem like a good feature.

Now if you looking for a hack, systemd_notify/sd_notify uses the NOTIFY_SOCKET env varible to find the socket.  I've never tried this, but if there's a way to link an abstract namespace socket to a file you could set NOTIFY_SOCKET to the path of that socket.  Or maybe you can setup your own socket and process that reads the socket that will call sd_notify for you.  That could be tricky, because I assume the caller of sd_notify needs to be in the same cgroup as the process being started.

Darren

[James Sharp]

Darren has nailed it really. For now I'd be happy enough just calling sd_notify from the running application inside the container if this could be made to work. In an ideal world I think a generic notification mechanism for docker would be great (whether this would solely convey READY/NOT READY state or something more complex, or something completely arbitrary I don't know). Since a large number of containers are likely to have non-zero startup time this seems like a very useful feature. If, as Darren says, this could seamlessly get translated to sd_notify when appropriate then all the better.

In terms of hacking around I haven't been able to find a way to map the abstract ns socket onto a file. Socat seemed like a good option for creating a new (file) socket and forwarding anything onto the abstract one, but unfortunately socat isn't available on coreos so I'm out of ideas really.

James

[Alex Polvi]

We had a PR up to add socat, so I think it should be in the last release. Will check on that and follow up.

-Alex

[Darren S]

I don't think socat made it in the latest release. nmap netcap did, which has -U for unix sockets, but I highly doubt abstract namespace sockets work.  But I could be wrong.

Hey, maybe you can hack up nspipe to support sockets.

Darren

[Anand Patil]

Hi everyone, I just wanted to check if there are any updates on this.

If not, is this right? To send systemd ready notifications from containers currently, I have to:

1. Build a static binary of socat and copy it to the CoreOS machine

2. Create a regular Unix domain socket

3. Use socat to forward from the Unix domain socket to systemd's abstract namespace socket

4. Bind-mount the Unix domain socket into the container

5. Inside the container, do "NOTIFY_SOCKET=$unix-socket-path-in-container systemd-notify --ready" when ready

Assuming that's right, my 2c is that if you of CoreOS could bundle socat and install a script that does 2-3, so my job becomes pretty easy:

unixSocketPath=`coreos-make-regular-notify-socket`

docker run ... -v $unixSocketPath:/var/run/notify.sock ...

NOTIFY_SOCKET=$unix-socket-path-in-container systemd-notify --ready

A generic notification system in Docker would be cool because it would make containers more portable across init systems, but I'd rather not wait for it.

Thanks,

Anand

[Anand Patil]

In case anyone else is looking for a workaround, it is possible to get away without socat. In the container, you can do 

echo "$token" | nc -U $socket

and in CoreOS, you can run something like the following script in the background, in the same service where you're running Docker:

#!/bin/bash

sock=$1

token=$1

received=`ncat -U -l $socket`

if [ $received=$token ]; then

  echo $received

  echo $token

  echo "Ready notification received"

  systemd-notify --ready

  exit 0

else

  echo "$received"

  exit 1

fi

[James Sharp]

Cool - thanks Anand

The workaround I've come up with (which is a bit specific to my setup) is to run a separate unit file which curls /health on my webservice container. Once it receives a 200 repsonse it then calls `systemd-notify --ready`. Once this discovery unit is running I know the original is running so all is OK.

The discovery unit also handles registering the web server in etcd - if anyone's interested the script that does the business is here: https://gist.github.com/jamessharp/0feeca62e2cddecb4da6

[Darren S]

Is systemd-notify working for anyone?  It seems to be that it's completely broken and unreliable.  You get errors like "Cannot find unit for notify message of PID X".  This thread seems to indicate that its broken http://comments.gmane.org/gmane.comp.sysutils.systemd.devel/18649

Darren