Sidekicks for service discovery vs systemd ExecStartPost

[Lou Scalpati]

Hi,

I'm new to coreos and am trying to get my arms around the best practices for setting up a set of services that make up an application and getting it running on a cluster.  I have read from the coreos site that service discovery can be accomplished by creating sidekicks (see here).  But after reading the documentation for systemd it seems that you could accomplish the same thing by using the ExecStartPost command to register the container with etcd.  I have included a line from the sample code from the systemd unit documentation (see here)  below.  The example shows: 

 

ExecStartPost=/usr/bin/etcdctl set /domains/example.com/10.10.10.123:8081 running

My question is are the 2 methods equivalent?  I am missing the point as to why you would need to add more complexity to application by also creating sidekicks. 

Thanks in advance.

[Rob Szumski]

Hi Lou,

You’re right, both strategies currently accomplish the same thing.

The problem with running etcdctl in the ExecStartPost is that it’s dumb. It will announce that the container is running as soon as the systemd unit starts. If the container isn’t cached locally and must be pulled, you just announced your app was ready for traffic before it can serve any. While that doesn’t sound ideal, its simplicity can be valuable.

Using a sidekick allows you to manage and version the logic for the service registration separately from the app. This makes sense when you’re running an application that doesn’t speak etcd natively and isn’t going to be modified. The sidekick method allows you to start a service registration container that can run more complicated logic. It could read the docker API can make sure the container is actually up and running before announcing it. It could curl /health and read a status code. It could check connectivity to another dependent container.

If you end up tweaking the strategy or create something cool, please share!

 - Rob

[Lou Scalpati]

Thanks for the reply.  I see the extra functionality that a sidekick can provide.   It makes sense now except...

The documentation states:

Since our container will be started in ExecStart, it makes sense for our etcd command to run as ExecStartPost to ensure that our container is started and functioning.

This would imply that ExecStartPost waits until the container is successfully started before the command is run.  Is this just a mistake in the documentation or a current issue with systemd?  

Thanks again for taking the time to answer!

[Rob Szumski]

You’re right, that documentation could be a little more specific. The intention was to show that it runs after the ExecStart, which will run as long as the docker command doesn’t fail. This doesn’t guarantee anything about the software inside of the container, since it isn’t being tested.

Short term this is a mistake in the documentation. Long term, docker is planning on integrating more closely with systemd so that containers are owned by the unit that started them instead of all being spawned with dockerd.

 - Rob

[Lou Scalpati]

Rob - thanks for all your help!