Detecting when the server is up after a reboot

[Szymon Juraszczyk]

Hi

I have a use case where I need to wait for the server to come up after a
reboot. I have noticed that after the minion starts, some useful events
appear on the event bus, with tags 'minion_start',
'salt/minion/<minionname>/start' and their variations with
'syndic/<syndicname>/' in front of the tag.

Just what I need, but... In a multi-master/multi-syndic setup those
events appear only on one of the masters in an apparently random
fashion. This is really a blocker for me to make use of those events. I
would like my software which integrates with SaltStack to work
independently on each master for failover reasons. If you create a job
on one master, the result comes back to that one (thanks to master_id),
but obviously those minion_start events have no master_id set for them.

Wouldn't that make more sense to send such events to both masters
instead of a random one?

Or maybe someone has found another reliable way for the master executing
system.reboot on a minion to be notified when the system is up again
after the reboot?

Obviously polling is out of the question as it would create many
unnecessary jobs (depending on the interval), never picked up by that
minion while it is down. Potentially I could also poll every possible
syndic for information about the minions that are up, but that seems
far-fetched too.

--
Regards,
Szymon Juraszczyk, szy...@juraszczyk.com

[Dmitry Golubenko]

В Чтв, 18/08/2016 в 07:26 +0200, Szymon Juraszczyk пишет:

> Hi
>
> I have a use case where I need to wait for the server to come up after a
> reboot. I have noticed that after the minion starts, some useful events
> appear on the event bus, with tags 'minion_start',
> 'salt/minion/<minionname>/start' and their variations with
> 'syndic/<syndicname>/' in front of the tag.
>
> Just what I need, but... In a multi-master/multi-syndic setup those
> events appear only on one of the masters in an apparently random
> fashion. This is really a blocker for me to make use of those events. I
> would like my software which integrates with SaltStack to work
> independently on each master for failover reasons. If you create a job
> on one master, the result comes back to that one (thanks to master_id),
> but obviously those minion_start events have no master_id set for them.

Are you sure events appear on random master, not the one minion
connected? I was sure event appear on the master minion choosen to
connect to (randomly) ... if so, then just watch work with every master
and process events on master minion happen to connect to. imho.

[Szymon Juraszczyk]

On 2016-08-18 at 07:38, Dmitry Golubenko wrote:
> Are you sure events appear on random master, not the one minion
> connected? I was sure event appear on the master minion choosen to
> connect to (randomly) ... if so, then just watch work with every master
> and process events on master minion happen to connect to. imho.

The minions don't connect to the top level masters - they connect to the
syndics (yes, semi-randomly using master_shuffle). But then those
syndics forward those minion_start events to a random master. I believe
that each syndic stays connected to every top-level master, so there
must be some randomness involved in that process of forwarding those
events up the chain.

[Szymon Juraszczyk]

On 2016-08-18 at 07:47, Szymon Juraszczyk wrote:
> The minions don't connect to the top level masters - they connect to the
> syndics (yes, semi-randomly using master_shuffle). But then those
> syndics forward those minion_start events to a random master. I believe
> that each syndic stays connected to every top-level master, so there
> must be some randomness involved in that process of forwarding those
> events up the chain.
>

Sorry for replying to my own post. What I found out after a busy morning
spent on debugging the code is that MultiSyndic._forward_events()
forwards self.raw_events to a first syndic on the list, calling
_call_syndic() without any master_id. I don't think that is a valid
approach for multi-master/multi-syndic setups. Or maybe there is a
legitimate reason for that, which I don't know of?