replay persisted events upon startup

[o8ixi8o]

I would like to replay some events into Riemann, early in its startup, before network servers come online and start accepting events from clients. This is part of an attempt at persistence across restarts.

I have a function that tries to do this by calling reinject. I call the function late in my riemann.config, like (replay reinject): replay repeatedly calls reinject with my initial-state events. But it seems to have no effect -- for example, if I add #(info %) to my streams to log all events, I don't see these replayed events.

But if I call it as (future (Thread/sleep 5000) (replay reinject)), it works: The events flow throw my streams, are logged, etc. I guess this is  because Riemann is, 5 seconds later, in a state to receive and process events.

But I don't want to rely on guessed delays and, anyway, my tcp-server is by this time  online and potentially accepting events from clients.

Assuming I can find a way to check that Riemann is online, I can imagine more complicated mechanisms to poll-wait in a future (or similar) until then, and then replay events, and then set up tcp-server. But this seems overly complicated, so I'd rather not.

Any ideas on how I could replay selected events via riemann.config and have them actually take effect, to establish the initial state? I tried calling (start!) just before replay, to no avail: I see the Hyperspace core online message but, again, my reinjected events seem to have no effect.

Thanks in advance for any ideas!

paul

[Sanel Zukan]

o8ixi8o <o8i...@gmail.com> writes:
> But if I call it as (future (Thread/sleep 5000) (replay reinject)), it
> works: The events flow throw my streams, are logged, etc. I guess this is
> because Riemann is, 5 seconds later, in a state to receive and process
> events.

This is because Riemann core is fully initialized after config is read [1].

> But I don't want to rely on guessed delays and, anyway, my tcp-server is by
> this time online and potentially accepting events from clients.
>

> Assuming I can find a way to *check* that Riemann is online, I can imagine

> more complicated mechanisms to poll-wait in a future (or similar) until

> then, and *then* replay events, and *then* set up tcp-server. But this

> seems overly complicated, so I'd rather not.
>
> Any ideas on how I could replay selected events via riemann.config and have
> them actually take effect, to establish the initial state? I tried calling
> (start!) just before replay, to no avail: I see the Hyperspace core online
> message but, again, my reinjected events seem to have no effect.

AFAIK, Riemann service lifecycle does not support dependency between
services (e.g. so you can say: "let B wait until A is fully
done"). Also, ordering services in riemann.config does not mean they
will be initialized in that order - they are started using "pmap", which
means they will be run in parallel, but the core will not be marked
as initialized until all of them are running (Riemann will
not start passing events through streams until the core is initialized).

What I've noticed is that running tcp-server *after* the core is
initialized (e.g. in a separate thread) does not work reliably well.

What you can do (and this probably is not the most optimal approach) is
to create own tcp-server with a custom handler that will start accepting &
processing remote events after it finished loading replayed
events. Check tcp.clj and (tcp-handler). Also, for this to work,
(start!) implementation must replay events and run tcp handler it in the
background, so Riemann can mark core as initialized.

There might be a better/simpler solution out there...

> Thanks in advance for any ideas!
>
> paul

Best,
Sanel

[1] https://github.com/riemann/riemann/blob/main/src/riemann/bin.clj#L132

[o8ixi8o]

Sanel,

Thanks for the insight and the good idea about providing a customized implementation of tcp-server -- I'll look into that.

paul