Correct way to shutdown ruote?

[Steve Sexton]

 

Related to my previous message where I ended up with a process that needed a manual reply_to_parent, I wanted to ask, what is the correct way to shutdown ruote?  I suspect that the reason I ended up losing a reply message is because the worker process was killed and restarted at the OS level while ruote was busy processing replies.

 

I know that some database servers have the concept of a quiesce, is there something similar in ruote?  If I pause the workers does that mean that all messages are sitting idle in the storage (which is what I want)?  What about the shutdown method on the engine?

 

Thanks,

  Steve

[Farrel Lifson]

Ruote::Dashboard#shutdown will propagate a shutdown call to any worker or storate it was initialised with.

[John Mettraux]

On Tue, Apr 02, 2013 at 06:59:17AM -0700, Farrel Lifson wrote:
>
> Ruote::Dashboard#shutdown will propagate a shutdown call to any worker or
> storate it was initialised with.

Hello Steve, hello Farrel,

yes, you'd better call shutdown on Engine/Dashboard. It will, as Farrel
mentioned, call the shutdown methods of all the ruote components. When a
ruote worker receives the shutdown call, it flags itself as "stopped" and will
exit its processing loop right after handling the current "msg" or, if there is
none, immediately. The processing thread is joined until it exits.

"msg" documents that come after the worker shut down are kept in the storage,
they'll be processed when a worker picks them up there.


Best regards,

--
John Mettraux - http://lambda.io/jmettraux

[Steve Sexton]

On Tuesday, April 2, 2013 5:27:13 PM UTC-4, John Mettraux wrote:

yes, you'd better call shutdown on Engine/Dashboard. It will, as Farrel
mentioned, call the shutdown methods of all the ruote components.

I had a chance to test this today, and sadly, it does not shutdown "all" ruote components.  It only does a shutdown for components active in the OS process in which the shutdown method was called.  Specifically, if you use the rake task provided with ruote-kit to run a worker in a separate process, it will not receive the shutdown message.  Which means that if shutdown is important, the rake task in ruote-kit should never be used.

 

Looking through the code, it appears that the shutdown methods of services are called directly, via instance variables and local methods, and not by using a broadcast message like I was hoping/expecting. 

 

[John Mettraux]

On Fri, Apr 05, 2013 at 11:31:09AM -0700, Steve Sexton wrote:
>
> On Tuesday, April 2, 2013 5:27:13 PM UTC-4, John Mettraux wrote:
> >
> > yes, you'd better call shutdown on Engine/Dashboard. It will, as Farrel
> > mentioned, call the shutdown methods of all the ruote components.
> >
> I had a chance to test this today, and sadly, it does not shutdown "all"
> ruote components. It only does a shutdown for components active in the OS
> process in which the shutdown method was called.

Hello Steve,

I just re-read the thread. Sorry, I did not guess you wanted to shut down all
the workers in one go.

> Specifically, if you use
> the rake task provided with ruote-kit to run a worker in a separate
> process, it will not receive the shutdown message. Which means that if
> shutdown is important, the rake task in ruote-kit should never be used.

Clearly, it lacks signal/exit handling. That can be taken care of. Any
thing I should be wary of before I proceed?

An at_exit handler could help, but I guess it doesn't prevent a cold kill.

> Looking through the code, it appears that the shutdown methods of services
> are called directly, via instance variables and local methods, and not by
> using a broadcast message like I was hoping/expecting.

Maybe this can help:

https://github.com/jmettraux/ruote/blob/1c8a0741ee5d338787bda6743975ef3eab7704cc/lib/ruote/dashboard.rb#L1051-L1097

I'm not too happy with it though. The state checking loop needs tightening.