Oort graceful shutdown

22 views
Skip to first unread message

Roman Zaynetdinov

unread,
Aug 13, 2020, 6:46:21 AM8/13/20
to cometd-users

Hi,

How can we gracefully terminate a node in Oort cloud? Shall the node simply disconnect from all other nodes with oortComet.disconnect() ? We use OortMap and would like the entries there to be cleaned as soon as the owning node is terminated.

When would be the best place to disconnect from other nodes? We run our sevlet application with Jetty distribution and I can't seem to find a way to run clean up code after servlet starts shutting down but before it is destroyed. If I add a listener on destroy lifecycle event I can't disconnect the comet because WS connection is closed even earlier.

There is also a maxInterval setting which could be used to disconnect dead nodes. Is it possible to set different maxInterval setting depending on the client "type". For oort clients we would like to set it to lower value than we set for "normal"/web clients.

Thank you for your time!

Simone Bordet

unread,
Aug 24, 2020, 4:46:48 PM8/24/20
to cometd-users
Roman,

On Thu, Aug 13, 2020 at 12:46 PM Roman Zaynetdinov
<roman.za...@lekane.com> wrote:
>
>
> Hi,

Sorry for the late reply, was in vacation.

> How can we gracefully terminate a node in Oort cloud?

Can you define what "gracefully" means for you?

> Shall the node simply disconnect from all other nodes with oortComet.disconnect() ? We use OortMap and would like the entries there to be cleaned as soon as the owning node is terminated.

It really depends on how you have configured your system.
If the JVM runs only your CometD webapp, undeploying the webapp should
do the job.
This is equivalent to calling Oort.stop(), immediately followed by
BayeuxServer.stop().

Typically this causes all clients connected to that node to
re-handshake with other nodes.

> When would be the best place to disconnect from other nodes? We run our sevlet application with Jetty distribution and I can't seem to find a way to run clean up code after servlet starts shutting down but before it is destroyed. If I add a listener on destroy lifecycle event I can't disconnect the comet because WS connection is closed even earlier.

I'm a bit lost here.
Why is it a problem if the WS connection is already closed?

> There is also a maxInterval setting which could be used to disconnect dead nodes. Is it possible to set different maxInterval setting depending on the client "type". For oort clients we would like to set it to lower value than we set for "normal"/web clients.

This is possible, but I would not necessarily recommend it.
You can tell if a client is an Oort client via
Oort.isOort(ServerSession), and then
ServerSession.setMaxInterval(long).

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.

Roman Zaynetdinov

unread,
Aug 27, 2020, 8:08:22 AM8/27/20
to cometd-users
Thanks for responding!



> Sorry for the late reply, was in vacation.

Absolutely no worries!



> Can you define what "gracefully" means for you?

When shutting down node A I want it to notify node B that it is about to leave the cluster. Then I want node B to remove node A locally and to clear OortMap from node A's entries.
At the moment when I terminate node A, node B will then try to connect to it for the duration of maxInterval and only then remove the entry.

Due to my limited knowledge I was thinking that if I explicitly try to disconnect the comet I would I achieve what I want.

So I guess my real question is how to stop Oort client on node A so that node B will remove the comet immediately from its cluster list?



> I'm a bit lost here.
> Why is it a problem if the WS connection is already closed?

I was planning to notify node B when node A is terminated. Sorry this question was more about JVM and servlets than about cometD.
I fail to figure out how to run code when servlet context is about to terminate but not yet.

Example of trying to call oortComet.disconnect() manually:

W|14:56:27.884|JettyShutdownThread| (OortConfig.java:84) ##### Disconnecting elv8gc8yn5hgo1er9rshqcwln6: http://127.0.0.1:18080/test/cometd
D|14:56:27.885|JettyShutdownThread| (BayeuxClient.java:1511) State updated: UNCONNECTED -> DISCONNECTING
D|14:56:27.887|JettyShutdownThread| (BayeuxClient.java:469) Sending messages [{clientId=elv8gc8yn5hgo1er9rshqcwln6, channel=/meta/disconnect, id=66}]
D|14:56:27.887|JettyShutdownThread| (WebSocketTransport.java:93) Opening websocket session to ws://127.0.0.1:18080/test/cometd
D|14:56:27.960|JettyShutdownThread| (BayeuxClient.java:946) Messages failed [{clientId=elv8gc8yn5hgo1er9rshqcwln6, channel=/meta/disconnect, id=66}]
java.io.IOException: Connect failure
    at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(ClientContainer.java:263) ~[javax-websocket-client-impl-9.4.30.v20200611.jar:9.4.30.v20200611]
  ...
Caused by: java.lang.RuntimeException: org.eclipse.jetty.websocket.jsr356.ClientContainer is not running yet
    at org.eclipse.jetty.websocket.jsr356.DecoderFactory.init(DecoderFactory.java:171) ~[javax-websocket-client-impl-9.4.30.v20200611.jar:9.4.30.v20200611]
  ...

Simone Bordet

unread,
Oct 9, 2020, 11:36:15 AM10/9/20
to cometd-users
Hi,

On Thu, Aug 27, 2020 at 2:08 PM Roman Zaynetdinov
<roman.za...@lekane.com> wrote:
> When shutting down node A I want it to notify node B that it is about to leave the cluster. Then I want node B to remove node A locally and to clear OortMap from node A's entries.
> At the moment when I terminate node A, node B will then try to connect to it for the duration of maxInterval and only then remove the entry.

The right way to do this would be via: oortA.stop().

Stopping OortA will disconnect oortCometAB, will cause oortCometBA to
disconnect, and will fire an event which OortObjects listen to, so
that they can get rid of the part correspondent to OortA.

Give it a go and let know how it goes.
Reply all
Reply to author
Forward
0 new messages