How to shutdown crossbar node when container worker exits

[Scott Wittenburg]

Hi All,

   I want my Javascript clients to be able to shutdown the server side process via an rpc method.  This is working fine when I start my ApplicationSession class and router manually, as I have implemented an rpc method that calls reactor.stop().  However, something is different when crossbar.io starts my application session.  I see some "unhandled error" in the debug output, and then the crossbar-controller process is still running, though the crossbar-worker has ended.  Here is the debug output from a session where the client calls an rpc method which triggers reactor.stop() on the server side:

scott@solaris:~/Documents/crossbar-tests/pvtest$ crossbar start

2014-09-02 11:11:15-0600 [Controller  12190] Log opened.

2014-09-02 11:11:15-0600 [Controller  12190] ============================== Crossbar.io ==============================

2014-09-02 11:11:15-0600 [Controller  12190] Crossbar.io 0.9.7-5 starting

2014-09-02 11:11:15-0600 [Controller  12190] Running on CPython using EPollReactor reactor

2014-09-02 11:11:15-0600 [Controller  12190] Starting from node directory /home/scott/Documents/crossbar-tests/pvtest/.crossbar

2014-09-02 11:11:15-0600 [Controller  12190] Starting from local configuration '/home/scott/Documents/crossbar-tests/pvtest/.crossbar/config.json'

2014-09-02 11:11:15-0600 [Controller  12190] No WAMPlets detected in enviroment.

2014-09-02 11:11:15-0600 [Controller  12190] Starting Router with ID 'worker1' ..

2014-09-02 11:11:15-0600 [Router      12199] Log opened.

2014-09-02 11:11:16-0600 [Router      12199] Running under CPython using EPollReactor reactor

2014-09-02 11:11:16-0600 [Router      12199] Entering event loop ..

2014-09-02 11:11:16-0600 [Controller  12190] Router with ID 'worker1' and PID 12199 started

2014-09-02 11:11:16-0600 [Controller  12190] Router 'worker1': PYTHONPATH extended

2014-09-02 11:11:16-0600 [Controller  12190] Router 'worker1': realm 'realm1' started

2014-09-02 11:11:16-0600 [Controller  12190] Router 'worker1': role 'role1' started on realm 'realm1'

2014-09-02 11:11:19-0600 [Router      12199] Inside _VisualizerServer constructor, config:

2014-09-02 11:11:19-0600 [Router      12199] ComponentConfig(realm = vtkweb, extra = None)

2014-09-02 11:11:19-0600 [Controller  12190] Router 'worker1': component 'component1' started

2014-09-02 11:11:19-0600 [Router      12199] Site starting on 8080

2014-09-02 11:11:19-0600 [Controller  12190] Router 'worker1': transport 'transport1' started

2014-09-02 11:17:41-0600 [Router      12199] Connection to node controller lost.

2014-09-02 11:17:41-0600 [Router      12199] Unhandled Error

2014-09-02 11:17:41-0600 [Router      12199] Traceback (most recent call last):

2014-09-02 11:17:41-0600 [Router      12199] File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext

2014-09-02 11:17:41-0600 [Router      12199] return self.currentContext().callWithContext(ctx, func, *args, **kw)

2014-09-02 11:17:41-0600 [Router      12199] File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext

2014-09-02 11:17:41-0600 [Router      12199] return func(*args,**kw)

2014-09-02 11:17:41-0600 [Router      12199] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/process.py", line 216, in connectionLost

2014-09-02 11:17:41-0600 [Router      12199] self.proc.childConnectionLost(self.name, reason)

2014-09-02 11:17:41-0600 [Router      12199] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_posixstdio.py", line 90, in childConnectionLost

2014-09-02 11:17:41-0600 [Router      12199] self.connectionLost(reason)

2014-09-02 11:17:41-0600 [Router      12199] --- <exception caught here> ---

2014-09-02 11:17:41-0600 [Router      12199] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_posixstdio.py", line 109, in connectionLost

2014-09-02 11:17:41-0600 [Router      12199] protocol.connectionLost(reason)

2014-09-02 11:17:41-0600 [Router      12199] File "/usr/local/lib/python2.7/dist-packages/crossbar/worker/process.py", line 146, in connectionLost

2014-09-02 11:17:41-0600 [Router      12199] sys.exit(1)

2014-09-02 11:17:41-0600 [Router      12199] exceptions.SystemExit: 1

2014-09-02 11:17:41-0600 [Router      12199] (TCP Port 8080 Closed)

2014-09-02 11:17:41-0600 [Router      12199] Main loop terminated.

2014-09-02 11:17:42-0600 [Controller  12190] Worker 12199: Process connection gone (Connection was closed cleanly.)

 

So my question is, what is the right way to shut down everything on the server side cleanly, in response to a specific rpc method?  In certain use cases, I would like to like to ensure that the crossbar controller ends as well as the worker.

Thanks!

Scott

[Tobias Oberstein]

Hi Scott,

> So my question is, what is the right way to shut down everything on the
> server side cleanly, in response to a specific rpc method? In certain

If you do reactor.stop() in a container, it doesn't have a chance to
orderly shutdown.

Could you try the following?

class MyComponent(ApplicationSession):

@wamp.register(u"com.myapp.shutdown_component")
def shutdown(self):
self.leave()

def onLeave(self, details):
self.disconnect()

The callbacks and methods are described here:

http://autobahn.ws/python/reference/autobahn.wamp.html#autobahn.wamp.interfaces.ISession

==

You could try that in a container hosting your component. Plus try it
when the component runs side-by-side in a router.

At least, this is how "ordered component shutdown" is supposed to work ..

Cheers,
/Tobias

[Scott Wittenburg]

Hi Tobias,

   Thanks for the pointer to that documentation, that's just what I was looking for.  I'll give your suggestion a try in both the use cases you mentioned, thanks!

Cheers,

Scott

[Tobias Oberstein]

Am 02.09.2014 20:22, schrieb Scott Wittenburg:
> Hi Tobias,
>
> Thanks for the pointer to that documentation, that's just what I was
> looking for. I'll give your suggestion a try in both the use cases you

Yes, please. Let me know if it doesn't work as advertised;)

There is one thing I forgot to mention: while a _component_ can be shut
down gracefully as below, there is no way you can shut down the _worker_
itself (e.g. the container process hosting the component) from _within_
the application component.

The idea is to keep the app component agnostic to and isolated from its
hosting environment. Well, of course in Python, that's not really
enforceable. But that's the idea.

In fact, for a container, even when all components have gone away, it's
still there. A container can run without any components. Since: Crossbar
can dynamically start new components in an already running container.

Regarding the node controller: in this case, since it is running no
application code at all, there is technically no way (even if you "break
rules") to shut it down from app code. That would be a security issue.

Full dynamic, programmatic control of Crossbar is possible via the
management API: http://crossbar.io/docs/Management-API/

The management API is fully separate from any application stuff. In
fact, Crossbar internally runs a router dedicated to management of the
node itself. Well, there would be a lot more to say, but I leave it at
that for now;)

> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Scott Wittenburg]

Hi Tobias,

On Tuesday, September 2, 2014 12:59:50 PM UTC-6, Tobias Oberstein wrote:

Am 02.09.2014 20:22, schrieb Scott Wittenburg:
> Hi Tobias,
>
>     Thanks for the pointer to that documentation, that's just what I was
> looking for.  I'll give your suggestion a try in both the use cases you

Yes, please. Let me know if it doesn't work as advertised;)

There is one thing I forgot to mention: while a _component_ can be shut
down gracefully as below, there is no way you can shut down the _worker_
itself (e.g. the container process hosting the component) from _within_
the application component.

The idea is to keep the app component agnostic to and isolated from its
hosting environment. Well, of course in Python, that's not really
enforceable. But that's the idea.

In fact, for a container, even when all components have gone away, it's
still there. A container can run without any components. Since: Crossbar
can dynamically start new components in an already running container.

Regarding the node controller: in this case, since it is running no
application code at all, there is technically no way (even if you "break
rules") to shut it down from app code. That would be a security issue.

Full dynamic, programmatic control of Crossbar is possible via the
management API: http://crossbar.io/docs/Management-API/

Ok, thanks for making this distinction clear.  So I'll look into how we can restart our application component within the same running container then, so that we don't come up with some solution that keeps the number of container processes on the system forever growing.

Cheers,

Scott

 

> <mailto:autobahnws+unsub...@googlegroups.com>.

[Scott Wittenburg]

Hi Tobias,

   Thanks for explaining more about the lifecycle of node vs. container vs component.  Now I'm wondering if there's a way that I could keep my application component (ApplicationSession) running like it was before (not controlled by Crossbar.io), while the router is running under crossbar.  This would be like the separate crossbar nodes deployment approach, except the application component would not run in a crossbar node.

   In our previous "home brew" autobahn python server approach, we manually created a RouterFactory, a RouterSessionFactory, and a transport factory (WampWebSocketServerFactory), along with our ApplicationSession class.  So which of those would we need to do manually if we we hope to get our ApplicationSession to bind to the router running within a crossbar node?

   The problem I'm trying to avoid is keeping long-running processes going on systems where you are charged by the minute, like on a supercomputer.  There we would like all processes related to the application component to be shut down cleanly when the user is done with the session.

   Hope this is clear, let me know if I should explain anything in more detail.

Thanks!

Scott

[Tobias Oberstein]

Hi Scott,

> Thanks for explaining more about the lifecycle of node vs. container
> vs component. Now I'm wondering if there's a way that I could keep my
> application component (ApplicationSession) running like it was before
> (not controlled by Crossbar.io), while the router is running under
> crossbar. This would be like the separate crossbar nodes deployment
> approach, except the application component would not run in a crossbar node.
>
> In our previous "home brew" autobahn python server approach, we
> manually created a RouterFactory, a RouterSessionFactory, and a
> transport factory (WampWebSocketServerFactory), along with our
> ApplicationSession class. So which of those would we need to do
> manually if we we hope to get our ApplicationSession to bind to the
> router running within a crossbar node?

No worry: you don't have to host your app component in a Crossbar node.

Recent Autobahn's have made it much simpler to fire up a "standalone"
component against a router (e.g. Crossbar). That's just 3 lines of code
in your __main__:

http://autobahn.ws/python/wamp/programming.html#running-components

In that case, you can shut down the whole OS process

class MyComponent(ApplicationSession):
def onDisconnect(self):
reactor.stop()

However, this points to a problem: if you would now take MyComponent to
be hosted again under Crossbar, you run into the issues discussed
before. What I'd like to really achieve is this:

You write MyComponent. Then, without touching a single line in
MyComponent, you decide how to deploy MyComponent:

a) "standalone" (e.g. using ApplicationRunner)
b) hosted in a Crossbar container
c) hosted side-by-side in a Crossbar router

>
> The problem I'm trying to avoid is keeping long-running processes
> going on systems where you are charged by the minute, like on a
> supercomputer. There we would like all processes related to the
> application component to be shut down cleanly when the user is done with
> the session.

I'm not sure I get this completely: could you expand a little "when the
user is done with the session" (or expand in general). I'd really like
to achieve a)-c) above. Which doesn't limit you from doing a) anyway.

However, I need to get some sleep;) 6 hours left, need to get up and
travel=(

cu
/Tobias

> > <mailto:autobahnws+...@googlegroups.com>.

> > To post to this group, send email to autob...@googlegroups.com
> > <mailto:autob...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com

> <mailto:autobahnws+...@googlegroups.com>.

> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/eb269db0-9487-4549-9f0a-5dbe8ea2536b%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/eb269db0-9487-4549-9f0a-5dbe8ea2536b%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Scott Wittenburg]

Hi Tobias,

On Tuesday, September 2, 2014 3:17:10 PM UTC-6, Tobias Oberstein wrote:

Hi Scott,

 >     Thanks for explaining more about the lifecycle of node vs. container
> vs component.  Now I'm wondering if there's a way that I could keep my
> application component (ApplicationSession) running like it was before
> (not controlled by Crossbar.io), while the router is running under
> crossbar.  This would be like the separate crossbar nodes deployment
> approach, except the application component would not run in a crossbar node.
>
>     In our previous "home brew" autobahn python server approach, we
> manually created a RouterFactory, a RouterSessionFactory, and a
> transport factory (WampWebSocketServerFactory), along with our
> ApplicationSession class.  So which of those would we need to do
> manually if we we hope to get our ApplicationSession to bind to the
> router running within a crossbar node?

No worry: you don't have to host your app component in a Crossbar node.

Recent Autobahn's have made it much simpler to fire up a "standalone"
component against a router (e.g. Crossbar). That's just 3 lines of code
in your __main__:

http://autobahn.ws/python/wamp/programming.html#running-components

Perfect, this is exactly what i was looking for, thanks!  I'm going to try it out.

 

In that case, you can shut down the whole OS process

class MyComponent(ApplicationSession):
    def onDisconnect(self):
       reactor.stop()

However, this points to a problem: if you would now take MyComponent to
be hosted again under Crossbar, you run into the issues discussed
before. What I'd like to really achieve is this:

You write MyComponent. Then, without touching a single line in
MyComponent, you decide how to deploy MyComponent:

a) "standalone" (e.g. using ApplicationRunner)
b) hosted in a Crossbar container
c) hosted side-by-side in a Crossbar router

>
>     The problem I'm trying to avoid is keeping long-running processes
> going on systems where you are charged by the minute, like on a
> supercomputer.  There we would like all processes related to the
> application component to be shut down cleanly when the user is done with
> the session.

I'm not sure I get this completely: could you expand a little "when the
user is done with the session" (or expand in general). I'd really like
to achieve a)-c) above. Which doesn't limit you from doing a) anyway.

However, I need to get some sleep;) 6 hours left, need to get up and
travel=(

By all means, get some sleep!  I'll try to expand on what I mean here and get back to you.

Thanks again for your rock-star level of help answering all my questions :-)

Cheers,

Scott 

 

>          > <mailto:autobahnws+unsub...@googlegroups.com>.

>          > To post to this group, send email to autob...@googlegroups.com
>          > <mailto:autob...@googlegroups.com>.
>          > To view this discussion on the web visit
>          >
>         https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com
>         <https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com>
>
>          >
>         <https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com?utm_medium=email&utm_source=footer
>         <https://groups.google.com/d/msgid/autobahnws/38bec994-2070-460b-a1a1-bb37a649a77e%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>          > For more options, visit https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com

> <mailto:autobahnws+unsub...@googlegroups.com>.

[Scott Wittenburg]

Hi Tobias,

   I thought I would just expand a bit on what I meant by "when the user is done with the session".  

   Our goal is to have a system whereby a user can visit a webpage and interact in some way (e.g. clicking a link) which results in the launch of a new visualization process (or "session") on the server, and then they can connect to that "session" and interact with it.  When they navigate away from the page or close their browser, that "session" process on the server should go away.  The "session" is actually an instance of ApplicationSession.

   We have currently achieved this goal using Autobahn (Python and JS), as well as an Apache webserver (taking advantage of proxying, url re-writing, and websocket tunnelling).  To avoid causing confusion, I won't go into all the details of how we do that.  But now we are investigating how crossbar.io could be used to ease some of the deployment.

   I don't know if this helps you understand better what I mean by "when the user is done with the session".

Cheers,

Scott

[Scott Wittenburg]

Hi Tobias,

On Tuesday, September 2, 2014 12:13:17 PM UTC-6, Tobias Oberstein wrote:

Hi Scott,

 > So my question is, what is the right way to shut down everything on the
 > server side cleanly, in response to a specific rpc method?  In certain

If you do reactor.stop() in a container, it doesn't have a chance to
orderly shutdown.

Could you try the following?

class MyComponent(ApplicationSession):

    @wamp.register(u"com.myapp.shutdown_component")
    def shutdown(self):
       self.leave()

    def onLeave(self, details):
       self.disconnect()

I've been trying to implement this approach to app session shutdown outside of the crossbar.io scenario (manual creation of RouterFactory, RouterSessionFactory, and WampWebSocketServerFactory), but I get an error in the browser console:

d {error: "wamp.error.runtime_error", args: Array[1], kwargs: Object}args: Array[1]0: "RouterApplicationSession.send: unhandled message WAMP GOODBYE Message (message = None, reason = wamp.close.normal)" ...

Any idea what I could be doing wrong?

Thanks!

Scott