crossbar stop not ending all processes

[Samuel Coombs]

Hello, 

I just started using crossbar.io (python) and I have a program that has 4 workers(3 containers and 1 router). 

I have a problem when I stop crossbar. When I use CTRL-C to stop it, all the processes end and I am able to start it again. When I use the command 'crossbar stop' only 3 of the 4 processes end. When I try to start crossbar again, 3 of the workers will start, and when the 4th one tries to start, it will fail and I will get the error 'Address in Use.'

I was wondering if there is something I am doing wrong or is there a bug somewhere?

Thanks!

[Alexander Gödde]

Hi Samuel!

This behavior is not something that we've encountered during our testing. 

Could you post more information about what precisely you are doing in the worker and how things are configured? With the current information we can't really give you any feedback.

Regards,

Alex

[Samuel Coombs]

Yeah I Can

Workers:

1. Router

2. Uses ZMQ to receive messages to forward to a screen

3. Reads a Tag from a PLC

4. Using ZMQ to receive messages that write to a database. 

The fourth on never ends when using crossbar stop, but ends when using ctrl-c. There might be something I am doing wrong, I am new to crossbar but it seems pretty straight forward. I basically took the hello app that is on the crossbar.io website and modified that, I just changed the port and classnames for the containers. 

[Alexander Gödde]

Hello Samuel!

Just tried with hello:nodejs extended to three workers on Ubuntu, and shutdown is clean.

A more complete list of information required to replicate your problem:

1. What operating system are you using?

2. What language are you using the hello example for? Do all guests use the same language/runtime?

3. What is your configuration file? 

You could also experiment a bit with the workers to give us a better idea for where the problem lies, e.g.: Is it the third worker in the config? (change the order or the workers) Is it the worker itself? (run this as the only worker)

Regards,

Alex

[Samuel Coombs]

My OS is an windows 7, 64 bit machine. I am using python for the backend. 

Attached is the config file.

For the DB Write i am using pyodbc to a SQLExpress server. 

And I moved all the order of the workers around and it always fails on the same worker(the one that writes to the database) .I have to go task manager and end the python.exe file. Then it will allow me to start crossbar again.

here is a snip of the error

←[34m2015-10-08T11:51:11-0400 [Controller   9312]←[39m Container 'worker2': component 'component1' started

←[34m2015-10-08T11:51:11-0400 [Controller   9312]←[39m Starting Container with ID 'worker3'...

←[31m2015-10-08T11:51:11-0400 [Container    1888]←[39m Fatal error in component: While firing onJoin - Address in use

←[31m2015-10-08T11:51:11-0400 [Controller   9312]←[39m Component 'component1' failed to start; shutting down node.

←[32m2015-10-08T11:51:11-0400 [Container    1888]←[39m Closed connection to 'component1' with code '1000'

And thanks for the response, I really appreciate it!

[Alexander Gödde]

Hi!

So it seems that this specific component handles shutdown signals differently than the others.

1. Do you have `psutil` installed on your machine? This should influence how shutdown is sent.

2. Could you try to run things on a Linux machine and see if the same problem occurs there (to see whether this is a Windows issue or some more general problem with Crossbar.io shutdown mechanisms)?

Regards,

Alex

[Samuel Coombs]

This is the latest error i just got. My company does not have a linux machine so i can't try that out. Ill look in the other thing you said to try!

Traceback (most recent call last):

  File "c:\users\sam\anaconda\lib\site-packages\twisted\internet\base.py", line 1194, in run

    self.mainLoop()

  File "c:\users\sam\anaconda\lib\site-packages\twisted\internet\base.py", line1203, in mainLoop self.runUntilCurrent()

  File "c:\users\sam\anaconda\lib\site-packages\twisted\internet\base.py", line 847, in runUntilCurrent self.fireSystemEvent("shutdown")

  File "c:\users\sam\anaconda\lib\site-packages\twisted\internet\base.py", line 640, in fireSystemEvent event.fireEvent()

--- <exception caught here> ---

  File "c:\users\sam\anaconda\lib\site-packages\twisted\internet\base.py", line 411, in fireEvent    result = callable(*args, **kwargs)

  File "c:\users\sam\anaconda\lib\site-packages\crossbar\controller\process.py", line 566, in _cleanup_worker    worker.proto.transport.signalProcess('TERM')

  File "c:\users\sam\anaconda\lib\site-packages\twisted\internet\_dumbwin32proc.py", line 255, in signalProcess    win32process.TerminateProcess(self.hProcess, 1) pywintypes.error: (5, 'TerminateProcess', 'Access is denied.')

[Tobias Oberstein]

Samuel,

as can be seen from the logs, Crossbar indeed tries to kill the worker

worker.proto.transport.

signalProcess('TERM')

but fails

'TerminateProcess', 'Access is denied.'

Here is a theory of what happens: the third worker is different from the others, because it holds a DB connection ("that write to a database. "). Crossbar will first try to orderly shutdown all the workers which succeeds for router and the first 2 containers, but doesn't for the third. (I don't use pyodbc, so I don't know if there is an option to hook into that so it can be shutdown properly). After a timeout, Crossbar will then try to hard terminate the remaining worker.

Now, why the behavior is different between CTRL-C'ing Crossbar versus `crossbar stop` is another question.

My guess on that would be: when CTRL-Cing, the killing is done by the program you originally started (the Crossbar node controller that forker the worker, and that you now CTRL-C).

But when you use `crossbar stop`, the killing is done by a different program (the new "crossbar stop"). And that fails, because killing arbitrary, different processes requires extended permissions.

Please try the following: run the "crossbar stop" command in a CMD shell started with "admin privileges".

Cheers,
/Tobias

[Samuel Coombs]

Tobias, thank you for your response.

Running it as an admin does not stop that process. Yes it is connected to a database but that connection is not always open. I receive a json file using ZMQ. Once I receive a message, I send it to a function which open a database connection and write the data, then closes the connection.

The problem is with ZMQ, it sits there and waits for a message. Only CTRL-C will kill that process, so I will have to fix that and will let you know if I fix it. 

If you would like to play around with this here is the code. I took out the database stuff because right now that is not the problem, It's ZMQ.

from twisted.internet.defer import inlineCallbacks
from twisted.logger import Logger
from autobahn import wamp
from autobahn.twisted.util import sleep
from autobahn.twisted.wamp import ApplicationSession
import zmq, json


class AppSession(ApplicationSession):
   log = Logger()

   @inlineCallbacks
   def onJoin(self, details):
      # setting up zmq to read from port 5555
      ctx1 = zmq.Context()
      OverViewSocket = ctx1.socket(zmq.PULL)
      OverViewSocket.setsockopt(zmq.RCVHWM, 1)
      OverViewSocket.bind('tcp://*:5555')
      self.log.info('Overview Ready')
      while True:
         msg = OverViewSocket.recv()                         
         yield self.publish("com.example.Overview", msg)          # publish to 'com.example.Overview'
         yield self.log.info(msg)   

[Samuel Coombs]

So when using zmq with crossbar you have to pass the flag zmq.NOBLOCK in the socket receive. Also there needs to be a yield sleep(#) somewhere in the while loop or else it will not exit. I do not understand why there needs to be a sleep, so i just put a small sleep period. 

Now crossbar will end using crossbar stop.

Thanks again for helping me out! 

[Alexander Gödde]

Hi Samuel!

Glad that you got things sorted out!

Regards,

Alex

[Tobias Oberstein]

are you using a non-blocking ZMQ client library?

https://github.com/smira/txZMQ

if not, this is an issue .. with a "normal" blocking client library, you
will effectively block the whole event reactor inside Twisted --

btw: same for DB libs. here, almost all are blocking. you will need to use

https://twistedmatrix.com/documents/current/core/howto/rdbms.html

which wraps those libs running the blocking DB connections on a
background worker thread pool

Am 09.10.2015 um 17:48 schrieb Samuel Coombs:
> So when using zmq with crossbar you have to pass the flag zmq.NOBLOCK in
> the socket receive. Also there needs to be a yield sleep(#) somewhere in
> the while loop or else it will not exit. I do not understand why there
> needs to be a sleep, so i just put a small sleep period.
>
> Now crossbar will end using crossbar stop.
>
> Thanks again for helping me out!
>
> On Friday, October 9, 2015 at 9:27:56 AM UTC-4, Samuel Coombs wrote:
>
> Tobias, thank you for your response.
>
> Running it as an admin does not stop that process. Yes it is
> connected to a database but that connection is not always open. I
> receive a json file using ZMQ. Once I receive a message, I send it
> to a function which open a database connection and write the data,
> then closes the connection.
>
> The problem is with ZMQ, it sits there and waits for a message. Only
> CTRL-C will kill that process, so I will have to fix that and will
> let you know if I fix it.
>
> If you would like to play around with this here is the code. I took
> out the database stuff because right now that is not the problem,
> It's ZMQ.
>
>
> from twisted.internet.deferimport inlineCallbacks
> from twisted.loggerimport Logger
> from autobahnimport wamp
> from autobahn.twisted.utilimport sleep

> from autobahn.twisted.wampimport ApplicationSession

> import zmq, json
>
>
> class AppSession(ApplicationSession):
> log = Logger()
>
> @inlineCallbacks
> def onJoin(self, details):
> # setting up zmq to read from port 5555
> ctx1 = zmq.Context()
> OverViewSocket = ctx1.socket(zmq.PULL)
> OverViewSocket.setsockopt(zmq.RCVHWM, 1)
> OverViewSocket.bind('tcp://*:5555')

> self.log.info <http://log.info>('Overview Ready')

> while True:
> msg = OverViewSocket.recv()
> yield self.publish("com.example.Overview", msg)# publish to 'com.example.Overview'

> yield self.log.info <http://log.info>(msg)

> is on the crossbar.io <http://crossbar.io>

> website and modified that, I just changed
> the port and classnames for the containers.
>
> On Thursday, October 8, 2015 at 9:51:36 AM
> UTC-4, Alexander Gödde wrote:
>
> Hi Samuel!
>
> This behavior is not something that
> we've encountered during our testing.
>
> Could you post more information about
> what precisely you are doing in the
> worker and how things are configured?
> With the current information we can't
> really give you any feedback.
>
> Regards,
>
> Alex
>
>
> Am Donnerstag, 8. Oktober 2015 15:10:50
> UTC+2 schrieb Samuel Coombs:
>
> Hello,
>
> I just started using crossbar.io

> <http://crossbar.io> (python) and I

> have a program that has 4 workers(3
> containers and 1 router).
>
> I have a problem when I stop
> crossbar. When I use CTRL-C to stop
> it, all the processes end and I am
> able to start it again. When I use
> the command 'crossbar stop' only 3
> of the 4 processes end. When I try
> to start crossbar again, 3 of the
> workers will start, and when the 4th
> one tries to start, it will fail and
> I will get the error 'Address in Use.'
>
> I was wondering if there is
> something I am doing wrong or is
> there a bug somewhere?
>
> Thanks!
>

> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.
> To post to this group, send email to cross...@googlegroups.com
> <mailto:cross...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crossbario/1c14eace-10b1-4b59-b3d4-31b3b7e647a2%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/1c14eace-10b1-4b59-b3d4-31b3b7e647a2%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.