ApplicationSession socket.publish() unable to publish to Crossbar WAMP router after "a while"

[Camron Levanger]

Hi All,

I have recently upgraded my Autobahn/Twisted libraries, and I am now using Crossbar as my WAMP router.

As part of the upgrade, my publishing code now is run by Crossbar as an ApplicationSession instance.

I use WAMP to support some real-time web applications, and it supports a fair number of users. This is a Pub/Sub model, also using AutobahnJS libraries for the client.

I am currently having a problem where my application is eventually no longer able to publish messages to the crossbar router for delivery. Clients are still able to connect and subscribe to topics no problem, they just stop getting updates due to the inability to publish.

If it makes any difference, this seems to always occur when I get up in between 20,000 to 30,000 client connections. I am publishing an average of 1500 messages per second.

The basic publishing code is as follows:

def on_message(self, headers, message):
        """
        STOMP message handler, when I get a message from the queue, dispatch it
        to my users
        """
        self.log.debug('received a STOMP message %s', message)


        try:
            response = json.loads(message)


            destination = '%s.%s' % (
                response['topic'],
                response['destination_id']
            )


            self.socket.publish(
                destination,
                json.dumps(response)
            )


        except Exception as e:
            self.log.error(str(e))

str(e) is an empty string, I am working right now to gather any useful information from the exception,

I have made sure to follow all of the production Crossbar.io configuration suggestions, and the server itself is not under heavy load when this occurs.

Is this expected behavior, have I just hit some kind of known limit? I was hoping to support more throughput per instance, but I can certainly load balance if that is the answer. What can I do to prevent, or recover from this when it happens?

Thanks all,

Cam

[Camron Levanger]

Further information:

I wrote a nagios plugin to attempt publishing to crossbar and alert if it throws the exception. The nagios monitor is still able to publish to crossbar when the ApplicationSession is unable to publish to crossbar.

[Camron Levanger]

I haven't been able to get much more info on the exception unfortunately. At this point I think we will probably just go back to WAMP V1 and the built in router and leave it.

Python 2.7.7

Twisted==15.1.0
autobahn==0.10.2
cffi==1.0.2
characteristic==14.3.0
cryptography==0.9
enum34==1.0.4
idna==2.0
ipaddress==1.0.7
java-config==2.2.0
javatoolkit==0.3.0
mod-pywebsocket==0.7.9
ndg-httpsclient==0.4.0
portage==2.2.14
pyOpenSSL==0.15.1
pyasn1==0.1.7
pyasn1-modules==0.0.5
pycparser==2.13
requests==2.7.0
service-identity==14.0.0
six==1.9.0
virtualenv==1.11
wsgiref==0.1.2
zope.interface==4.1.1

[Tobias Oberstein]

To get feedback on publish, you can enable "acknowledged publication": http://autobahn.ws/python/reference/autobahn.wamp.html#autobahn.wamp.interfaces.IPublisher.publish

Rgd performance, Crossbar.io is able to route 30k calls/sec on a single core router worker process ..

[Camron Levanger]

Thank you for the hint Tobias! I have made that change and restarted the service. With any luck I can get some more useful information to report back soon.

[Tobias Oberstein]

Hi Camron,

just to be sure: without acknowledge, publish will return nothing and
not raise. If you enable acknowledge, it'll return a Twisted Deferred or
asyncio Future, and you need to attach an errback handler to that.

Or you can use co-routine style

@inlineCallbacks
def on_message(self, headers, message):
...
try:
yield self.socket.publish(...)
except Exception as e:
...

==

unrelated note:

self.socket.publish(
destination,
json.dumps(response)
)

Why conversion to string again?

self.socket.publish(
destination,
response
)

should be fine.

Q (just curious): why are you using STOMP on the backend and not WAMP
"end-to-end"? I guess you have some AMQP broker sitting around?

Cheers,
/Tobias

> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/2855bcb2-fd69-4ced-a228-b5da0d43eb58%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/2855bcb2-fd69-4ced-a228-b5da0d43eb58%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Camron Levanger]

Thanks again Tobias.

I do get an error now. 

repr(e) returns:

TransportLost()

But that is it.

Any ideas to move me further along?

On Thursday, July 9, 2015 at 5:26:19 AM UTC-6, Tobias Oberstein wrote:

[Tobias Oberstein]

Am 10.07.2015 um 00:08 schrieb Camron Levanger:
> Thanks again Tobias.
>
> I do get an error now.
>
> repr(e) returns:
>
> TransportLost()

This means, you can't publish since there is no connection anymore.

>
> But that is it.
>
> Any ideas to move me further along?

Override onLeave/onDisconnect on your ApplicationSession object (the
self.socket thing I suppose).

http://autobahn.ws/python/wamp/programming.html#session-lifecycle

A transport might get lost for different reasons. E.g. network
intermediaries timing out an idle underlying TCP. In this case,
WebSocket auto-ping/pong helps (avail in Autobahn and Crossbar.io).

> self.log.debug('received a STOMP message %s',message)
>
>
> try:
> response =json.loads(message)
>
>
> destination ='%s.%s'%(

> response['topic'],
> response['destination_id']
> )
>
>
> self.socket.publish(
> destination,
> json.dumps(response)
> )
>
>
> exceptExceptionase:
> self.log.error(str(e))
> |
>
> str(e) is an empty string, I am working right now to gather any
> useful information from the exception,
>
> I have made sure to follow all of the production Crossbar.io
> configuration suggestions, and the server itself is not under
> heavy load when this occurs.
>
> Is this expected behavior, have I just hit some kind of known
> limit? I was hoping to support more throughput per instance, but
> I can certainly load balance if that is the answer. What can I
> do to prevent, or recover from this when it happens?
>
> Thanks all,
>
> Cam
>

> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/057a4bf3-7b23-44f1-a3db-32383e997964%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/057a4bf3-7b23-44f1-a3db-32383e997964%40googlegroups.com?utm_medium=email&utm_source=footer>.