Does WAMP server use a message queue broker?

[PC]

Hi -

    I am planning to design a system (over websockets) that allows browser/non-browser "worker-clients" to publish data to a "server application process" and also the "server application process" to push data to the worker-clients. Note that the requirement is that I should be able to add new server-application-processes dynamically that register unique topics that they listen on.

I like the way WAMP has been defined. I was also looking into other options such as STOMP over websockets.

It is clear that STOMP requires a messsage queue broker server to which many clients can connect to - specifically in my case, the worker-clients and server application process would be a client to the broker.

Questions:

It appears to me that the Autobahn WAMP server does not have a broker server even though it implements a pubsub pattern. Am I correct?

If so, how can I dynamically add new WAMP applications that can register topics dynamically? Should each server process run its own WAMP server in a different port?

Thanks,

PC

[Daniel Faust]

Hey, think I've done this.

There's a central WAMP server, which has the dispatcher (broker) overridden. Other servers, let's call them services, because they are actually WAMP clients -- so that they don't get confused with the central server -- register at this central server. They tell them what kind of services they offer, ie "service/ping" or "service/stocks".

Then clients connect to the central server, and request interest in services by saying that they are interested in messages with certain tags, ie "service/ping" or "service/stocks".

When the service has new data, it sends an RPC message to the server (ie "service/stock" with stock data attached), the server then checks which clients registered interest, and publishes it to them.

If clients want to interact with services, they also post an RPC message to the server, who then hands it over to a specific service, or to all, if the client wanted some general information, ie which services are avaliable, and in which configuration state they are in (ie one ping service pings every 5 seconds with a certain message, and another one another message every 10 seconds).

Clients can also modify the configuration of services, make them save or load their configuration, and request all data to dynamically create a client-side user interface for the service.

I'm a bit confused about the complexity of this system, so I haven't touched it for a couple of weeks. I'm not sure if it's worth it continuing to research in that direction, somehow it feels amazingly flexible, but on the other hand a slim, custom system without WAMP might be a better solution.

Tell me if it goes in that direction, I'll free some time to post it to github.

[Tobias Oberstein]

Hi Paddy,

Am 07.08.2012 00:19, schrieb PC:
> Hi -
> I am planning to design a system (over websockets) that allows
> browser/non-browser "worker-clients" to publish data to a "server
> application process" and also the "server application process" to push
> data to the worker-clients. Note that the requirement is that I should
> be able to add new server-application-processes dynamically that
> register unique topics that they listen on.
>
> I like the way WAMP has been defined. I was also looking into other
> options such as STOMP over websockets.
>
> It is clear that STOMP requires a messsage queue broker server to which
> many clients can connect to - specifically in my case, the
> worker-clients and server application process would be a client to the
> broker.
>
> Questions:
> It appears to me that the Autobahn WAMP server does not have a broker
> server even though it implements a pubsub pattern. Am I correct?

No, already AutobahnPython includes a message broker. To get started,
have a look at the 2nd example here:

http://autobahn.ws/python

and the PubSub examples here

https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub

> If so, how can I dynamically add new WAMP applications that can register
> topics dynamically? Should each server process run its own WAMP server
> in a different port?

Running on different ports is of course possible, but there is no need
to .. and I'd guess it's probably more complex. You can dynamically
spawn processes from your single AutobahnPython WAMP server running
on a single port using

http://twistedmatrix.com/documents/current/core/howto/process.html

This allows to dynamically spawn new processes and talk to them,
forwarding between PubSub and the processes' stdin/stdout in both
directions.

You can register new topics for PubSub now only at the beginning of
a WAMP session, but also during it's lifetime.

Since you want to do something "more" than merely have the builtin
PubSub message broker dispatch events between clients (probably
talk to your processes), you might want to use custom handlers
which you can do:

registerHandlerForPubSub
registerHandlerForSub
registerHandlerForPub

http://autobahn.ws/python/reference
https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom

Using this feature allows you to do your "special stuff", but reuse
the builtin broker for all the boilerplate around PubSub.

Hope that helps,
Tobias

>
> Thanks,
> PC

[PC]

Hi Daniel,

    Thanks for the reply. What you have done sounds interesting to me. Please post the code and let me know. 

Thanks,

PC

On Monday, August 6, 2012 3:47:46 PM UTC-7, Daniel F. wrote:

Hey, think I've done this.

There's a central WAMP server, which has the dispatcher (broker) overridden. Other servers, let's call them services, because they are actually WAMP clients -- so that they don't get confused with the central server -- register at this central server. They tell them what kind of services they offer, ie "service/ping" or "service/stocks".

Then clients connect to the central server, and request interest in services by saying that they are interested in messages with certain tags, ie "service/ping" or "service/stocks".

When the service has new data, it sends an RPC message to the server (ie "service/stock" with stock data attached), the server then checks which clients registered interest, and publishes it to them.

If clients want to interact with services, they also post an RPC message to the server, who then hands it over to a specific service, or to all, if the client wanted some general information, ie which services are avaliable, and in which configuration state they are in (ie one ping service pings every 5 seconds with a certain message, and another one another message every 10 seconds).

Clients can also modify the configuration of services, make them save or load their configuration, and request all data to dynamically create a client-side user interface for the service.

I'm a bit confused about the complexity of this system, so I haven't touched it for a couple of weeks. I'm not sure if it's worth it continuing to research in that direction, somehow it feels amazingly flexible, but on the other hand a slim, custom system without WAMP might be a better solution.

Tell me if it goes in that direction, I'll free some time to post it to github.

[PC]

Hi  Tobias,

    Thanks.

   

 If I understand you right, I would have a single "main" WAMP server different server-application-processes communicating with it through IPC.  I would need to override the publish/subscribe routines of the base pubsub engine (of the main WAMP server) and do my own thing. Specifically in my case, for example, my publish routine would use IPC and send the data across to the appropriate server-process. Am I correct?

How do you suggest handling RPCs in this case? Note that the routine would actually be defined in one of the server-application-processes, not in the main WAMP server.

Thanks,

PC

[Tobias Oberstein]

Am 07.08.2012 20:29, schrieb PC:
> Hi Tobias,
> Thanks.
> If I understand you right, I would have a single "main" WAMP server
> different server-application-processes communicating with it through
> IPC. I would need to override the publish/subscribe routines of the

Assuming you actually need to run "other stuff" in separate OS processes
on the same machine (as running the WAMP server), the simplest IPC
mechanism to use is probably pipes (standard
input/output).

> base pubsub engine (of the main WAMP server) and do my own thing.
> Specifically in my case, for example, my publish routine would use IPC
> and send the data across to the appropriate server-process. Am I correct?

Yes. Technically, you don't override the WampServerProtocol base class
methods, but use registerHandlerForPub to install your custom logic
into the message broker.

>
> How do you suggest handling RPCs in this case? Note that the routine
> would actually be defined in one of the server-application-processes,
> not in the main WAMP server.

Now you lost me: do you want a) to have logic in your background server
processes that you can call from WAMP clients via RPC, or b) have
logic in your background processes that get called when events are
published by WAMP clients?

> <http://autobahn.ws/python/reference>
> https://github.com/tavendo/AutobahnPython/tree/master/examples/wamp/pubsub/custom

[PC]

On Tuesday, August 7, 2012 2:21:50 PM UTC-7, Tobias Oberstein wrote:

Am 07.08.2012 20:29, schrieb PC:
> Hi Tobias,
>      Thanks.
>   If I understand you right, I would have a single "main" WAMP server
> different server-application-processes communicating with it through
> IPC.  I would need to override the publish/subscribe routines of the

Assuming you actually need to run "other stuff" in separate OS processes
on the same machine (as running the WAMP server), the simplest IPC
mechanism to use is probably pipes (standard
input/output).

> base pubsub engine (of the main WAMP server) and do my own thing.
> Specifically in my case, for example, my publish routine would use IPC
> and send the data across to the appropriate server-process. Am I correct?

Yes. Technically, you don't override the WampServerProtocol base class
methods, but use registerHandlerForPub to install your custom logic
into the message broker.

>
> How do you suggest handling RPCs in this case? Note that the routine
> would actually be defined in one of the server-application-processes,
> not in the main WAMP server.

Now you lost me: do you want a) to have logic in your background server
processes that you can call from WAMP clients via RPC, or b) have
logic in your background processes that get called when events are
published by WAMP clients?

a). similar to what you have defined in WAMP. 

[Tobias Oberstein]

> Now you lost me: do you want a) to have logic in your background server
> processes that you can call from WAMP clients via RPC, or b) have
> logic in your background processes that get called when events are
> published by WAMP clients?
>
> a). similar to what you have defined in WAMP.

In this case you can use

registerHandlerMethodForRpc
registerHandlerProcedureForRpc

where you then can implement the communication with your background
processes.

When your background processes boot up, they also need to announce the
RPC endpoints they provide via IPC to the WAMP server which then needs
to setup the fowarding like above.

What language(s) are the background programs written in?

[PC]

PHP. But they can be written in any language since they are different processes. That's one of the features that I'm shooting for.

[Tobias Oberstein]

Am 08.08.2012 19:06, schrieb PC:

> PHP currently. But since they are different processes, they can be
> written in any language. This is one of the features that I'm
> shooting for.
>

Ok, I really know too little to give a thorough advice on architecture
on such a thing. Just a couple of further hints:

- Who will be responsible for starting and monitoring the background
processes (if one fails, who restarts etc)?
- Will those programs always run locally together with the frontend WAMP
server?
- Essentially, you will need to have something like WAMP (at least the
RPC part) over IPC
- In a way thats not sufficient, since the frontend WAMP server needs to
know what RPC endpoint are available .. on which program

For PHP (standalone), there is a WAMP server implementation: Ratchet.

For transparent forwarding of WAMP RPCs to anything that exposes
a REST/HTTP or Ext.Direct API, our commercial offering Tavendo WebMQ
is capable of doing that. You can call from any WAMP client via
WebSocket into anything with REST or Ext.Direct API today, and
database stored procedures in the coming future. You can also push
PubSub events from within your PHP i.e. This is a feature of WebMQ.
WebMQ is an integration server based on Autobahn, with a whole range
of flexible integration options.

Should a commercial offering be of interest to you, we should talk
and determine if it already could do what you want, and if not if
we can make that feature available.

Cheers,
Tobias

[PC]

Hi Tobias,
You probably have done a good amount of research on scalability of
WAMP Servers. Can you please let me know how it (built-in broker)
compares with a standalone broker (such as Apache ActiveMQ) in terms
of scalability?
-PC

On Aug 8, 10:18 am, Tobias Oberstein <tobias.oberst...@gmail.com>
wrote: