Websocket configuration

[Bogdan Glinskiy]

Good evening.

I have an application in which rep and pub sockets listen their ports. Can you give me an example of zerogw configuration file for websockets? I want to use them as transport in my application.

 Thanks a lot,
 Bogdan.

Hi,

Here is an example, with unix sockets:

https://github.com/tailhook/zenith/blob/master/config/zerogw.yaml

You should easily adapt it to TCP, if you know how zeromq addresses look like.

And here is also some (unfinished) tutorial:

https://github.com/tailhook/zenith/blob/master/draft.rst

 

-- 

Paul 

 

 I should say, that i already had similar configuration file. But seems it doesn't work the way it should. Here are my files:

 conf.yaml
 ---------------

 

Server:
 listen:
 - host: 0.0.0.0
 port: 8080

 Routing:
 websocket:
 enabled: yes
 forward: !zmq.Pub
 - !zmq.Connect "tcp://127.0.0.1:5000" ------->(i guess error is here)
 subscribe: !zmq.Sub
 - !zmq.Connect "tcp://127.0.0.1:5001" ------->(i guess error is here too)

 publisher.py
 --------------------

 

 import zmq
 import random
 import sys
 import time

 port = "5001"
 if len(sys.argv) > 1:
 port = sys.argv[1]
 int(port)

 context = zmq.Context()
 socket = context.socket(zmq.PUB)
 socket.bind("tcp://*:%s" % port)

 while True:
    topic = random.randrange(9999,10005)
    messagedata = random.randrange(1,215) - 80
    print "%d %d" % (topic, messagedata)
    socket.send("%d %d" % (topic, messagedata))
    time.sleep(1)

 replier.py
 --------------------

 

 import zmq
 import time
 import sys

 port = "5000"
 if len(sys.argv) > 1:
 port = sys.argv[1]
 int(port)

 context = zmq.Context()
 socket = context.socket(zmq.REP)
 socket.bind("tcp://*:%s" % port)

 while True:
    # Wait for next request from client
    message = socket.recv()
    print "Received request: ", message
    time.sleep (1)
    socket.send("Data from Replier")

 With starting zerogw with --log-error 7 i have debug information about the websockets:

 [DEBG] websocket.c:239: Websocket started
 [DEBG] websocket.c:281: Websocket sent hello

 but that's all... Data doesn't pass to publisher or replier like it was with http tutorial.


 But i should admit, that http tutorial from  http://zerogw.readthedocs.org/en/latest/httptut.html works as i expected (but i replaced sock.connect('tcp://127.0.0.1:5000') to sock.bind('tcp://127.0.0.1:5000') and in conf file !zmq.Bind to !zmqBind) 

Sorry, but you can't use REP socket for websocket connections. Use pub-sub or push-pull (with respective other counterpart). You are currently trying to publish from zerogw to REP socket in python

 

--

Paul 

So let me get this straight: in my application, that uses both req-rep(to send commands to app) and pub-sub(to receive updates from app) patterns, i can't use websockets as transport for the first one but only can use it for the second one?

The short answer is no. The long answer is below.

 If so, is there any way to do it with zerogw?

Yes. Just make request reply on top of pub-sub. I.e. we usually send request with the json like:

["object.method", 1234, "arg1", "arg2"]

And reply like this:

["_reply", 1234, "response"]

So client matches response for client by id (1234 in this case).

In python code it looks like:

cli, _msg, json_data = pull_socket.recv_multipart()
meth, id, *args = json.loads(json_data)
assert _msg == 'message', _msg
obj_name, meth_name = meth.split('.')
value = getattr(objects[obj_name], meth_name)(*args)
pub_socket.send_multipart(['send', cli, json.dumps(["_reply", id, value])])

-- 
Paul

[Bogdan Glinskiy]

Thank you!! I think, i did what i wanted :) My final test files are:

zerogw.yaml

-------------------

Server:
  listen:
  - host: 0.0.0.0
    port: 8080

Routing:
  websocket:
    enabled:

yes
    forward: !zmq.Push

    - !zmq.Connect "tcp://127.0.0.1:5001"
    subscribe: !zmq.Sub

    - !zmq.Connect "tcp://127.0.0.1:5002"

publisher.py

------------------
import zmq
import random
import sys
import

time

pullPort = "5001"
pubPort  = "5002"

context = zmq.Context()

pull_socket = context.socket(zmq.PULL)
pub_socket = context.socket(zmq.PUB)

pull_socket.bind("tcp://*:%s" % pullPort)
pub_socket.bind("tcp://*:%s" % pubPort)

while True:
    cli, *argv = pull_socket.recv_multipart()
    print ("incomming: ", argv)
   
    for arg in argv:
        if (arg.decode('utf-8') == "connect"):
            pub_socket.send_multipart([b'send', cli, b'hello, new client'])
        elif (arg.decode('utf-8') == "message"):
            pub_socket.send_multipart([b'send', cli, b'reply'])

    #some subscriber, that works in a separated thread
    pub_socket.send_multipart([b'sendall', b'some event to all subscribers'])

Of course, example is not perfect and subscriber doesn't work in a separated thread, but it demonstrate what i wanted. 

But there is a kind of problem: after the socket has been disconnected, i still receive heartbeats.. What it can be? Should i use in a conf file "heartbeat-interval: 0" for that purpose?

[Paul Colomiets]

Hi Bogdan,

On Wed, Nov 19, 2014 at 1:44 PM, Bogdan Glinskiy <fett...@gmail.com> wrote:
> Thank you!! I think, i did what i wanted :) My final test files are:
>

Great.

> Of course, example is not perfect and subscriber doesn't work in a separated
> thread, but it demonstrate what i wanted.
>

Why not? You can use zeromq sockets in separate threads if you run
socket operations under mutex, or you can connect two sockets to
zerogw (that's where the real power of zeromq is).

> But there is a kind of problem: after the socket has been disconnected, i
> still receive heartbeats.. What it can be? Should i use in a conf file
> "heartbeat-interval: 0" for that purpose?
>

Heartbeats are for detecting zerogw server, not websocket connection.
So yes they are always sent. But probably with the current
implementation they are useless, so yes you should probably just turn
them off.

--
Paul

[Bogdan Glinskiy]

Hi, Paul

I have another questions:

1) how can i replace python's function "send_multipart([b'send', cli, b'reply'])" from the scripts above?

2) how can i get "cli" for sending the reply only to him?

My C code is following:

#include <zmq.h>
#include <pthread.h>


#define BUFF_SIZE 1024


void *context;
void *publisher;
void *puller;


void *doPublish()
{
    while (1) {
        zmq_send (publisher, "sendall", 7, ZMQ_SNDMORE);
        zmq_send (publisher, "some event to all subscribers", 29, 0);
        sleep(2);
    }
}


void *doReply()
{
    while (1) {
        char type[BUFF_SIZE];
        char data[BUFF_SIZE];
   
        zmq_recv (puller, type,  BUFF_SIZE, ZMQ_RCVMORE);
        zmq_recv (puller, data,  BUFF_SIZE, 0);


        printf("Type: %s\nData: %s\n\n", type, data);


        // Need to get cli somehow and send reply to him
    }
}


int main (void)
{  
    context = zmq_ctx_new ();
    publisher = zmq_socket (context, ZMQ_PUB);
    puller = zmq_socket (context, ZMQ_PULL);


    zmq_bind (publisher, "tcp://*:5002");
    zmq_bind (puller, "tcp://*:5001");


    int major, minor, patch;
    zmq_version(&major, &minor, &patch);
    printf ("Current ØMQ version is %d.%d.%d\n", major, minor, patch);


    pthread_t threads[2];      
    pthread_create(&threads[0], NULL, doPublish, (void*)NULL);
    pthread_create(&threads[1], NULL, doReply  , (void*)NULL);


    pthread_join( threads[0], NULL);
    pthread_join( threads[1], NULL);


   
    //  We never get here, but clean up anyhow
    zmq_close (publisher);
    zmq_ctx_destroy (context);
    pthread_exit(NULL);
    return 0;
}

[Paul Colomiets]

Hi Bogdan,

On Wed, Nov 19, 2014 at 7:35 PM, Bogdan Glinskiy <fett...@gmail.com> wrote:
> Hi, Paul
>
> I have another questions:
> 1) how can i replace python's function "send_multipart([b'send', cli,
> b'reply'])" from the scripts above?

The basic workflow is:

int more = true
while(more) {
zmq_recv(sock, buf)
zmq_getsockopt(sock, ZMQ_RCVMORE, &more, &sz);
}

I.e. read until ZMQ_RCVMORE flag is set. Note that `buf` in my example
is overwritten on each recv, so you should use it. Also note that the
while loop in this example receives single multi-part message, so it
corresponds to single websocket message.

> 2) how can i get "cli" for sending the reply only to him?
>

Each zmq_recv receives a one "part". A message in terms of zeromq.
`cli` is just first message received. But be careful, you should
always check if ZMQ_RCVMORE is not set, before stopping reading. (just
like the recv_multipart does for you in python).

You should consider using czmq (https://github.com/zeromq/czmq) as it
helps handling multi-part messages in the right way.

Hope this helps!

--
Paul

[Bogdan Glinskiy]

Hi again!

Well, i thought correctly about the writing python's send_multupart and about the cli. After fixing few lines i can read all the parts correctly (in my opinion) but i still can't send reply to client.. Seems this is a problem of receiving the wrong client id or settings the wrong client id in the send function.. my code:

void *doReply()
{
    while (1) {
        int64_t more;
        size_t more_size = sizeof(more);
        int part = 0;
   
        char content[3][BUFF_SIZE];

        do {
            zmq_recv (puller, &content[part], BUFF_SIZE, 0);
            zmq_getsockopt (puller, ZMQ_RCVMORE, &more, &more_size);
            printf("Received part %i: %s of length %d\n", part, content[part], strlen(content[part])); // length of client is 1...
            part++;
        } while (more);


        printf("Receiving finished\n\n\n");


        if (strstr(content[1], "message") != NULL) {
            printf("Trying to send reply...\n\n\n");
            zmq_send(publisher, "send", strlen("send"), ZMQ_SNDMORE);           // works well, displaying Websocket got SEND request
            zmq_send(publisher, content[0], strlen(content[0]), ZMQ_SNDMORE);   // error here, Searching for hole 1970479212...
            zmq_send(publisher, "reply", strlen("reply"), 0);
        }
    }    
}

[Paul Colomiets]

Hi Bogdan,

On Thu, Nov 20, 2014 at 4:52 PM, Bogdan Glinskiy <fett...@gmail.com> wrote:
> Hi again!
>
> Well, i thought correctly about the writing python's send_multupart and
> about the cli. After fixing few lines i can read all the parts correctly (in
> my opinion) but i still can't send reply to client.. Seems this is a problem
> of receiving the wrong client id or settings the wrong client id in the send
> function.. my code:
>

You can't use strstr() and strlen() (and most of the str*) functions
on zeromq mesages. They are not ended with nul character. You must
store lengths of message received (returned by zmq_recv call) and use
that length, and memcmp() for comparison (don't forget to compare
length first).

> void *doReply()
> {
> while (1) {
> int64_t more;
> size_t more_size = sizeof(more);
> int part = 0;
>
> char content[3][BUFF_SIZE];
>
> do {
> zmq_recv (puller, &content[part], BUFF_SIZE, 0);
> zmq_getsockopt (puller, ZMQ_RCVMORE, &more, &more_size);
> printf("Received part %i: %s of length %d\n", part,
> content[part], strlen(content[part])); // length of client is 1...
> part++;
> } while (more);
>
>
> printf("Receiving finished\n\n\n");
>
>
> if (strstr(content[1], "message") != NULL) {
> printf("Trying to send reply...\n\n\n");
> zmq_send(publisher, "send", strlen("send"), ZMQ_SNDMORE);
> // works well, displaying Websocket got SEND request
> zmq_send(publisher, content[0], strlen(content[0]),
> ZMQ_SNDMORE); // error here, Searching for hole 1970479212...
> zmq_send(publisher, "reply", strlen("reply"), 0);
> }
> }
> }
>

--
Paul

[Bogdan Glinskiy]

Thanks a lot again! Your answer helped me :) I just forgot that "The zmq_recv() function shall return number of bytes in the message if successful". Now everything works like i want.