Heartbeat event format

[Gal Ben-Haim]

I'm working on a Golang library for ZeroRPC - https://github.com/bsphere/zerorpc

I'm stuck on the client implementation regarding heartbeats,

should heartbeat events be addressed to a specific channel (have "responds_to" header) ? its unclear from the protocol docs which 

states that any channel should send heartbeats.

as I understand the protocol:

client sends event, which opens a channel with the id of the event's message_it

client sends heartbeats on that channel until there's a response (or 2 heartbeats are skipped)

client "closes" the channel, doesn't use it anymore

am I correct ?

[François-Xavier Bourlet]

Good news :)

You should probably rename your repository in something like
zerorpc-go, to make it easier for people to find it :)

I think you understand correctly the order of events for the heartbeat.

Hearbeats a per channels. So when you open a channel, you send
heartbeats on it. An hearbeat is a _zpc_hb event that doesn't expect
any answer. Simply send an heartbeat msg at the heartbeat frequency
(the default is 5s).

At the same time, if no heartbeat messages a received on the channel
for 2 times the heartbeat frequency, you consider the connection lost.

You can see how it is implemented in a really simply in the python version:

Emitting the hearbeat event and checking how long we got any heartbeat
msg from the peer:
https://github.com/bombela/zerorpc-python/blob/master/zerorpc/heartbeat.py#L69

Every-time we receive an heartbeat event on the channel, we update the
timestamp, as simple as that:
https://github.com/bombela/zerorpc-python/blob/master/zerorpc/heartbeat.py#L90

Let me know if you have any more question,

Best,
fx

> --
> You received this message because you are subscribed to the Google Groups
> "zerorpc" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to zerorpc+u...@googlegroups.com.
> To post to this group, send email to zer...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/zerorpc/524d80fb-4520-4fa5-bfbf-d9ca76ee680a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Gal Ben-Haim]

so the heartbeat event should have the "_zpc_hb" set as name, null args, a unique "message_id" header and the channel id in the "response_to" header ?

for some reason it doesn't work with my current implementation, sending such heartbeat doesn't do anything nor do I get any heartbeats.

but.. if I drop the "response_to" header, and simply send a new event with the "_zpc_hb" name, I do receive heartbeat events, and the Python server raises the remote lost exception..

any ideas what can be wrong ?

btw - it is suggested no to use package names such as zerorpc-go, see http://www.pkgname.com/

packages can be easily discovered at http://godoc.org

 

You received this message because you are subscribed to a topic in the Google Groups "zerorpc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zerorpc/-DDXRqI2aGk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zerorpc+u...@googlegroups.com.

To post to this group, send email to zer...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/CAG9r-DZhr_wxBXAUBCS4LuLCrfZiSNOjucg%2BeXNjvxBomR7-dA%40mail.gmail.com.

[François-Xavier Bourlet]

On Fri, Jun 6, 2014 at 1:46 PM, Gal Ben-Haim <gben...@gmail.com> wrote:
> so the heartbeat event should have the "_zpc_hb" set as name, null args, a
> unique "message_id" header and the channel id in the "response_to" header ?
>

Yep, an heartbeat is a simple event sent on the channel. Feel free to
use the python implemention as reference, and print events as they are
sent/received. For this, you can simply print events (there is an
__str__ overload).

You could print here:
https://github.com/bombela/zerorpc-python/blob/master/zerorpc/events.py#L265
And here: https://github.com/bombela/zerorpc-python/blob/master/zerorpc/events.py#L239

To trace all messages sent/recv by zerorpc :)

The response_to is basically the channel_id, which is based on the
first message ID that was sent to open the channel. Yes, the name
response_to was initially not really the notion of channel, but really
a single response_to.

client server
msg {id: 1} ->
<- msg {id: 42, response_to: 1}
msg {id: 2, response_to: 1} ->
<- msg {id: 43, response_to: 1}
etc

> for some reason it doesn't work with my current implementation, sending such
> heartbeat doesn't do anything nor do I get any heartbeats.
> but.. if I drop the "response_to" header, and simply send a new event with
> the "_zpc_hb" name, I do receive heartbeat events, and the Python server
> raises the remote lost exception..
>
> any ideas what can be wrong ?
>
>
> btw - it is suggested no to use package names such as zerorpc-go, see
> http://www.pkgname.com/
>
> packages can be easily discovered at http://godoc.org
>

Of course the package name should be zerorpc in go, but on github, the
repository could be named differently. It surely easier for people to
find zerorpc-go when looking up for zerorpc + go :) But do like you
want :)

> https://groups.google.com/d/msgid/zerorpc/CADm_bVCcEeYKc1YVMvWbm1HQCOOjy_zLSAxUJnUifT%2BnCporDw%40mail.gmail.com.

[Gal Ben-Haim]

the server doesn't seem to send any heartbeats to my client,

if I send an event with name "_zpc_hb" without assigning it to a channel, the server both sends heartbeats and raises the remote lost exception as needed..

what am I missing here ?

To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/CAG9r-DauZaz%3Dp28U6%2BH%3DOC2ORQCyqqMUiNjM71_GhXLUWz0hbw%40mail.gmail.com.

[François-Xavier Bourlet]

are you specifying the protocol version in your messages headers? { v: 3 }

If you send a _zpc_hb outside of a channel, this will be interpreted
as a method call by the server, that will open a new channel, and the
message will be consumed by the heartbeat mechanism, and the server
will be in fact waiting forever for your real request to come in.
Because the heartbeat is active, you are receiving the heartbeats
messages, and since you do not respond back to them, the server drop
the channel.

You can test this case easily:

Patch zerorpc python so it always prints the messages (seriously, we
should add that as a configurable option in zerorpc-python!)

diff --git a/zerorpc/events.py b/zerorpc/events.py
index c358951..bd2dc21 100644
--- a/zerorpc/events.py
+++ b/zerorpc/events.py
@@ -236,6 +236,7 @@ class Events(object):
return event

def emit_event(self, event, identity=None):
+ print '->', event
if identity is not None:
parts = list(identity)
parts.extend(['', event.pack()])
@@ -262,6 +263,7 @@ class Events(object):
event = Event.unpack(blob)
if identity is not None:
event.header['zmqid'] = identity
+ print '<-', event
return event

def setsockopt(self, *args):

Then run a server:
zerorpc --server --bind tcp://127.0.0.1:4242 time

And a client:
zerorpc tcp://127.0.0.1:4242 _zpc_hb --heartbeat 500

(the zerorpc cli can change the heartbeat value, but cannot disable
it, so I put an arbitrarily long heartbeat frequency. In code this
works: zerorpc.Client('tcp://127.0.0.1:4242', heartbeat=None))

So in conclusion: you are simply opening a new channel on the server
by sending a message outside of any existing channel.

> https://groups.google.com/d/msgid/zerorpc/CADm_bVB4Qh81PREWh2jmQ9dSPYDxRMqHUKPvV%2BUc0HrfmALW3w%40mail.gmail.com.

[Gal Ben-Haim]

I'm sending the protocol version..

What can cause the new channels created from a regular RPC event to not have heartbeats sent over by the server ?

To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/CAG9r-DYGit%2B0rw7fiDe2TvLN29gD9y7vJ5Ys57%3DLRDfqZ-i9xw%40mail.gmail.com.

[François-Xavier Bourlet]

If you dump here the messages received by the python server and the
one that you are sending from your go client, I can probably take a
look and try to guess what's wrong.

> https://groups.google.com/d/msgid/zerorpc/CADm_bVAt3%2Bb9AfcCrO0r45Y%2BnQywVO%2BfB5CL8hS%3DOp1j8DNQJg%40mail.gmail.com.

[Gal Ben-Haim]

this is the Golang event struct I'm sending:

&{map[message_id:597490bf-90a5-4ef6-7e2f-faadb3f04565 v:3] hello [[John]]}

packed MsgPack:

[147 130 170 109 101 115 115 97 103 101 95 105 100 218 0 36 53 57 55 52 57 48 98 102 45 57 48 97 53 45 52 101 102 54 45 55 101 50 102 45 102 97 97 100 98 51 102 48 52 53 54 53 161 118 3 165 104 101 108 108 111 145 145 164 74 111 104 110]

then I send heartbeats on the channel:

&{map[message_id:fbb88928-afae-4f19-6df6-6d8c7b5481a3 v:3 response_to:36efdac9-a0bd-4e80-6954-99bd76d8f536] _zpc_hb [<nil>]}

and packed MsgPack of the heartbeat event:

[147 131 170 109 101 115 115 97 103 101 95 105 100 218 0 36 102 98 98 56 56 57 50 56 45 97 102 97 101 45 52 102 49 57 45 54 100 102 54 45 54 100 56 99 55 98 53 52 56 49 97 51 161 118 3 171 114 101 115 112 111 110 115 101 95 116 111 218 0 36 51 54 101 102 100 97 99 57 45 97 48 98 100 45 52 101 56 48 45 54 57 53 52 45 57 57 98 100 55 54 100 56 102 53 51 54 167 95 122 112 99 95 104 98 145 192]

this is what's happening on the Python server:

<- hello {'zmqid': ['\x00O\xd1\xef{'], 'message_id': '36efdac9-a0bd-4e80-6954-99bd76d8f536', 'v': 3} [['John']]

-> OK {'response_to': '36efdac9-a0bd-4e80-6954-99bd76d8f536', 'message_id': 'fab6786c-eabe-4c21-b3e9-d3b59b56062a', 'v': 3} ("Hello, ['John']",)

<- _zpc_hb {'response_to': '36efdac9-a0bd-4e80-6954-99bd76d8f536', 'zmqid': ['\x00O\xd1\xef{'], 'message_id': 'fbb88928-afae-4f19-6df6-6d8c7b5481a3', 'v': 3} [None]

as you can see, no heartbeats are sent by the server..
(there's a 10 seconds time.sleep in the hello function on the server, so heartbeats should be sent..)

To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/CAG9r-DYnd8mpAZ9d5gpRy6OjK%2BC0UC6_t3D1%2BcmL3s9L8g1xwQ%40mail.gmail.com.

[Gal Ben-Haim]

the packed MsgPacks are byte arrays.,..

[François-Xavier Bourlet]

This is exactly what I described. You are sending a new ID, which in
turns open a new independent channel on the server.

On Sun, Jun 8, 2014 at 8:57 PM, Gal Ben-Haim <gben...@gmail.com> wrote:
> the packed MsgPacks are byte arrays.,..
>
>
> On Sat, Jun 7, 2014 at 10:43 AM, Gal Ben-Haim <gben...@gmail.com> wrote:
>>
>> this is the Golang event struct I'm sending:
>> &{map[message_id:597490bf-90a5-4ef6-7e2f-faadb3f04565 v:3] hello [[John]]}
>>

Ok, so this is a request creating the channel
"597490bf-90a5-4ef6-7e2f-faadb3f04565".
Any further message on this channel, from the server, or the client
should be using response_to: 597490bf-90a5-4ef6-7e2f-faadb3f04565 to
send message on this channel.

>> packed MsgPack:
>> [147 130 170 109 101 115 115 97 103 101 95 105 100 218 0 36 53 57 55 52 57
>> 48 98 102 45 57 48 97 53 45 52 101 102 54 45 55 101 50 102 45 102 97 97 100
>> 98 51 102 48 52 53 54 53 161 118 3 165 104 101 108 108 111 145 145 164 74
>> 111 104 110]
>>
>> then I send heartbeats on the channel:
>> &{map[message_id:fbb88928-afae-4f19-6df6-6d8c7b5481a3 v:3
>> response_to:36efdac9-a0bd-4e80-6954-99bd76d8f536] _zpc_hb [<nil>]}
>>

That is not the channel that you had already opened:
"36efdac9-a0bd-4e80-6954-99bd76d8f536"
The right response_to value should be "597490bf-90a5-4ef6-7e2f-faadb3f04565".

>> and packed MsgPack of the heartbeat event:
>> [147 131 170 109 101 115 115 97 103 101 95 105 100 218 0 36 102 98 98 56
>> 56 57 50 56 45 97 102 97 101 45 52 102 49 57 45 54 100 102 54 45 54 100 56
>> 99 55 98 53 52 56 49 97 51 161 118 3 171 114 101 115 112 111 110 115 101 95
>> 116 111 218 0 36 51 54 101 102 100 97 99 57 45 97 48 98 100 45 52 101 56 48
>> 45 54 57 53 52 45 57 57 98 100 55 54 100 56 102 53 51 54 167 95 122 112 99
>> 95 104 98 145 192]
>>
>> this is what's happening on the Python server:
>> <- hello {'zmqid': ['\x00O\xd1\xef{'], 'message_id':
>> '36efdac9-a0bd-4e80-6954-99bd76d8f536', 'v': 3} [['John']]

You sent "597490bf-90a5-4ef6-7e2f-faadb3f04565" and the server
received "36efdac9-a0bd-4e80-6954-99bd76d8f536"??? How's that
possible? Maybe by mistake you rewrite your ids?

>> -> OK {'response_to': '36efdac9-a0bd-4e80-6954-99bd76d8f536',
>> 'message_id': 'fab6786c-eabe-4c21-b3e9-d3b59b56062a', 'v': 3} ("Hello,
>> ['John']",)

The server is correct, and replies to the right channel:
"36efdac9-a0bd-4e80-6954-99bd76d8f536"

>> <- _zpc_hb {'response_to': '36efdac9-a0bd-4e80-6954-99bd76d8f536',
>> 'zmqid': ['\x00O\xd1\xef{'], 'message_id':
>> 'fbb88928-afae-4f19-6df6-6d8c7b5481a3', 'v': 3} [None]
>>

And again... "36efdac9-a0bd-4e80-6954-99bd76d8f536" until it considers
the peer not responding.

So, something is fishy here. The ID that you are sending is not the
one received on the server.

[Gal Ben-Haim]

this log indeed looks strange,

I run it again:

client log:

2014/06/11 06:59:09 ZeroRPC socket connected to tcp://0.0.0.0:4242

2014/06/11 06:59:09 ZeroRPC client invoked hello with args [John]

2014/06/11 06:59:09 ZeroRPC socket listening for incoming data

2014/06/11 06:59:09 ZeroRPC socket created new channel 

2014/06/11 06:59:09 Channel 408da07d-99c0-4202-5047-5d26446cf1f8 sending event 408da07d-99c0-4202-5047-5d26446cf1f8

2014/06/11 06:59:09 ZeroRPC socket sent event 408da07d-99c0-4202-5047-5d26446cf1f8

2014/06/11 06:59:09 ZeroRPC socket send 67 bytes

2014/06/11 06:59:14 Channel 408da07d-99c0-4202-5047-5d26446cf1f8 sending event 6521c017-d294-4a8b-5d5f-25dfa565bcc1

2014/06/11 06:59:14 ZeroRPC socket sent event 6521c017-d294-4a8b-5d5f-25dfa565bcc1

2014/06/11 06:59:14 ZeroRPC socket send 115 bytes

2014/06/11 06:59:14 Channel 408da07d-99c0-4202-5047-5d26446cf1f8 sent heartbeat

2014/06/11 06:59:17 ZeroRPC socket received 122 bytes

2014/06/11 06:59:17 ZeroRPC socket recieved response event b9311440-c18e-43df-b049-77fb44395656

2014/06/11 06:59:17 ZeroRPC socket routing event b9311440-c18e-43df-b049-77fb44395656 to channel 408da07d-99c0-4202-5047-5d26446cf1f8

2014/06/11 06:59:17 Channel 408da07d-99c0-4202-5047-5d26446cf1f8 closed

2014/06/11 06:59:17 ZeroRPC socket closed

server log (has time.sleep(8)):

<- hello {'zmqid': ['\x00|\x06\xe0^'], 'message_id': '408da07d-99c0-4202-5047-5d26446cf1f8', 'v': 3} ['John']

-> OK {'response_to': '408da07d-99c0-4202-5047-5d26446cf1f8', 'message_id': 'b9311440-c18e-43df-b049-77fb44395656', 'v': 3} ('Hello, John',)

<- _zpc_hb {'response_to': '408da07d-99c0-4202-5047-5d26446cf1f8', 'zmqid': ['\x00|\x06\xe0^'], 'message_id': '6521c017-d294-4a8b-5d5f-25dfa565bcc1', 'v': 3} None

it looks fine to me message_id and response_to wise..

still the server doesn't send hearbeats

--
You received this message because you are subscribed to a topic in the Google Groups "zerorpc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zerorpc/-DDXRqI2aGk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zerorpc+u...@googlegroups.com.
To post to this group, send email to zer...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/CAG9r-DYuMkwd_Z%2BTT6dku5G%2B-ffkncsO6L0t8FkRTNjF3hCFPQ%40mail.gmail.com.

[François-Xavier Bourlet]

After sending the message "-> OK " then the server closes the channel.
Don't expect receiving anything on a closed channel.

If you add a gevent.sleep(30) in the hello method on the server.
You will see an exchange of heartbeats. But in your case, it looks
like you are getting the response faster than the heatbeat frequency.

> You received this message because you are subscribed to the Google Groups
> "zerorpc" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to zerorpc+u...@googlegroups.com.
> To post to this group, send email to zer...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/zerorpc/CADm_bVAx_zxGiD2DRBPTi84nqN_XF-PmNM5g28Ej4k1pjAPYAg%40mail.gmail.com.

[Gal Ben-Haim]

I finally found the problem, I used time.sleep in the Python server,

but didn't monkey patch it... so it blocked everything..

To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/CAG9r-DYQ_iGPJ7q5g4F4LE5WMC2CnrG14zzvG%2BoFO4emkib3NA%40mail.gmail.com.

[Gal Ben-Haim]

check out my lib - https://github.com/bsphere/zerorpc

I think it supports most of the client functionality by now (channels, streaming responses, heartbeats, first class exceptions)