Message retransmission on shared topics

[José Micó]

Given the following scenario:

  1. Three clients (A, B and C) connect to VerneMQ using Protocol Version 5
  2. Client A subscribes to topic '$share/GROUP/foo', with Maximum QoS 1
  3. Client B subscribes to topic '$share/GROUP/foo', with Maximum QoS 1
  4. Client C publish a message to topic 'foo', with QoS 1

Lets say that Client A receives the message (just by chance).

Then, before Client A acknowledges the message, if any of...

  a) Client A abruptly closes the TCP connection
  b) Client A sends a DISCONNECT packet, the closes the TCP connection
  c) Client A sends a PUBACK packet with Reason Code 0x83

...I expect the message to be retransmitted to client B.

But the message doesn't seem to be retransmitted (I am testing on 1.11.0).

Is this the expected behavior? Is it possible to configure VerneMQ in some way to avoid message loss?

Thanks in advance,
José

[André Fatton]

Hi José,

This is expected behaviour, yes.

The reason for this is that between a shared subscriber and the server all the client-server session state handling still applies. (btw you did not mention whether those sessions use a clean_session, or not).

So, if Client A re-connects, it would find that message in its offline queue. That is, the way to avoid message loss here is to use persistent sessions. You will have out-of-order delivery for those messages.

I see why this might seem non-intuitive: the routing (determined by the subscriber policy) is decoupled from the session state handling. As a consequence, a message is not re-routed if it ends up in an offline queue.

Cheers,

André

[José Micó]

Thanks for your reply, it has pointed me in the right direction.

The case of PUBACK with code 0x83 is answered in the section 4.8.2 of the spec: "If a Client responds with a PUBACK or PUBREC containing a Reason Code of 0x80 or greater to a PUBLISH packet from the Server, the Server MUST discard the Application Message and not attempt to send it to any other Subscriber".

In the disconnection case, the messages should be resent: "If the Client's Session terminates before the Client reconnects, the Server SHOULD send the Application Message to another Client that is subscribed to the same Shared Subscription".

I will explain a bit what I am trying to do. I am trying to port a distributed task queue to VerneMQ. So "tasks" are published to a shared topic, and the broker balances the load among a pool of equipotent "workers" which consume and process these tasks.

When a "task" is published to the topic (with QoS 1) it should be sent as quick as possible to a (preferably idle) "worker" client subscribed to the topic (with QoS 1). After the task is processed, the "task" message is acknowledged signaling the broker to send more "tasks" messages.

If a worker crashes, any "task" assigned to it (in flight messages) should be sent to another "worker" client.

So I am using:

- "Session Expiry Interval" = 0 (to make the session ends when the network connection is closed)

- "Clean Start" = 1 (start a new session)

- "Receive Maximum" = 1  (so no messages are sent to busy clients)

But it doesn't work: messages are not being retransmitted if the network connection is closed. I guess that the in flight messages already assigned to the expired session are not reassigned to another client. This may or may not be a bug, based on the opininon about what "SHOULD" means... I suggest calling this a "non fully supported feature" and move on :)

Waiting anxiously for VerneMQ 2.0,

José

[André Fatton]

Thanks, José, nice assessment & write-up. And yes, there's a huge difference between MUST and SHOULD in those MQTT specs :)

From an implementor's perspective it's about technical complexity vs. gained benefit.

With message replay at session expiry, you'd have to store provenance for every message in the queue (over which subscription match did the message end up in the queue).

Best,

A.