Native MQTT / 3.12.0-rc.1 timestamp

[guillaume blessas]

Hi, I modded rabbitmq-message-timestamp to be usable with 3.12.0-rc.1 and MQTT Native qos0.

In my actual 3.11 use case i collect iot information with MQTT plugin and process them with amqp connections.

I use publish time of received MQTT messages as timestamp with plugins message_timestamp_in_ms.

My problem is the following: 

- in mqtt there is no timestamp in messages.

- plugin message time stamp is base on channel interceptor  and it seeems MQTT native plugin does not declare channels.

How could my rabbitmq server add server referenced timestamp to messages collected in mqtt with 3.12 ? 

Is it possible with 3.11 and i wish 3.12 to get timestamp of messages in MQTT without modifying payload ?

Regards,

[David Ansari]

Hi Guillaume,

thanks for testing Native MQTT in 3.12.0-rc.1 and reporting this issue.

Any plugins that implement rabbit_channel_interceptor will break with Native MQTT as Native MQTT will not use rabbit_channel anymore.

To not break your use case, we moved the rabbitmq-message-timestamp plugin to rabbitmq-server core.

As described in https://github.com/rabbitmq/rabbitmq-message-timestamp#supported-rabbitmq-versions, starting with 3.12.0 (or 3.12.0-rc.3) disable the plugin and include the following in your rabbitmq.conf file:

message_interceptors.incoming.set_header_timestamp.overwrite = false

This will add the timestamps to incoming messages by both the channel process and the MQTT connection process (behaving exactly like the plugin being enabled in 3.11).

Cheers

David

[guillaume blessas]

Hi David,

Nice answer and feature !! 1 plugin less and backward compatibility !

Still related with 3.12 features and MQTT.

 Some of my use cases are more looking like recurrent events dispatch with no storage needed and no acknowledge.

I read somewhere that this kind of "fake queue" will be present in MQTT in 3.12.

Will it be possible to have same feature on other topic exchange access (AMQP) ?

My need is to maximize throughput and latency.

Regards,

Guillaume

[David Ansari]

That's a good question. I think such a queue type could also be used in future for consumers of other protocols (including AMQP 0.9.1) who consume with "at-most-once" semantics and whose session only lasts as long as the network connection meaning the "fake queue" will be auto deleted once the consumer disconnects. (The queue is exclusive to that consumer connection). In essence the queue process and writing the message to the disk will be skipped. It's a bit described in https://blog.rabbitmq.com/posts/2023/03/native-mqtt/#queue-type-naming that we might add such a feature in the future.

For now, let's see how that new queue type proves in practice with Native MQTT in 3.12 before allowing other protocol clients to create such queues.

[guillaume blessas]

Hi, still according to my use case il may replace amqps connection with mqtt, but i'm lacking publisher timestamp. Is there a way to get this timestamp available to mqtt client ?

Regards,

guillaume.

[David Ansari]

So, you want the RMQ broker to add a timestamp header and then send this header to an MQTT subscriber?

This is not possible with MQTT 3.1.1 as the PUBLISH packet does not allow for such headers, see https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718037

The MQTT 5.0 protocol is a bit more flexible in that regard.

[guillaume blessas]

And will mqtt 5 be included in 3.12 future update or 3.13 ?

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/x8ACs4dBlkI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/29436d57-2d7b-483c-9ed6-10b24c0155e3n%40googlegroups.com.

[David Ansari]

MQTT 5.0 will not be part of 3.12.x. It will be released in 3.13 earliest (possibly later).

Where in the PUBLISH Variable Header would you put the timestamp value set by the server? https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901106

User Property might come closest, however:

"The Server MUST send all User Properties unaltered in a PUBLISH packet when forwarding the Application Message to a Client [MQTT-3.3.2-17]."

Therefore, even though the MQTT 5.0 protocol is a bit more flexible with more headers, whether we can include a server set timestamp into the headers is up for discussion.

Strictly following the spec, we cannot include server set headers into the PUBLISH packet.

Note that User Properties in other packet types allow interpretation between Server and Client:

User Property in the CONNECT packet:

"User Properties on the CONNECT packet can be used to send connection related properties from the Client to the Server. The meaning of these properties is not defined by this specification."

User Property in the PUBACK packet:

"This property can be used to provide additional diagnostic or other information."

User Property in the SUBSCRIBE packet:

"User Properties on the SUBSCRIBE packet can be used to send subscription related properties from the Client to the Server. The meaning of these properties is not defined by this specification."