Disable logging of MQTT messages

[Daniel Goepp]

I'd like to turn down logging a bit, but am not seeing a way to do what I want. I want to log the application output normally, so I don't want to just turn it all off completely, but I don't need every MQTT message published to get logged. I tried various levels of verbosity, but none seem to affect this. I am running 21.12-60-g5f0ff6db branch master at 202202121409. My example would be:

Feb 13 07:18:25 rtl-acurite rtl_433[16986]: Exact sample rate is: 250000.000414 Hz
Feb 13 07:18:26 rtl-acurite rtl_433[16986]: [R82XX] PLL not locked!
Feb 13 07:18:26 rtl-acurite rtl_433[16986]: Sample rate set to 250000 S/s.
Feb 13 07:18:26 rtl-acurite rtl_433[16986]: Tuner gain set to Auto.
Feb 13 07:18:26 rtl-acurite rtl_433[16986]: Tuned to 433.920MHz.
Feb 13 07:18:26 rtl-acurite rtl_433[16986]: MQTT Connected...
Feb 13 07:18:27 rtl-acurite rtl_433[16986]: MQTT Connection established.
Feb 13 07:18:30 rtl-acurite rtl_433[16986]: MQTT Message publishing acknowledged (msg_id: 1)
Feb 13 07:18:30 rtl-acurite rtl_433[16986]: MQTT Message publishing acknowledged (msg_id: 2)
Feb 13 07:18:30 rtl-acurite rtl_433[16986]: MQTT Message publishing acknowledged (msg_id: 3)

Any thoughts are appreciated. Thanks in advance.

[Christian Z.]

From the "-F mqtt" output remove the "qos" option.

[Daniel Goepp]

Oh interesting! Thank you! I would not have thought that QOS would have affected that, but once you point it out, I refined my search and see conversations on this now. I was wondering about that since the docs on https://triq.org/rtl_433/#running show it as "n" when normally qos is 0|1|2, so I just assumed N = 0. I then found the conversation you had with someone else on this, and it sounds like just removing is the right thing to do for now. https://giters.com/merbanan/rtl_433/issues/1937. I will investigate further, but yes for now your recommendation did exactly what I want.

[Christian Z.]

Yes, sorry, the QoS levels are not fully implemented but rather just show the protocol steps verbosely. More of a "show me what MQTT is doing".

The trouble is that we would need to retain a list of in-flight messages and run a retry timer. The framework is not set up to support that easily.

[Greg Troxel]

"Christian Z." <chri...@zuckschwerdt.org> writes:

> The trouble is that we would need to retain a list of in-flight messages
> and run a retry timer. The framework is not set up to support that easily.

It seems obvious that rtl_433 should be using an existing mqtt library
rather than being homegrown, to avoid having to deal with all of this.

I am fuzzy, but I remember finding the builtin mqtt support not enough
for what I wanted (no TLS?) and stared using the json-over-UDP to the
contrib mqtt reporting python script, which uses paho. I need to
return to that and improve it and send changes back.

[Christian Z.]

We do use a network library, which comes with e.g. MQTT support: Mongoose.

But (like most things C) it is pretty low-level. TLS is supported and enabled in rtl_433 for some time now -- the piece needed was the certificate setup, parsing options and retaining them.

Similar with MQTT QoS, keeping sent messages around is supported, as are timers. But we'd need to manage memory for that and handle timeout/resend logic.

Currently messages are on the stack, so if QoS, copy to heap, keep a list, remove from list or resend until some retry limit, and the hard question: what to do on repeated fail? Warn and give up? Scale the retry time indefinitly? What if the retry list fills up?

All very much do-able, but opinionated on the use-case. (there a are good reason to "just want QoS", but what exactly is the detailed behaviour needed?)

I guess you never get anything for free (in the sense that it happens like magic), in C, except for, well, very fine grained control over things ;)

[Greg Troxel]

"Christian Z." <chri...@zuckschwerdt.org> writes:

> We do use a network library, which comes with e.g. MQTT support: Mongoose.
> But (like most things C) it is pretty low-level. TLS is supported and
> enabled in rtl_433 for some time now -- the piece needed was the
> certificate setup, parsing options and retaining them.
>
> Similar with MQTT QoS, keeping sent messages around is supported, as are
> timers. But we'd need to manage memory for that and handle timeout/resend
> logic.
> Currently messages are on the stack, so if QoS, copy to heap, keep a list,
> remove from list or resend until some retry limit, and the hard question:
> what to do on repeated fail? Warn and give up? Scale the retry time
> indefinitly? What if the retry list fills up?

I see - the issue is that the mqtt library is too low level. I have
written some (very simple) code in python with the paho python library,
and it deals with all of this for me.

> All very much do-able, but opinionated on the use-case. (there a are good
> reason to "just want QoS", but what exactly is the detailed behaviour
> needed?)

That's a much deeper question. MQTT is pubsub, not state transfer and
not database sync. One could desire that every report generated by
rtl_433 should appear in some remote database, and that leads to writing
it locally and dropping it only when it is acked by the rmeote database
(as committted to stable storage), vs the broker.

With qos 1, you avoid loss due to network breaking (sending into a TCP
connection that has failed, but that you don't know has failed), but I
don't see much point with qos 2, absent a design to do something more
complicated on top of it. However, I'm not sure qos1 messages survive
broker disconnect, even though to me obviously that's the point.

> I guess you never get anything for free (in the sense that it happens like
> magic), in C, except for, well, very fine grained control over things ;)

C libraries could certainly be written that manage all that and hook
into the event loop (or have their own thread). But yes, there's a
complexity tradeoff. I'm not trying to say you should, just that "using
a C library" and "have to do it all yourself" aren't necessarily so
tightly linked.


But fair enough that this is all hard given how things are now.

[Daniel Goepp]

This makes sense. Honestly since all of this is based on receiving radio signals that are not guaranteed to start with, I don't see how someone could justify much more than qos 0 ;) Just my two cents. I really don't care if I miss a bunch of packets. However, I'm sure others have more interesting use cases than mine where this is necessary. I actually do the opposite on the the sub side of MQTT. I publish all the events from rtl_433, and then I sub with a python script that drops all retransmissions and then pops it back on another topic for telegraph/influx and home assistant to ingest, nice and cleaned up. My atlas weather station sends every message three times. At least they include a seq number so it's easy to clean.

[Greg Troxel]

Daniel Goepp <d...@goepp.com> writes:

> This makes sense. Honestly since all of this is based on receiving radio
> signals that are not guaranteed to start with, I don't see how someone
> could justify much more than qos 0 ;) Just my two cents. I really don't
> care if I miss a bunch of packets. However, I'm sure others have more
> interesting use cases than mine where this is necessary. I actually do the
> opposite on the the sub side of MQTT. I publish all the events from
> rtl_433, and then I sub with a python script that drops all retransmissions
> and then pops it back on another topic for telegraph/influx and home
> assistant to ingest, nice and cleaned up. My atlas weather station sends
> every message three times. At least they include a seq number so it's easy
> to clean.

Interesting point about dedup.

I see your point about radio and reliability but radio loss mechanism
are different from network loss mechanisms. Not rtl_433, but I have a
Davis weather station, outside RF to console. But the console has
batteries and non-voltatile storage, so if computer/intenet is down for
a few days, I end up with all the data afterwards. That's the sort of
thing I was getting at with database sync.

[Daniel Goepp]

On Sunday, February 13, 2022 at 3:16:03 PM UTC-5 Greg Troxel wrote:

Interesting point about dedup.

I see your point about radio and reliability but radio loss mechanism
are different from network loss mechanisms. Not rtl_433, but I have a
Davis weather station, outside RF to console. But the console has
batteries and non-voltatile storage, so if computer/intenet is down for
a few days, I end up with all the data afterwards. That's the sort of
thing I was getting at with database sync.

That's it, I'm running ethernet out to my weather station! :) My wife is going to ask why I'm hitching up the backhoe again.

[Christian Z.]

That's it, I'm running ethernet out to my weather station! :)

(paraphrasing Ripley) "Cable the site with ethernet. It's the only way to be sure. "

[Jörg Hedtmann (GMX)]

Not really, I had cables gnawed by moles... 😎

Get BlueMail for Android

[Christian Z.]

had cables gnawed by moles...

High powered optical communication then? In essence, a sophisticated heat beam which we call a "laser".