Introducing a robust MQTT publisher for rtl_433

[Helge Weissig]

After dabbling some with Home Assistant and MQTT, I have finally come to see the benefits of the latter for rtl_433 reporting. 

For reference, in my current setup I am parsing rtl_433 with a perl script and pushing the data both to a MariaDB and an InfluxDB database. Since I have had some network issues lately, I have tried to introduce some robustness to this by running a MariaDB instance on my RPi0 that is in charge of running rtl_433. On my main "monitoring" server, a slave instance of MariaDB and the InfluxDB patiently wait for data from the RPi0, if either device should be offline. Over the past few years (I have been collecting rtl_433 data consistently for more than 4 years now), especially the MariaDB has grown substantially and running a copy of it on the RPi0 has added more of a load than the poor little guy is really happy with (zabbix sends me at least on "disk is overloaded" alert per day).

So, I finally had a longer look at MQTT and its capabilities after I started using some of my weather data in Home Assistant. After all, it claims to do just what I need here: according to the golden reference, mqtt is "a lightweight messaging protocol for small sensors and mobile devices, optimized for high-latency or unreliable networks." Sure, I had seen the basic `rtl__433 -F json | mosquitto_pub ...` example before, but wasn't really sure how that would be better than my treasured perl script.

To make a long story short, I wanted a little more of the publisher and wrote a little tool that does just that: a) fully asynchronous connection and publication handling, b) resilience and persistence of data across client and/or network disconnects, c) publication of devices to their own topics under a main subtopic.

The tool is written in java and is available here: https://github.com/helgew/mqtt4rtl433 . It's still a bit bare-bones, really, but it does the trick for me. There are no releases yet, but all you will need to compile the code is a Java 8+ SDK installation and Apache Maven. 

Would love to get your feedback on this!

cheers,

h.

[Helge Weissig]

I did see Christian’s PR to embed mqtt access in rtl_433. Personally, I am not a big fan of that idea at all as it goes against the “do one thing and do it really well” philosophy. Looking at the  implementation, I cannot tell off-hand how it would handle network issues, for example.

There have been discussions on this list about adding DB connectivity and such before and inevitably, the argument was that those went beyond the rtl_433 scope. I am surprised to see mqtt “on the road map.” Here is the last example I remember: https://groups.google.com/d/msg/rtl_433/dUMXV8YQgkQ/bdfqmkoGDAAJ. 

 

cheers,

h.

On Apr 2, 2019, at 7:27 PM, Chris Weakland <chris.w...@gmail.com> wrote:

I ended up doing this:  https://community.home-assistant.io/t/ge-concord-interlogix-rtl-sdr-mqtt/107910

Read the last note in this thread, it looks like we may have an easier implementation of this soon.

--
You received this message because you are subscribed to the Google Groups "rtl_433" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.
To post to this group, send email to rtl...@googlegroups.com.
Visit this group at https://groups.google.com/group/rtl_433.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/a97ccd22-3a96-4a08-bf5b-041b01889245%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Woodhouse]

This seems like a good idea to me. Although you lost me at 'Java'. I'm
doing this on an OpenWRT router with 128MiB of RAM.

I also need to convert the rtl_433 output into MQTT suitable for
feeding directly to Domoticz as 'commands' on the domoticz/in channel.
So what I have now looks like this:


rtl_433 -R 43 -F json -l 0 -q | tee /tmp/tankmon-out-$$ |
while read line ; do
if [[ "$line" =~ \"id\"\ :\ 676252812,.*\"temperature_C\"\ :\ ([0-9.]+),.*\"depth\"\ :\ ([0-9]+)} ]]; then
echo -e "{ \"command\" : \"udevice\", \"idx\" : 16, \"svalue\" : \"${BASH_REMATCH[1]}\" }
{ \"command\" : \"udevice\", \"idx\" : 15, \"svalue\" : \"${BASH_REMATCH[2]}\" }" |
mosquitto_pub -h wndr3800.infradead.org --cafile /etc/ssl/certs/ca-certificates.crt -t domoticz/in -u domoticz -P e55542ec115c -p 1883 -l
fi
done

Since this particular setup gets readings only every 35 minutes, it
doesn't leave mosquitto_pub running.

I'd be happy with doing the MQTT connection directly from rtl_433
although then I'd either need to be able to tweak precisely what it
sends and to what channel, or I'd need an external agent to do that
mangling and feed it to Domoticz. But at least there would be fewer
moving parts with the data collection — it's started by an initscript,
and could more reliably quit and be restarted when it's stopped
working.

[Christian Zuckschwerdt]

If it works well for you or you need a special translation layer then by all means use custom bridges. (It's expected that the "raw" MQTT output needs to be supplemented by some scripts for various auto-discovery mechanisms.) We now have a folder "examples" to give some starting point for that. Happy to include other scripts, perhaps DB bridges -- though TSDB are recommended with this kind of data. (and light weight solutions, say Python, Shell, Perl, are better suited as examples. Last time I looked the JVM takes quite a generous chunk of memory, even compared to script languages.)

rtl_433 had a UDP output for quite some time and that's now joined by TCP outputs for reliability. MQTT is the most wished for but things like e.g. REST, JSONRPC and Webhooks will follow.

We always try hard to keep rtl_433 lean and focused, but network connectivity is needed, pipes are just too bare and don't fit well, e.g. people do want to easily use NodeRED -- which I see as a great way to start sensor and automation projects fast.

It would be great if we could collect interesting projects like this mqtt4rtl433 for a wiki page or Readme section. Just dump links with a short description here or in issue https://github.com/merbanan/rtl_433/issues/1033

[Greg Troxel]

Christian Zuckschwerdt <chri...@zuckschwerdt.org> writes:

> We always try hard to keep rtl_433 lean and focused, but network
> connectivity is needed, pipes are just too bare and don't fit well, e.g.
> people do want to easily use NodeRED -- which I see as a great way to start
> sensor and automation projects fast.

I understand most of your points, but what's wrong with a helper program
where one runs rtl_433 with json output and pipes that into the helper?
That seems like the right narrow interface, and other than json
serialization I don't really see any issues.

Are you suggesting a way to have rtl_433 be reconfigured over mqtt, or
just that linking in an mqtt library is somehow better than pipes
(compared to the downside of adding dependencies), or ?


And definitely agreed that Java is awkward in many places, due to memory
requirements. Plus, there will be places where e.g. python runs but
where java hasn't been ported.

[Christian Zuckschwerdt]

Either side of a pipe breaks or stalls the whole thing will go down. Also fan-out and the like will be difficult. Clean separation over a socket is a better choice for daemons.

We tried out linking to Paho and Mosquitto, but that's not the current implementation. MQTT currently is completly built-in. It's not a complicated protocol. Sockets are not much harder than printf(), just a little setup and state.

If you roll your own solution have a look at the _relay.py scripts in exampes. UDP is really is no code at all.

[Greg Troxel]

Christian Zuckschwerdt <chri...@zuckschwerdt.org> writes:

> Either side of a pipe breaks or stalls the whole thing will go down. Also
> fan-out and the like will be difficult. Clean separation over a socket is a
> better choice for daemons.

That's a fair point; I was thinking of pipe vs include library.

> We tried out linking to Paho and Mosquitto, but that's not the current
> implementation. MQTT currently is completly built-in. It's not a
> complicated protocol. Sockets are not much harder than printf(), just a
> little setup and state.

It would be nice to have them as AF_UNIX, so users can manage
permissions. I'll have a look at things - have been not updating for
while.

> If you roll your own solution have a look at the _relay.py scripts in
> exampes. UDP is really is no code at all.

Thanks - what I think I want is mqtt, so I hope to reuse.

[Helge Weissig]

> Either side of a pipe breaks or stalls the whole thing will go down.

I think the argument against pipes does not hold very well, Christian. If the left side goes down (rtl_433 in this case), it would presumably go down for the same reason with baked in support for someone’s favorite network/DB communication protocol. One could even make the argument that the more complex rtl_433 gets, the more likely that event becomes.

> Also fan-out and the like will be difficult.

Maybe I don’t understand your point here, but how do you enable “fan out” with your current mqtt implementation.

> Clean separation over a socket is a better choice for daemons.
>
> We tried out linking to Paho and Mosquitto, but that's not the current implementation. MQTT currently is completly built-in. It's not a complicated protocol. Sockets are not much harder than printf(), just a little setup and state.

Yes, socket communication might be more robust, but it should not be done using one specific communication protocol. What was wrong with the UDP/TCP implementation that was there already?

The fact that you are not using the paho C library is another weak point. Yes, sockets might not be hard, but robust and resilient communication across sockets in an unreliable and resource-strapped deployment environment is not easy either.

Again, with plans on supporting a REST API, pull requests for WebSocket implementations, and so forth, rtl_433 is going very far beyond its original scope. Maybe focusing on a pluggable architecture and/or a robust API with example implementations might be a better approach. As it is, the number of command line options already is dazzling and the current mqtt implementation lacks support for SSL, certificates, or full client-side QOS support.

Obviously, anyone can “roll their own solution” and it’s nice to see some examples being added to the project. However, when open source projects like rtl_433 get overloaded with features that really only work for a few, they tend to eventually go the way of the dodo bird.

To keep it constructive, let me reiterate and extend on my suggestion above:

- let’s keep rtl_433 focused on what it does best: discover and decode device packets and communicate the data in a standardized data format.
- break out existing functionality into separate domains with separate implementations of a discovery tool and a full-featured decoding tool for production deployment
- build a “plugin” API that can be used to add functionality but is configurable by the end user so as to not bloat the production decoding tool

I haven’t been active in this project for a while and do not claim any kind of authority over roadmap decisions, but it would be great to see at least a semblance of a process for those.

cheers,
h.

[Peter Smith]

I must add my support to what Helge says. For my own project I stripped out a vast amount of 'features' that really do not belong and added an API based on JSON/RPC. The core functions of rtl_433 are really well implemented and it should stick to this while providing a nice, well documented API that others can use to extend its functionality.

[Benjamin Larsson]

On 2019-04-03 16:45, Greg Troxel wrote:
> I understand most of your points, but what's wrong with a helper program
> where one runs rtl_433 with json output and pipes that into the helper?

On some platforms it might not be possible to run any helper programs.
Running rtl_433 on routers for example has always been a goal and
embedding mqtt output will help realizing that.

Making it more easy for users to run rtl_433 is probably worth it in the
long run for the project.

MvH
Benjamin Larsson

[Christian Zuckschwerdt]

I think the argument against pipes does not hold very well, Christian. If the left side goes down (rtl_433 in this case),

More likely a problem with the recevier ;) But we just need to restart rtl_433 (Supervisord) and the state in the consumer is kept, it even gets a chance to record and report outage.

 

Maybe I don’t understand your point here, but how do you enable “fan out” with your current mqtt implementation.

E.g. I run multiple rtl_433 instances that each connect to multiple consumers (TSDB, MQTT, …) -- an N:M setup. Pipes only offer a single connection unless you go to serious trouble with redirects.

 

What was wrong with the UDP/TCP implementation that was there already?

We only had UDP, it's still there. TCP-wise MQTT was the single most requested feature for a long time. But a simpler TCP option, a socket with a stream of events, alternativly in HTTP semantics will follow.

 

The fact that you are not using the paho C library is another weak point.

I don't see where Paho offers more than Mongoose for our limited scope. The Paho/Mosquitto branch is still there but I don't think I want to support that. Maybe as an example with the API (say "librtl_433") later.

 

Maybe focusing on a pluggable architecture and/or a robust API with example implementations might be a better approach.

Working on that.

 

As it is, the number of command line options already is dazzling and the current mqtt implementation lacks support for SSL, certificates, or full client-side QOS support.

With the upcoming API'ization there are ideas to create smaller and more focused CLIs. TLS can be added, it needs infrastructure to define and handle certs and keys.

 

- let’s keep rtl_433 focused on what it does best: discover and decode device packets and communicate the data in a standardized data format.

That's still the idea. The catch is that there is an ecosystem of depended projects and we do want to align format and communication to make integration easy.

E.g. NodeRED, start rtl_433 up and configure an MQTT input. It really shouldn't be any more complicated.

 

- break out existing functionality into separate domains with separate implementations of a discovery tool and a full-featured decoding tool for production deployment

It's on the roadmap. When "librtl_433" is ready we could refactor major outputs into separate tools.

 

- build a “plugin” API that can be used to add functionality but is configurable by the end user so as to not bloat the production decoding tool

Something like that will be in the "librtl_433"-API approach. Single purpose programs then can configure an rtl_433 "core" and use that programmatically.

Thanks for your ideas, always a good chance to sort though some vague concepts.

[Peter Smith]

To continue, since I feel quite strongly about this.

In my opinion, the function of rtl_433 should be to make available a message body, the 'payload' to other cooperating programs, via an API. The 'meaning' of the message body is no business of rtl_433, whether it be temperature, pressure, wind speed, an event, or anything else. The client, requesting or invoking program should decode the message body and decide on its 'meaning', taking any appropriate action.

[Helge Weissig]

> Running rtl_433 on routers for example has always been a goal and embedding mqtt output will help realizing that.

Now that’s just crazy talk!! :D

Routers are for networking! What’s next? Run rtl_433 from your smart toaster?? (I jest, but there is a kernel of truth in that humor!!)

> Making it more easy for users to run rtl_433 is probably worth it in the long run for the project.

Fully agree! IMHO, making rtl_433 do EVERYTHING and adding tons of features not everyone needs or wants is not the way to go though.

cheers,
h.

> --
> You received this message because you are subscribed to the Google Groups "rtl_433" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.

> To post to this group, send an email to rtl...@googlegroups.com.

> Visit this group at https://groups.google.com/group/rtl_433.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/1834bfce-88a7-a7ca-4e76-fa8a253a2234%40ludd.ltu.se.

[Benjamin Larsson]

On 2019-04-03 19:30, Helge Weissig wrote:
>> Running rtl_433 on routers for example has always been a goal and embedding mqtt output will help realizing that.
>
> Now that’s just crazy talk!! :D
>
> Routers are for networking! What’s next? Run rtl_433 from your smart toaster?? (I jest, but there is a kernel of truth in that humor!!)
>
>> Making it more easy for users to run rtl_433 is probably worth it in the long run for the project.
>
> Fully agree! IMHO, making rtl_433 do EVERYTHING and adding tons of features not everyone needs or wants is not the way to go though.
>
> cheers,
> h.

Not sure if I understand you correctly but rtl_433 has been ported to
OpenWRT since quite some time. Many routers have usb ports so combining
rtl-sdr with a router is totally feasible. The problem with routers is
that they often are feature limited. Thus you have to provide everything
you need on your own.

MvH
Benjamin Larsson

[Christian Zuckschwerdt]

btw. I'm running rtl_433 *on* a receiver -- the ADALM Pluto SDR…

[Simon Eigeldinger]

Hi,

On Windows you are also lost because many unix tools are not available
as well.
And you would need to rewrite or recompile all the subtools if splitted.

Greetings,
Simon

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

[Helge Weissig]

Just because it can be done does not mean it *should* be done! :)

Seriously though, I am all for making rtl_433 work on resource-limited platforms! Consumer-grade WiFi routers are actually good candidates, although with RPi0’s costing just a few bucks, why go through all the hassle of flashing your router and trying to deploy software to it?

cheers,
h.

> --
> You received this message because you are subscribed to the Google Groups "rtl_433" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.
> To post to this group, send an email to rtl...@googlegroups.com.
> Visit this group at https://groups.google.com/group/rtl_433.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/87f04b3e-4482-83c3-74e6-4fe1fd99d078%40ludd.ltu.se.

[Benjamin Larsson]

On 2019-04-03 19:39, Helge Weissig wrote:
> Just because it can be done does not mean it *should* be done! :)
>
> Seriously though, I am all for making rtl_433 work on resource-limited platforms! Consumer-grade WiFi routers are actually good candidates, although with RPi0’s costing just a few bucks, why go through all the hassle of flashing your router and trying to deploy software to it?
>
> cheers,
> h.

Because I can. And routers are free for me (related to my dayjob) thus a
more suitable platform in my case.

MvH
Benjamin Larsson

[Helge Weissig]

Routers are part of my day job as well. No USB device or custom software will ever touch them! They are all level 3 switches and enterprise level firewalls and WiFi access points. Different strokes for different folks :D

cheers,
h.

> --
> You received this message because you are subscribed to the Google Groups "rtl_433" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.
> To post to this group, send an email to rtl...@googlegroups.com.
> Visit this group at https://groups.google.com/group/rtl_433.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/fa9ac121-e762-cd17-5dfe-b9ce5a8a747f%40ludd.ltu.se.

[Helge Weissig]

I understand the issue with having to install and run additional software for a specific need being an undesirable burden. Hence my suggestion to create pluggable modules for the most popular use cases.

Out of curiosity, what sensors are you using and why couldn’t you just run a vanilla web server pointed to a file that gets updated by rtl_433 as it runs?

cheers,
h.

> On Apr 3, 2019, at 10:47 AM, Simon Eigeldinger <simon.ei...@vol.at> wrote:
>
> Hi,
>
> At the moment i would need a small web server that would display the latest x elements received on rtl_433.
> and i would like to have as less runtimes as possible.
> installing tons of runtimes nowadays. *lol*
>
> Am 03.04.2019 um 19:40 schrieb Helge Weissig:
>> Good point! However, check out my java tool… it should work just fine :) (he says, desperately trying to get back to the original topic)
>> cheers,
>> h.

>>> --
>>> You received this message because you are subscribed to the Google Groups "rtl_433" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.
>>> To post to this group, send an email to rtl...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/rtl_433.

>>> To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/f7f442fc-1343-c704-8adf-f57f6f725f65%40vol.at.

[Peter Smith]

"No USB device or custom software will ever touch them! "

Indeed.

But the real issue is how we should achieve a clean, understandable, maintainable and extensible architecture. We already, through much experience, know the answer to that question. It is realised by simplicity through the clear separation of duties into separate small modules with well organised communication between the modules. This is the hallmark of Linux.

On Wednesday, 3 April 2019 19:55:49 UTC+2, Helge Weissig wrote:

Routers are part of my day job as well. No USB device or custom software will ever touch them! They are all level 3 switches and enterprise level firewalls and WiFi access points. Different strokes for different folks :D

cheers,
h.

> On Apr 3, 2019, at 10:53 AM, Benjamin Larsson <ba...@ludd.ltu.se> wrote:
>
> On 2019-04-03 19:39, Helge Weissig wrote:
>> Just because it can be done does not mean it *should* be done! :)
>> Seriously though, I am all for making rtl_433 work on resource-limited platforms! Consumer-grade WiFi routers are actually good candidates, although with RPi0’s costing just a few bucks, why go through all the hassle of flashing your router and trying to deploy software to it?
>> cheers,
>> h.
>
> Because I can. And routers are free for me (related to my dayjob) thus a more suitable platform in my case.
>
> MvH
> Benjamin Larsson
>
> --
> You received this message because you are subscribed to the Google Groups "rtl_433" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rtl...@googlegroups.com.

[Benjamin Larsson]

On 2019-04-03 19:55, Helge Weissig wrote:
> Routers are part of my day job as well. No USB device or custom software will ever touch them! They are all level 3 switches and enterprise level firewalls and WiFi access points. Different strokes for different folks :D
>
> cheers,
> h.

To be clear, I am talking about home routers.

MvH
Benjamin Larsson

[Simon Eigeldinger]

Hi,

I am not talking mostly about me here.
i have more or less installed everything. *lol*
We just need to see how we gonna plug those modules.
While compilation or afterwards as downloads.

Mostly i have temperature sensors though i would have more in the future
i guess.
but maybe in a different topic. *smile*

[Helge Weissig]

Yes, I figured!

That said, I am not sure if you will still be able to fit the rtl_433 executable into the memory of your router once it includes support for communicating with all standard relational databases, MongoDB and other no-SQL database, TSDBs, REST and web socket features, 3G, LTE, 5G, and dial-up modem support, as well as HomeKit, Alexa, and Google Assistant functionalities ;)

cheers,
h.

> --
> You received this message because you are subscribed to the Google Groups "rtl_433" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.
> To post to this group, send an email to rtl...@googlegroups.com.
> Visit this group at https://groups.google.com/group/rtl_433.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/4ff9b275-be97-1006-81ee-f263b5fd9cfe%40ludd.ltu.se.

[Michael Campanelli]

Liked the idea of what you did so installed maven on raspberry pi3+ and cloned mqtt4rtl433.  failed to build but not sure why.

[INFO] Scanning for projects...

[INFO] 

[INFO] ----------------------< org.grajagan:mqtt4rtl433 >----------------------

[INFO] Building mqtt4rtl433 1.0-SNAPSHOT

[INFO] --------------------------------[ jar ]---------------------------------

[WARNING] The artifact org.apache.commons:commons-io:jar:1.3.2 has been relocated to commons-io:commons-io:jar:1.3.2

[INFO] 

[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ mqtt4rtl433 ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] Copying 1 resource

[INFO] 

[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ mqtt4rtl433 ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 6 source files to /home/pi/mqtt4rtl433/target/classes

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  14.700 s

[INFO] Finished at: 2020-01-18T19:20:49Z

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project mqtt4rtl433: Fatal error compiling: java.lang.ExceptionInInitializerError: com.sun.tools.javac.code.TypeTags -> [Help 1]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Tried the -e switch and got a bunch of java exceptions.  Again, not sure what is missing.

Any help is greatly appreciated.

-Mic

[Christian Zuckschwerdt]

Did you try the MQTT publisher recently included in rtl_433?

There is some info here: https://triq.org/rtl_433/OPERATION.html#mqtt-output

If you need to supplement the MQTT data there is an HASS auto-discovery example at: https://github.com/merbanan/rtl_433/blob/master/examples/rtl_433_mqtt_hass.py

[Helge Weissig]

Which JDK are you using (output from ‘java -version’)?

I think the error comes from one of the dependencies. I have upgraded that to the latest version. Could you please try again?

cheers,

h.

--
You received this message because you are subscribed to the Google Groups "rtl_433" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/e716ca4b-d907-44fd-83e4-fa7983c21d36%40googlegroups.com.