Differentiating output from multiple devices

[t...@quarendon.net]

If you run rtl_433 without specifying a protocol, it will decode anything it can find. 

I'm using the "-F json" to output the data in a JSON format to be read by another program.

The question is how do I determine the "device" for each record?

I see that the ones I'm looking at have a "model" field in the output. 

Is the intention that the "model" uniquely determine the format of output?

Thanks.

[Benjamin Larsson]

You have to make things unique. Otherwise you can not address everything
correctly. For example some hardware support setting a channel. Then you
can combine those to identify a single unit. Model is probably what you
are looking for.


MvH
Benjamin Larsson

[t...@quarendon.net]

You have to make things unique. Otherwise you can not address everything
correctly. For example some hardware support setting a channel. Then you
can combine those to identify a single unit. Model is probably what you
are looking for.

Sorry I'm not sure I understand. 

This was a question about the design of the output from rtl_433. 

I can specify one protocol on the command line. I therefore know that all output is from that protocol.

But if I want to pick up multiple signals and pipe them into something else I need to know how to distinguish the different output types. 

So I was expecting  some kind of unique "type id" for each output type that would allow me to filter or route the output as appropriate. 

For example I might want to publish all output from one device on one MQTT topic and output from another device on another MQTT topic. 

It's not clear how you do that in general. It *appeared* that each output has a different "model", but I wanted to confirm that that was the intention.

Consider a general purpose tools that might pipe the output from rtl_433 and publish each different "type" of output to a different MQTT topic such that I can then subscribe to a particular topic and get, say, only one "type" of record. It was unclear whether it was possible to write such a general purpose tool or whether the tool necessarily required knowledge of all of the particular types of output.

[Robert Terzi]

The current state of rtl_433 is the result of a bit of evolution that has generally been successful in letting people get to the raw data from the RF devices they own.   With one (or maybe more) exceptions, most users are using their own scripts for doing what they want with the data.  (The one exception I'm aware of is WeeWX-sdr by @mwall.)

There is a move toward structured data output for all devices.  Devices that don't output structured data are disabled by default.  There hasn't been a lot of work yet on a data model across all devices and device classes.  I believe @merbanan has a rough plan for what he'd like to see happen in order to reach a stable 1.0 release.

Note the primary goal of rtl_433, as I understand it, is to handle the low level RF details providing access to the data, enabling people to build what ever they need on top of it.   In terms of the output desires have been expressed to have both somewhat normalized data across devices within the same class, as well as to provide access to raw enough data that nothing is obscured from a consumer that might need it.  Therefore any sort of state keeping, unit conversions, duplicate suppression, is up to the data consumer.

In the rtl_433_tests repo there are sample signals as well as JSON output for many of the devices that are currently decoded.  What I would do is clone that repo and then find/cat all the json files together to get a test base of rtl_433 output for what ever you are thinking of implementing.

I believe some people are already using MQTT/mosquitto for dealing with the output data.  @vestom provided a sample python client.  I'm sure additional MQTT tooling will be welcome by many.

Yes, the trend is that "model" contains the specific "type" identifier.  ID is used for a device specific ID, IFF one is provided in the protocol. Some devices have fixed IDs, other change on every power-up.  Some types have a channel that can be physically set on the device that is used in conjunction with a physical console/display.  Other than the tests repo, there isn't a good base of information on the devices/protocols that are decoded.  Some of the source code contains a fair bit of comments about the protocols, in other it's completely lacking.

Hope this helps and please keep us all posted on whatever you build,
--Rob

--
You received this message because you are subscribed to the Google Groups "rtl_433" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.
To post to this group, send email to rtl...@googlegroups.com.
Visit this group at https://groups.google.com/group/rtl_433.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/a9c80581-a9b1-4d08-8a2a-590e929b1523%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Larsson]

Use multiple -R on the command line to only decode those type of
signals. The idea is to have structured output. We might not really be
there 100% but not far. If you want to filter the output then just do
that in a script that you pipe the records into. Just by looking at the
output from the transmitters that you are using it should be pretty
simple to create rules on how to filter the records in the way you want.

If this still not answer your question then just supply some sample
records that we can look at.

MvH
Benjamin Larsson

[Andrew Quayle]

I have a similar issue
I have 2 temperature sensors - sending data to mqqt.
they both have temperature_C as an identifier, so I can't differentiate between them.

[Robert Terzi]

Ok, not much to go on here:   So what kind of sensors do you have?   What are you using to pipe the output into MQTT?

Have you looked at the output from rtl_433 directly?

Since many/most devices have a model string and either an ID and/or channel number to identify the sensor, whatever is putting the data into MQTT might be  incorrectly configured or need to be changed to key on different fields.

--Rob

--
You received this message because you are subscribed to the Google Groups "rtl_433" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.
To post to this group, send email to rtl...@googlegroups.com.
Visit this group at https://groups.google.com/group/rtl_433.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/ced099dd-f7c5-4bee-8bc8-df980470ba7e%40googlegroups.com.

[Andrew Quayle]

sensors: THGR800 - part of the Oregon WMR86 Weather station and Acurite 6002
the acurite (from rtl_433 -G reports as a LaCrosse

rtl_433 -R 12 -R8  -F json -U | mosquitto_pub -p port -u username -P password -t home/rtl_433 -l

reply:-
Client mosqsub/4430-dadraspi3 received PUBLISH (d0, q0, r0, m0, 'home/rtl_433', ... (157 bytes))
{"time" : "2017-06-19 21:06:14", "brand" : "OS", "model" : "THGR810", "id" : 247, "channel" : 1, "battery" : "OK", "temperature_C" : 24.000, "humidity" : 55}
Client mosqsub/4430-dadraspi3 received PUBLISH (d0, q0, r0, m0, 'home/rtl_433', ... (101 bytes))
{"time" : "2017-06-19 21:06:19", "model" : "LaCrosse TX Sensor", "id" : 18, "temperature_C" : 25.600}

both temperature sensore report back the topic as temperature_C

[Andrew Quayle]

The Home Assistant part of this is:

sensor:
 
  - platform: mqtt   
  state_topic: 'home/rtl_433'
  name: 'Oudoor Temperature'   
  unit_of_measurement: '°C'   
  value_template: '{{ value_json.temperature_C }}'

[Christian Zuckschwerdt]

Maybe try something like https://github.com/zuckschwerdt/rtl_433_relay/blob/master/rtl_433_mqtt_relay.py
Note that this is a modular JSON-UDP to MQTT bridge, not a pipe. So you'd need https://github.com/zuckschwerdt/rtl_433/tree/develop
Otherwise just adapt the script to process piped data.

[Christian Zuckschwerdt]

btw. here is an example of a few sensors on MQTT: https://zuckschwerdt.org/mqtt-example.png
(the CNY70 tracks the water meter and the BSB is my boiler system bus.)

[Andrew Quayle]

Thanks for your help..

I think that my problem is with Home Assistant not with your code.
and I don't want to stray into a different topic.

[Robert Terzi]

This is the first time, I've come across Home Assistant, it looks interesting and possibly somewhat comprehensive in what it already supports. Are you just getting started with it?

Anyone else here using it?

Tip: most people here will want to look at the Developers page, https://home-assistant.io/developers/ to get an overview.

[Andrew Quayle]

been using it for a week.
The support is a bit lacking - there is a forum, but lots of questions go un-answered.

[Andrew Quayle]

Been trying to read the topic from a python script - got an error

TypeError: Can't convert 'bytes' object to str implicitly

which suggests an errant byte is returned somewhere.

script attached

[Andrew Quayle]

All sorted - thanks for the help
data.decode('ascii')

[Johannes Schoeller]

hi christian

i found your scripts while looking for a way to differentiate two temperature sensor from each other, and because i was not able to tell home assistant to ignore sensor request that don't include valid data for the specific sensor, leading to empty data.

so i want to make sensor specific topics, and i guess your script does exactly this.

e.g. add the manufaturer, or any other stuff (if you edit the script) to the "main" sensors/rtl_433 topic.

but how would i use the script? what's the command line to start rtl_433 with, and then do what? pip it into the .py script?

your rtl_433 mod seems to add a udp 1024 port. but how?

may you please add your command line here? thanks!!

[Christian Zuckschwerdt]

The script consumes the (UDP) Syslog output from rtl_433 (or a legacy plain JSON datagram). Basically run rtl_433 with -F syslog:127.0.0.1:1433 (or -F syslog and change the port in the script to 514). That should be all. No pipes needed -- nicely decoupled.

[Johannes Schoeller]

thanks christian!!

i thought it would work, got one "new topic published", but then the script stopped

MQTT connected: Connection Accepted.
Traceback (most recent call last):
  File "/usr/local/bin/rtl_433_mqtt_relay.py", line 120, in <module>
    run()
  File "/usr/local/bin/rtl_433_mqtt_relay.py", line 116, in run
    rtl_433_probe()
  File "/usr/local/bin/rtl_433_mqtt_relay.py", line 102, in rtl_433_probe
    data = parse_syslog(line)
  File "/usr/local/bin/rtl_433_mqtt_relay.py", line 83, in parse_syslog
    if line.startswith("<"):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

i guess i will need to look into the code and see what happens here.

[Christian Zuckschwerdt]

You are using Python 3 and this script was made for Python 2. Just Insert line.decode("ascii") before working with the string data :) I'll update the repo too. Thanks.

[Johannes Schoeller]

thanks christian for your super fast reply.

unfortunately i'm no coder, so i can't interpret your answer. WHERE exactly do i insert "line.decode("ascii")"?

thank you

btw: when you update the repo you may add some info to the readme as well. it may save you time to reply here to every individual ;)

[Christian Zuckschwerdt]

In the line before the error :) https://github.com/zuckschwerdt/rtl_433_relay/blob/master/rtl_433_mqtt_relay.py#L82
The repo has the fix now.

[Johannes Schoeller]

it's all working fine so far! i receive new individual topics for each sensor.

but home assistant seems to be unable to analyze them now.

this is what they looked before:

{"time" : "2018-03-21 08:07:55", "model" : "Smoke detector GS 558", "id" : 18316, "unit" : 21, "learn" : 0, "code" : "78f195"}
{"time" : "2018-03-21 08:08:26", "brand" : "OS", "model" : "CM160", "id" : 11, "power_W" : 466}

and now they are a bit bloated:

<165>1 2018-03-21T08:06:17Z hassbian rtl_433 - - - {"time":"2018-03-21 08:06:17","model":"Smoke detector GS 558","id":18316,"unit":21,"learn":0,"code":"78f195"}

<165>1 2018-03-21T08:02:26Z hassbian rtl_433 - - - {"time":"2018-03-21 08:02:26","brand":"OS","model":"CM160","id":11,"power_W":644}

i'm sure there is a way to tell HA how to parse it, but i tried to alter the .py script's code to get rid of the extra information: "<165>1 2018-03-21T08:06:17Z hassbian rtl_433 - - - ".

guess what: i was NOT successfull ;)

this is HA's error message:

Error parsing value: 'value_json' is undefined (value: <165>1 2018-03-21T07:56:26Z hassbian rtl_433 - - - {"time":"2018-03-21 07:56:26","brand":"OS","model":"CM160","id":11,"power_W":740}, template: {{ value_json.power_W }}

[Christian Zuckschwerdt]

Thanks for testing. Thats not intented. The script now outputs plain JSON to MQTT.

[Khalid Baheyeldin]

Christian,

Is this syslog feature (which effectively eliminates pipes) only in the 'develop' branch, or is it in 'master' as well?

[Khalid Baheyeldin]

And my question was for rtl_433 itself, not the relay Python script. In other words, does rtl_433 master have the syslog socket feature, or only the 'develop' branch for rtl_433 has that?

[Christian Zuckschwerdt]

You are right, it originated in my develop-branch. It was merged into master on 2018-01-26: https://github.com/merbanan/rtl_433/commit/c30a2a6 and IPv6 has been added: https://github.com/merbanan/rtl_433/commit/74b809 but it's disabled on windows: https://github.com/merbanan/rtl_433/commit/ad7799a (someone needs to adapt the network code for that...)

[Khalid Baheyeldin]

Great.

Cloned the merbanan master and compiled it, and it works like a charm!

Thanks to all of the contributors ...

Much better than pipes, buffering ...etc.

[Johannes Schoeller]

i thank you! all working perfectly fine now!

[Christian Zuckschwerdt]

Glad you both found it useful. I'm not a fan of shell pipes in a server/service environment. I think that decoupling and controlling each process on it's own has advantages :)