New signal decode architecture

[Benjamin Larsson]

Right now every time a protocol is added the cpu usage increases. This
is because each protocol processes each sample input in it's own
instance of the bit decoder.

signal -> ook_signal_demod -> signal_type_demod -> protocol_decode

This needs to be changed to another model to be able to handle more
protocols without increasing the cpu usage. The model I am thinking of
is the one used in analyze mode.

signal -> ook_signal_demod -> signal_re-represenation -> bit_decode ->
protocol_decode

This signal_re-represenation is a table that lists the pulses start time
and their lengths. This table should then be decoded to bits (somehow).
And the resulting bits can then later be sent to all protocols that
matches the signal properties.

Hopefully I will get the motivation and time to implement this sometime
soon.

MvH
Benjamin Larsson

[Robert Terzi]

On 1/23/2015 8:05 PM, Benjamin Larsson wrote:

> signal -> ook_signal_demod -> signal_re-represenation -> bit_decode -> protocol_decode
>
> This signal_re-represenation is a table that lists the pulses start
> time and their lengths. This table should then be decoded to bits
> (somehow). And the resulting bits can then later be sent to all
> protocols that matches the signal properties.

So I was thinking about that just before the holidays.

What I had in mind for the signal re-representation is the list of
on-off times, from the edge detection that analyze is already doing.
The decoders can be easily rewritten to use that instead of checking
every sample.

I got stuck on how to represent the signals and figure out which
protocolcall backs should get called.

Lately I've been wondering if the current model of calling all the
callbacks is actually the right idea, let the protocol decode
decide if it should process the signal or not.

signal -> ook_signal_demod -> signal on/off times -> protocol call back
appropriate signal? -> call appropriate bit_decode -> process.

There could be a couple of provided ways of deciding if the signal
is the one you want for the protocol bits to call based on signal
length, initial pulse length, etc.

One result is the bit_decoders would be simpler, the protocol routines
could choose to lop off any start/sync bits before calling the bit decoder.

There would be a lot of flexibility, however tThe downside is there would
be more boiler plate in the protocol code.

I never got around to coding anything, yet.

(I should mention, I wrote an external decoder that just takes the -a -D
debug output (which has the on-off times, in length/distance). It was
necessary because the DSC security contacts send no signal during a zero
period. The can be silence for up to a whole byte worth of bits. They
inject a sync bit between each byte.)

Thoughts?
--Rob

[Batilan]

Op zaterdag 24 januari 2015 03:47:43 UTC+1 schreef Robert Terzi:

On 1/23/2015 8:05 PM, Benjamin Larsson wrote:

> signal -> ook_signal_demod -> signal_re-represenation -> bit_decode -> protocol_decode
>
> This signal_re-represenation is a table that lists the pulses start
> time and their lengths. This table should then be decoded to bits
> (somehow). And the resulting bits can then later be sent to all
> protocols that matches the signal properties.

signal -> ook_signal_demod -> signal on/off times -> protocol call back
     appropriate signal? -> call appropriate bit_decode -> process.

There could be a couple of provided ways of deciding if the signal
is the one you want for the protocol bits to call based on signal
length, initial pulse length, etc.

One result is the bit_decoders would be simpler, the protocol routines
could choose to lop off any start/sync bits before calling the bit decoder.

There would be a lot of flexibility, however tThe downside is there would
be more boiler plate in the protocol code.

I never got around to coding anything, yet.

(I should mention, I wrote an external decoder that just takes the -a -D
debug output (which has the on-off times, in length/distance).  It was
necessary because the DSC security contacts send no signal during a zero
period.  The can be silence for up to a whole byte worth of bits.  They
inject a sync bit between each byte.)

Thoughts?
--Rob

I agree we should only once do the edge detection, So I guess we all agree on the

signal -> ook_signal_demod -> signal on/off times part?

Maybe we can define some subscription rules that allow a device callback to specify which type of signal it is interested in.

Most (but probably not all) signals can be qualified  by pulse and/or pause widths (if we can reliably exclude pre-amble and post-amble effects),

if we can reliably exclude pre and post-amble effects for getting the 'standard' PWM timings, then we can also reiliably include them (as special pre and post-amble characteristics)

I'm not sure if this is feasible as I have only seen a handfull of signal characteristics (for which I think it should be feasible).

I guess the required code won't be DSP-like but needs to be more heuristics based?

So then it would be like:

signal -> ook_signal_demod -> signal on/off times -> pre/post-amble processing + frame detection -> packet characteristics (normal PWM timings, pre/post-amble timings, #frames, length of frames) -> call appropriate device callbacks with bitt buffer

A device decoder would then specify the characteristics of packets its interested in and only be called when the characteristics of the packet match, it can also specify how it wants the packets decoded

(PWM short = 1, PWM short =0, Manchester,...) 

This would allow for simple device decoders that only need to be concerned with checking (preferably using CRC/checksum when available) and decoding the data bits.

Optionally we can still have device callbacks (of another type) that we supply the raw on/off timings for devices that have more rare device characteristics. 

On another subject: How de we feel about using google groups for discussion? I find the (user) interface rather arcane and it doesn't help a whole lot in keeping an overview on discusions.

This might also be partly to me not being very familiar with it.

Helge suggested using Discourse which at least has better looks, and looks better on the eyes. 

I don't have an issue with continuing over here. However as we have not yet built op a large history now is the time to switch to something better if we feel like it.

[Robert Terzi]

Another thought that I've had w.r.t. a new architecture for rtl_433
has been brought back to mind by the output discussion. rtl_433,
will never be able to address all of the requests that people have
for ways of handling the data.

In some cases, the desired would be to have rtl_433 embedded in some
home automation platform that is processing events from other sources
as well.

Or rtl_433 could be a lightweight server that could send messages to
what ever platform was desired. (It might be simpler as a standalone
server than as a library when trying to deal with the threading and/or
main loop issues of what ever you wanted to integrate it with.)

For my personal interests, I've thought about splitting the pieces
into a small rtl_433 C based core that just handles the rtl and
on-off time/edge detection, and a Python piece that could do what ever
you want. There is a lot of precedent for Python in the rtl-sdr world.

Finally, I could also see that there would people that a small self
contained C based executable with minimal parsing close to the current
structure is a good match for low power embedded devices RPi, OpenWRT, etc.

I know that's a whole bunch of conflicting thoughts, but maybe this
will help others in contributing.

The one core / common thing is what is a good, general representation
of the signal where the "heavy lifting" of dealing with RTL and the
digital signal processing bits can be handled by the main rtl_433
that makes sense to pass off to the hopefully simple protocol decoders?

Thoughts?

[MGL]

Perhaps this is heretical, but is it thinkable that the solution would also work with a GPIO-connected 433.92 MHz receiver as a source? 

[Benjamin Larsson]

On 01/25/2015 07:56 PM, Robert Terzi wrote:
> Another thought that I've had w.r.t. a new architecture for rtl_433
> has been brought back to mind by the output discussion. rtl_433,
> will never be able to address all of the requests that people have
> for ways of handling the data.

I am naive but I think the aim should be to support all the requests
people have. Right now the analyse mode can handle lots of signals
automatically. With some more development and I think we could handle
all simple signals.

>
> In some cases, the desired would be to have rtl_433 embedded in some
> home automation platform that is processing events from other sources
> as well.
>
> Or rtl_433 could be a lightweight server that could send messages to
> what ever platform was desired. (It might be simpler as a standalone
> server than as a library when trying to deal with the threading and/or
> main loop issues of what ever you wanted to integrate it with.)
>
> For my personal interests, I've thought about splitting the pieces
> into a small rtl_433 C based core that just handles the rtl and
> on-off time/edge detection, and a Python piece that could do what ever
> you want. There is a lot of precedent for Python in the rtl-sdr world.
>
> Finally, I could also see that there would people that a small self
> contained C based executable with minimal parsing close to the current
> structure is a good match for low power embedded devices RPi, OpenWRT, etc.
>
> I know that's a whole bunch of conflicting thoughts, but maybe this
> will help others in contributing.
>
> The one core / common thing is what is a good, general representation
> of the signal where the "heavy lifting" of dealing with RTL and the
> digital signal processing bits can be handled by the main rtl_433
> that makes sense to pass off to the hopefully simple protocol decoders?
>
> Thoughts?

I think that the main focus should be a small executable that is able to
handle several different signals. But I don't see any problem with
outputting just the pulses with some command flag. For example pulseview
(http://sigrok.org/wiki/Protocol_decoder_output) mentions that they
would like to be able to feed data into rtl_433 and have it decoded. So
basically whatever people want or need should be able to get added into
rtl_433 as long as it is part of the core functionality.

MvH
Benjamin Larsson

[Benjamin Larsson]

>
> I agree we should only once do the edge detection, So I guess we all
> agree on the
> signal -> ook_signal_demod -> signal on/off times part?

It is the best way I could think of at least.

>
> Maybe we can define some subscription rules that allow a device callback
> to specify which type of signal it is interested in.
>
> Most (but probably not all) signals can be qualified by pulse and/or
> pause widths (if we can reliably exclude pre-amble and post-amble effects),
> if we can reliably exclude pre and post-amble effects for getting the
> 'standard' PWM timings, then we can also reiliably include them (as
> special pre and post-amble characteristics)
>
> I'm not sure if this is feasible as I have only seen a handfull of
> signal characteristics (for which I think it should be feasible).
> I guess the required code won't be DSP-like but needs to be more
> heuristics based?

We need to be able to classify signals for this to work. Right now the
analyse mode can sort of figure out if the signal uses pulses or the
distance of the pulses as carrier of bits. This is a bit error prone so
I'd like to try and move the problem away from heuristics and to
something more deterministic.

What I propose is that we just pass the pulse table through all n-types
of signal bit-decoders we have and then pass the bit buffer that a
specific protocol decoder require, for example there is no need for a
protocol that uses manchester coding to get a buffer that has been
passed though the distance bit decoder. And we should also extract even
more data from the signal so we can figure out what signal it is. For
example total signal length and packet length and packet repetitions.
Then it is real easy to send the correct bit buffer to the right
protocol decoder.

>
> So then it would be like:
> signal -> ook_signal_demod -> signal on/off times -> pre/post-amble
> processing + frame detection -> packet characteristics (normal PWM
> timings, pre/post-amble timings, #frames, length of frames) -> call
> appropriate device callbacks with bitt buffer
>
> A device decoder would then specify the characteristics of packets its
> interested in and only be called when the characteristics of the packet
> match, it can also specify how it wants the packets decoded
> (PWM short = 1, PWM short =0, Manchester,...)
>
> This would allow for simple device decoders that only need to be
> concerned with checking (preferably using CRC/checksum when available)
> and decoding the data bits.
>
> Optionally we can still have device callbacks (of another type) that we
> supply the raw on/off timings for devices that have more rare device
> characteristics.

I agree with this, together with what I wrote above it should be really
easy to add new protocols without disturbing the detection of the old ones.

>
> On another subject: How de we feel about using google groups for
> discussion? I find the (user) interface rather arcane and it doesn't
> help a whole lot in keeping an overview on discusions.

> This might also be partly to me notbeing very familiar with it.

> Helge suggested using Discourse which at least has better looks, and
> looks better on the eyes.
> I don't have an issue with continuing over here. However as we have not
> yet built op a large history now is the time to switch to something
> better if we feel like it.

I choose google groups because you can enable email delivery and use it
as a mailing list. That way I can actively partake in discussions with
just my email client and people who join after a thread is started can
search the old archive and revive an old thread if needed. If we want
something else I would need an email interface otherwise I think I just
would forget to watch what happens just like I do with the github issue
tracker :/.

MvH
Benjamin Larsson

[Benjamin Larsson]

On 01/25/2015 08:14 PM, MGL wrote:
> Perhaps this is heretical, but is it thinkable that the solution would
> also work with a GPIO-connected 433.92 MHz receiver as a source?
>

As long as you can generate the pulse list this is totally possible. But
I think it will take some time before it is possible to implement it.

MvH
Benjamin Larsson

[Tommy Vestermark]

Hi,

As you might have seen, I have been giving it a shot at making a packet based decoding in this pull request: https://github.com/merbanan/rtl_433/pull/124

As we are now up to 20 devices the CPU overhead of making a sample based processing per device becomes significant. When running at a Raspberry Pi, I had to comment out the devices I didn't use to reduce the CPU usage. I have a new sensor, I wanted to add, but didn't want to add yet another CPU hog. Instead i have tried to make a new infrastructure with only one function running at full sample rate and keeping state between rtl buffers:
envelope sample data -> detect_pulse_package() -> pulse/gap time list -> pulse_demod_xxx() -> bit buffer -> driver callback()

It was a design goal to make the new infrastructure capable of co-existing with the existing drivers and demodulators to avoid major breakage. So right now there is no CPU reduction, as I have only converted 1 driver (fineoffset.c). But if the feedback is positive, I will go ahead and try to convert some of the other drivers which have available test data.

A potential issue is that the new infrastructure has to decide when to declare End Of Package and send the pulse list to the demodulators. As a result the new infrastructure will tend to send multi packet data separately. Some device drivers currently utilize that data is sent twice for error detection (and even to compensate for truncated first bit due to forced start bit removal by demodulator!). Personally I think this strategy is broken anyway, and drivers should not rely on this redundancy, which is made for increasing probability of reception in a busy channel in the first place. If really necessary, the device drivers may have to implement their own inter-packet state.

Please review and provide some feedback (positive or negative) on how to proceed.

Kind regards,
Tommy

[Robert Terzi]

On 6/29/2015 5:49 PM, Tommy Vestermark wrote:

> As you might have seen, I have been giving it a shot at making a
> packet based decoding in this pull request:
> https://github.com/merbanan/rtl_433/pull/124

I got your code running. Nice work. Your analyzer looks very cool
and shows a lot of promise. I need to put some signals it had problems
with in the test signal repo.

The LaCrosse TX 433 Mhz temperature/humidity sensors were detected
perfectly. I may give a look at converting the lacrosse tomorrow
or the next day.

It's a little confusing that you need debug (-D) but NOT analyze -a
to see the output of your analyzer.

Note: -std=gnu11 was too new for the compiler on my test system,
but changing it to gnu99 was fine. I think the only thing I saw
that really needed the newer standard definition was the declaration
in the initializer of the for loop.

I think we should try to get away from the constants for OOK_PWM, etc.
and just put a function pointer in the device struct to allow device
specific demodulators when necessary.

> A potential issue is that the new infrastructure has to decide when>
> to declare End Of Package and send the pulse list to the demodulators.

This is a bit tricky and there are some cases where I think I'd want
per protocol specification of what the end criteria is. Over the
weekend I was looking at a pair of signals from two related devices
that run into each other with out enough quiet time to decide the
first signal is done and the second signal has started.

>As a result the new infrastructure will tend to send multi packet
>data separately.

While I've come to appreciate the way the current code can group
retransmissions, I can see why this is necessary in a generic
demodulator.

Thanks. Looking forward to playing around with this more.

--Rob

[Tommy Vestermark]

Den tirsdag den 30. juni 2015 kl. 05.28.19 UTC+2 skrev Robert Terzi:

I got your code running.  Nice work. Your analyzer looks very cool
and shows a lot of promise.  I need to put some signals it had problems
with in the test signal repo.

Thanks :-) I saw some problems where the histogram function would make bins with overlapping pulse clusters - probably due to the first pulses being outliers. I will try to experiment with some "bin fusing" when bins are obviously overlapping. That should make detection more reliable. I don't know if that describes your problem?

 

The LaCrosse TX 433 Mhz temperature/humidity sensors were detected
perfectly.  I may give a look at converting the lacrosse tomorrow
or the next day.

Perfect... but be aware that the code is not merged yet and may change :-)
 

It's a little confusing that you need debug (-D) but NOT analyze -a
to see the output of your analyzer.

Yes, I know :-) I just did not want to mess with analyzer mode for now before discussing it. Is it good enough to replace the current analyzer? Do we miss some more information?

 

Note: -std=gnu11 was too new for the compiler on my test system,
but changing it to gnu99 was fine.  I think the only thing I saw
that really needed the newer standard definition was the declaration
in the initializer of the for loop.

You are right. I have changed the code to gnu99. We can probably go to gnu11 in 10 years time ;-) The device drivers are screaming for C99 structure initializers though...
 

I think we should try to get away from the constants for OOK_PWM, etc.
and just put a function pointer in the device struct to allow device
specific demodulators when necessary.

I also thought about that, but have aimed for not touching the drivers at this point. That is somewhat orthogonal to the pulse detection principle and can be a fine next step. I do not want to make it too complex at this stage.
 

> A potential issue is that the new infrastructure has to decide when>
> to declare End Of Package and send the pulse list to the demodulators.

This is a bit tricky and there are some cases where I think I'd want
per protocol specification of what the end criteria is.  Over the
weekend I was looking at a pair of signals from two related devices
that run into each other with out enough quiet time to decide the
first signal is done and the second signal has started.

>As a result the new infrastructure will tend to send multi packet
>data separately.

While I've come to appreciate the way the current code can group
retransmissions, I can see why this is necessary in a generic
demodulator.

There will always be a lower bound before the signals are effectively overlapping or disturbing each other (I believe that holds true for the current decoders as well). My strategy is to be a bit aggressive in declaring End Of Package. Currently I am using a heuristic of a gap of 10 times maximum pulse length. It seems to work well, but I will work a bit more at refining it and handling corner cases (e.g. looong FSK pulses). I am open for good suggestions!
(Hmmm. I can now see the prologue sensor actually has a pulse/gap ratio of up to ~10 in a PPM modulation scheme...)

Although it may be practical, I always disliked all the policy in the current decoders (startbit removal, multi packet handling). That really should be handled in the device drivers. Any good ideas for how to generalize it?

Thanks.  Looking forward to playing around with this more.

Looking forward to hear your results :-) Keep us updated.

/Tommy

[Tommy Vestermark]

Den tirsdag den 30. juni 2015 kl. 19.00.33 UTC+2 skrev Tommy Vestermark:

Den tirsdag den 30. juni 2015 kl. 05.28.19 UTC+2 skrev Robert Terzi:

I got your code running.  Nice work. Your analyzer looks very cool
and shows a lot of promise.  I need to put some signals it had problems
with in the test signal repo.

Thanks :-) I saw some problems where the histogram function would make bins with overlapping pulse clusters - probably due to the first pulses being outliers. I will try to experiment with some "bin fusing" when bins are obviously overlapping. That should make detection more reliable. I don't know if that describes your problem?

I have added a first try of bin fusing at this experimental branch: https://github.com/vestom/rtl_433/tree/demod_experimental
It seems to work fine with the ambient weather sensor data, which previously had some overlapping histogram bins (rtl_433_tests/tests/ambient_weather/01/f007th.data). The histogram manipulation is a bit hairy though and not very thoroughly tested. But please give it a try :-)

/Tommy

[Benjamin Larsson]

I'm quite busy right know but I will merge this after I have time to
look at the code.

MvH
Benjamin Larsson

[Tommy Vestermark]

Den torsdag den 2. juli 2015 kl. 00.17.09 UTC+2 skrev Benjamin Larsson:

I'm quite busy right know but I will merge this after I have time to
look at the code.

Looking forward to get your feedback :-)

I have converted the Manchester demodulator in the experimental branch (https://github.com/vestom/rtl_433/tree/demod_experimental). Both the Ambient Weather sensor and the Oregon Scientific is successfully tested with the test data shown in the commit comments. I have no live sensors though.

Regarding the End Of Package detection and close messages from different sensors, it occurred to me, that it should really not be a big issue for the demodulators, as they will walk through all pulses anyway and have their own "reset_limit" criteria for declaring their own End Of Message - as long as we are not too aggressive in declaring EOP. It will fool the analyzer though, if multiple messages are in the same package.

/Tommy

[Tommy Vestermark]

... and now made a start_bit removal mode for the PWM decoder and converted the Lacrosse sensor. Tested with data shown in the commit comment. (https://github.com/vestom/rtl_433/tree/demod_experimental). Had to change the reset_limit - but that didn't matter, as driver has a time based removal of redundant data anyway.

That will be enough for today...

/Tommy

[Robert Terzi]

On 7/2/2015 6:54 PM, Tommy Vestermark wrote:
> ... and now made a start_bit removal mode for the PWM decoder and
> converted the Lacrosse sensor. Tested with data shown in the commit
> comment. (https://github.com/vestom/rtl_433/tree/demod_experimental

> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fvestom%2Frtl_433%2Ftree%2Fdemod_experimental&sa=D&sntz=1&usg=AFQjCNHhjBGIS1zTubIFsfYooXWRLv5x5Q>).

> Had to change the reset_limit - but that didn't matter, as driver has
> a time based removal of redundant data anyway.

Cool. I'll give this a look in the morning.

Note: LaCrosse TX (433 Mhz) actually doesn't have a start bit
that needs removing. However, it was implemented using the
existing pulse width decoder that removes the start bit. Since
the first bit is always one I have a workaround that adds it
back in.

I think there is at least one other sensor that does something
similar, since it was also implemented when there was only
the two decoders.

--Rob

[Robert Terzi]

On 7/2/2015 6:54 PM, Tommy Vestermark wrote:

> ... and now made a start_bit removal mode for the PWM decoder and
> converted the Lacrosse sensor.

I did some testing of Tommy's updated demodulator with 4 LaCrosse TX
temperature/humidity sensors that I have running around the house.

The results are very good. I hope this gets merged soon.

First, I confirm it works well with live signals.

Next I did a side-by-side comparison using a recording file (-m 1 type,
recording of the whole rtl_433 run). The old demodulator correctly
decoded 13 messages, the new demodulator decoded 15. It was able
to correctly identify two messages the old demodulator didn't.

The new histogram analysis is extremely useful. If you having seen
it yet, here's what it looks like:



Analyzing pulses...
Total number of pulses: 44
Pulse width distribution:
[ 0] mean: 332 (328/335), count: 26
[ 1] mean: 131 (128/132), count: 18
Gap width distribution:
[ 0] mean: 261 (260/267), count: 43
Pulse period distribution:
[ 0] mean: 594 (591/596), count: 26
[ 1] mean: 393 (389/399), count: 17
Guessing modulation: Pulse Width Modulation with fixed gap


Analyzing pulses...
Total number of pulses: 44
Pulse width distribution:
[ 0] mean: 331 (330/335), count: 26
[ 1] mean: 131 (130/134), count: 18
Gap width distribution:
[ 0] mean: 261 (259/264), count: 43
Pulse period distribution:
[ 0] mean: 593 (590/598), count: 26
[ 1] mean: 393 (390/395), count: 17
Guessing modulation: Pulse Width Modulation with fixed gap

For comparison, here are the same two signals with the existing-a
analyze mode:


Iteration 1. t: 232 min: 132 (36) max: 333 (52) delta 18
Iteration 2. t: 232 min: 132 (36) max: 333 (52) delta 0
Pulse coding: Short pulse length 132 - Long pulse length 333

Short distance: 258, long distance: 260, packet distance: 7154

p_limit: 232

[00] {44} f5 ff 28 bd 8b 00 : 11110101 11111111 00101000 10111101 10001011 00000000
[01] {44} f5 ff 28 bd 8b 00 : 11110101 11111111 00101000 10111101 10001011 00000000


Note that the existing -a, grouped the two signals together, where
the new analysis treats them as separate signals.

Also note the change, -D (debug) mode, in Tommy's demod_experimental
branch, shows the new histogram analysis. It also no longer prints all
the pulse distance data. However, -a (analyze), gives the old debug
and old analyze output.

Tommy, that's for your hard work. I think this contribution will
move rtl_433 forward a big step.

--Rob

[Robert Terzi]

Tommy:

Minor feature request for the new signal analysis function,
a command line option to override the quiet time setting
for determining a message has finished.

As I understand it currently, the test for end of message
is to wait for a quiet period of 10 times the longest pulse.

(Right now I'm asking about command line for analysis
because I haven't spent enough time with how reset limits
work with the new pulse timing demodulator, yet.)

For many signals this works out nicely, but there are
times it would be good to change it.

I'm using X10 power line as an example. What's interesting
about this signal is the long wake-up pulse, also it is
the NEC consumer Infrared protocol sent over RF.

(I'm going into details because it might be helpful for
others.)

Your analyzer produces this output:

Total number of pulses: 204
Pulse width distribution:
[ 0] mean: 2244 (2242/2247), count: 6
[ 1] mean: 141 (135/145), count: 198
Gap width distribution:
[ 0] mean: 1118 (1118/1119), count: 6
[ 1] mean: 137 (135/141), count: 96
[ 2] mean: 415 (413/422), count: 96
[ 3] mean: 9981 (9979/9986), count: 5
Pulse period distribution:
[ 0] mean: 3363 (3360/3365), count: 6
[ 1] mean: 279 (275/284), count: 96
[ 2] mean: 557 (555/560), count: 96
[ 3] mean: 10124 (10122/10127), count: 5
Guessing modulation: No clue...

It's pretty easy to figure out what the signal looks like
from this, especially because I know what to look for.

There is are 6 long pulse of 2,244 samples and
there is are 5 quiet period of 10,124 samples, which if
you think about it, a good guess is that this is
6 messages, with 40 milliseconds between the 5 repeats.

If I could turn the quiet time multiplier down to 4,
I could easily test that theory without having to
use Audacity, etc.

Of course it could be helpful to turn it up too. I've
got a device that sends out 200 short pulses with a long
pause as a wake-up signal.

Thanks,
--Rob

[Robert Terzi]

Sorry, I realized I included the wrong number in the
example, the quiet period is 9,981 samples, not 10,124,
which is still just about 40 ms.

On 7/3/2015 3:28 PM, Robert Terzi wrote:

> Total number of pulses: 204
> Pulse width distribution:
> [ 0] mean: 2244 (2242/2247), count: 6
> [ 1] mean: 141 (135/145), count: 198
> Gap width distribution:
> [ 0] mean: 1118 (1118/1119), count: 6
> [ 1] mean: 137 (135/141), count: 96
> [ 2] mean: 415 (413/422), count: 96
> [ 3] mean: 9981 (9979/9986), count: 5
> Pulse period distribution:
> [ 0] mean: 3363 (3360/3365), count: 6
> [ 1] mean: 279 (275/284), count: 96
> [ 2] mean: 557 (555/560), count: 96
> [ 3] mean: 10124 (10122/10127), count: 5
> Guessing modulation: No clue...
>
> It's pretty easy to figure out what the signal looks like
> from this, especially because I know what to look for.
>
> There is are 6 long pulse of 2,244 samples and

> there is are 5 quiet period of 9,981 samples, which if

[Tommy Vestermark]

Den fredag den 3. juli 2015 kl. 21.02.24 UTC+2 skrev Robert Terzi:

Next I did a side-by-side comparison using a recording file (-m 1 type,
recording of the whole rtl_433 run).   The old demodulator correctly
decoded 13 messages, the new demodulator decoded 15.  It was able
to correctly identify two messages the old demodulator didn't.

Hmm. The only thing I can think of, which could cause this difference would be the more aggressive declaration of End Of Package. The old approach of trying to group messages in one package would likely cause messages to be lost e.g. when close to an un-related message or due to device driver only looking at first message in a package.
An all other aspects I believe the old and new approach to act the same.
 

Tommy, that's for your hard work.  I think this contribution will
move rtl_433 forward a big step.

Thanks :-) Nice to hear others are appreciating the work.
 
/Tommy

[Tommy Vestermark]

Den fredag den 3. juli 2015 kl. 21.28.52 UTC+2 skrev Robert Terzi:

Minor feature request for the new signal analysis function,
a command line option to override the quiet time setting
for determining a message has finished.

As I understand it currently, the test for end of message
is to wait for a quiet period of 10 times the longest pulse.

Yes, you are right. That method obviously has issues with long pulses, and will need some tweaking or extra criteria. In the end we would not want a command line option, but a more generally robust method that works with all(?!) sensors. I will try to look into it, but for development purposes you may just tweak the source itself (e.g. change PD_MAX_GAP_RATIO to 4). Maybe we can have an upper bound of gap limit - I think I will try to make a spreadsheet with the values for all sensors to try to evaluate some commonality.

 

Of course it could be helpful to turn it up too.  I've
got a device that sends out 200 short pulses with a long
pause as a wake-up signal.

Could you please add the signal to rtl_433_tests, and I will have a look?

/Tommy

[Robert Terzi]

On 7/8/2015 3:28 PM, Tommy Vestermark wrote:
> Den fredag den 3. juli 2015 kl. 21.02.24 UTC+2 skrev Robert Terzi:
>
> Next I did a side-by-side comparison using a recording file (-m 1 type,
> recording of the whole rtl_433 run). The old demodulator correctly
> decoded 13 messages, the new demodulator decoded 15. It was able
> to correctly identify two messages the old demodulator didn't.
>
> Hmm. The only thing I can think of, which could cause this difference would be the more aggressive declaration of End Of Package. The old approach of trying to group messages in one package would likely cause messages to be lost e.g. when close to an un-related message or due to device driver only looking at first message in a package.
> An all other aspects I believe the old and new approach to act the same.

I'd have to give the recording file a look after trying to figure out
which signal it is roughly. I think your explanation makes the most
sense. I have enough devices that the signals occasionally run together.

> Thanks :-) Nice to hear others are appreciating the work.

I've got some things I want to do that I'm waiting until merbanan
merges your work. Do you have a pull request in for the experimental
branch or just the one before you did the bin folding?

--Rob

[Tommy Vestermark]

Den onsdag den 8. juli 2015 kl. 23.05.43 UTC+2 skrev Robert Terzi:

I've got some things I want to do that I'm waiting until merbanan
merges your work.  Do you have a pull request in for the experimental
branch or just the one before you did the bin folding?

I have now merged the experimental branch into the pull request.

I am looking into whether the method is sane across all sensors and have made this overview of the currently known devices:
https://docs.google.com/spreadsheets/d/1x8nzCYxqt0mXZuKQb8sCsBo4o8NBxltL9-0Dou8deKo/edit?usp=sharing

My main problem is that many sensors don't have test data, which makes it difficult to ensure we will have no regression when converting...
I will try to go ahead with some more sensors.

/Tommy

[Benjamin Larsson]

Well that happens. This is clearly for the better.

> I will try to go ahead with some more sensors.
>
> /Tommy

/Benjamin

[Tommy Vestermark]

Den fredag den 10. juli 2015 kl. 13.18.57 UTC+2 skrev Benjamin Larsson:

Well that happens. This is clearly for the better.

Thanks for the merge :-)

I have made some tweaks to the detection and converted a couple of sensors. I am now out of sensors with good test data and will now work on adding a new one of my own :-) Later I will look into some of Roberts sensor data and make some stub drivers for those (it is very difficult to actually decode the meaning of the bits without having the sensors...)

@Robert: It looked like you have the Acurite and and Waveman - if you could add some good test captures to rtl_433_test it would be really helpfull and I could try to convert those as well.

/Tommy

[Robert Terzi]

On 7/10/2015 3:32 PM, Tommy Vestermark wrote:

> @Robert: It looked like you have the Acurite and and Waveman - if you could add some good test captures to rtl_433_test it would be really helpfull and I could try to convert those as well.

Sorry for the confusion, those are false positives from other devices (Lacrosse TX 433 and DSC security).  I don't have the Acurite or Waveman devices. 

If anyone has Acurite or Waveman devices, can you please add sample signals to the rtl_433_test repo please?

> Later I will look into some of Roberts sensor data and make some stub drivers for those (it is very difficult to actually decode the meaning of the bits without having the sensors...)

Don't worry about decoding those.  The recordings I've sent you were aimed at fine tuning the end-of-packet detection or signal analysis heuristics.

I have a decoder for the DSC 433 mhz security contacts (door/window, smoke, CO2, etc.)  I had written a custom demodulator based on pwm_p_decode.  I haven't made it available yet because I was struggling reliably detecting based on pulse width when the detect_level was changed.  I'm planning on rewriting it based on your new pulse demodulator and see if I can come up with something more robust when I can look at the list of pulses and distances.   Note: The DSC security contact is the device that doesn't transmit anything during a zero pulse. So when you receive a 1 pulse you need to figure out how many zero periods have elapsed.

I had also added to the rtl_433_test repository samples of the X10 power control signals.  At one point there was a decoder for X10 security contacts in rtl_433, but I think it got dropped somewhere along the way.  The are the signals that use the NEC IR format, the repeat gap is only about 4 x the length of the longest pulse, the first, wake-up pulse.

Thanks,
--Rob

Den fredag den 10. juli 2015 kl. 13.18.57 UTC+2 skrev Benjamin Larsson:

Well that happens. This is clearly for the better.

Thanks for the merge :-)

I have made some tweaks to the detection and converted a couple of sensors. I am now out of sensors with good test data and will now work on adding a new one of my own :-)

/Tommy

--
You received this message because you are subscribed to the Google Groups "rtl_433" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rtl_433+u...@googlegroups.com.
To post to this group, send email to rtl...@googlegroups.com.
Visit this group at http://groups.google.com/group/rtl_433.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rtl_433/fa8e3321-0065-4ffb-a030-df43c4b87c41%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tommy Vestermark]

Den fredag den 10. juli 2015 kl. 23.26.10 UTC+2 skrev Robert Terzi:

I have a decoder for the DSC 433 mhz security contacts (door/window, smoke, CO2, etc.)  I had written a custom demodulator based on pwm_p_decode.  I haven't made it available yet because I was struggling reliably detecting based on pulse width when the detect_level was changed.  I'm planning on rewriting it based on your new pulse demodulator and see if I can come up with something more robust when I can look at the list of pulses and distances.   Note: The DSC security contact is the device that doesn't transmit anything during a zero pulse. So when you receive a 1 pulse you need to figure out how many zero periods have elapsed.

I had also added to the rtl_433_test repository samples of the X10 power control signals.  At one point there was a decoder for X10 security contacts in rtl_433, but I think it got dropped somewhere along the way.  The are the signals that use the NEC IR format, the repeat gap is only about 4 x the length of the longest pulse, the first, wake-up pulse.

Hi Robert,

I have made a couple of stub drivers for both X10 and the DSC. You can find them here: https://github.com/vestom/rtl_433/tree/experimental
Please have a look and see, if at makes sense?

/Tommy
 

[Tommy Vestermark]

Hi,

I have refined the X10 and DSC drivers some more based on the excellent documentation links. I cannot interpret the bytes further without having some more documentation, but I believe they are ready for merging and a good starting point for someone else to finalize.

I have merged all of my experimental branch into the pull request - and will take a break for now at least until merge or reject :-)

On the plan ahead I have the following items (comments are welcome):
- Make acurite and waveman less noisy!
- Clean up all driver structs to compile with less warnings
- Make all drivers take bitbuffer as argument instead of discrete data. Then the drivers can use common helper functions from bitbuffer.
- Maybe convert remaining sample based drivers - even though they have no test data?? (Need some feedback)
- Try to implement FSK baseband decoding (I have a Danfoss sensor using that). That is a somewhat bigger task...

/Tommy

[Benjamin Larsson]

On 07/12/2015 08:39 AM, Tommy Vestermark wrote:
> Hi,
>
> I have refined the X10 and DSC drivers some more based on the excellent
> documentation links. I cannot interpret the bytes further without having
> some more documentation, but I believe they are ready for merging and a
> good starting point for someone else to finalize.

Goodie.

>
> I have merged all of my experimental branch into the pull request - and
> will take a break for now at least until merge or reject :-)
>
> On the plan ahead I have the following items (comments are welcome):
> - Make acurite and waveman less noisy!

I have the waveman transmitter. But it will take some time before I can
record some samples.

> - Clean up all driver structs to compile with less warnings
> - Make all drivers take bitbuffer as argument instead of discrete data.
> Then the drivers can use common helper functions from bitbuffer.
> - Maybe convert remaining sample based drivers - even though they have
> no test data?? (Need some feedback)

For long term development only decoders with ref signals can be
maintained. Feel free to do what you feel best.

> - Try to implement FSK baseband decoding (I have a Danfoss sensor using
> that). That is a somewhat bigger task...

I am a sucker for delay based FSK demodulation. You simply delay the
signal with x samples and then multiply it by itself. Then add a low
pass filter and a bias. After that a simple compare against zero will
tell you what frequency was transmitted. The computation cost for this
is very low and the low pass filter core could be reused. With some
trickery it is also possible to handle generic frequencies with the same
demodulation core. The tricky part is still the synchronization part.


>
> /Tommy
>

MvH
Benjamin Larsson