Adding FM/FSK demodulation and reworking baseband processing

[Tommy Vestermark]

Hi,

I am currently working on adding support for Frequency Shift Keying encoded signals in addition to On/Off Keying and is therefore making somewhat bigger changes to the baseband processing. Before going too far down the road, I would like to discuss some items:

1. Removal of hidden support for decimation. The code currently has an undocumented option (-c) for decimating the sample rate of the input signal. I believe this functionality is fundamentally broken, as the pulse timing in the decoders is not compensated for proper decoding. Furthermore the decimation is done without any anti-aliasing filtering and will result in excessive noise and artefacts. Instead I will recommend to just adjust the input sample rate, such that filtering, timing, CPU usage etc. scales correctly.
   Will anyone miss, if I remove the code for this?
2. Target hardware. The sample-by-sample processing in the baseband code is somewhat CPU sensitive. Some of the current code has some explicit optimizations like envelope_detect() ("AM demodulator"), which uses a look-up table instead of integer arithmetic. As most CPUs capable of running a contemporary Linux kernel will have cheap integer arithmetic and expensive memory access I believe some of these optimizations are dubious. Even somewhat usable floating-point is becoming prevalent.
   My personal definition of a low-end target for this program is something like a Raspberry Pi (using it myself :-) and I would make its performance characteristics determine the optimizations. What do other consider an optimization target?
3. Generic FSK device support. I have single FSK modulated device myself, driving my development. However there is a risk of optimizing too much for this single device. Do anyone have some FSK modulated devices they would like to have taken into consideration? Then please make some test data recordings, descriptions etc. and make a pull request to https://github.com/vestom/rtl_433_tests (I will push it upstream).

Comments welcome :-)

/Tommy

[Benjamin Larsson]

On 08/13/2015 09:08 PM, Tommy Vestermark wrote:
> Hi,
>
> I am currently working on adding support for Frequency Shift Keying
> encoded signals in addition to On/Off Keying and is therefore making
> somewhat bigger changes to the baseband processing. Before going too far
> down the road, I would like to discuss some items:
>

> 1. Removal of hidden support for decimation. The code currently has an

> undocumented option (-c) for decimating the sample rate of the input
> signal. I believe this functionality is fundamentally broken, as the
> pulse timing in the decoders is not compensated for proper decoding.
> Furthermore the decimation is done without any anti-aliasing
> filtering and will result in excessive noise and artefacts. Instead
> I will recommend to just adjust the input sample rate, such that
> filtering, timing, CPU usage etc. scales correctly.
> Will anyone miss, if I remove the code for this?

The aliasing doesn't harm IIRC. The origins of the decimation code was
from when I used a 1024k sample rate. It should be fairly easy to re-add
decimation support if someone needs it. Thus feel free to remove it for now.

> 2. Target hardware. The sample-by-sample processing in the baseband

> code is somewhat CPU sensitive. Some of the current code has some
> explicit optimizations like envelope_detect() ("AM demodulator"),
> which uses a look-up table instead of integer arithmetic. As most
> CPUs capable of running a contemporary Linux kernel will have cheap
> integer arithmetic and expensive memory access I believe some of
> these optimizations are dubious. Even somewhat usable floating-point
> is becoming prevalent.

Well these tables should fit in the data cache.

> My personal definition of a low-end target for this program is
> something like a Raspberry Pi (using it myself :-) and I would make
> its performance characteristics determine the optimizations. What do
> other consider an optimization target?

RPi as main target is fine by me. But I'd rather not use float at all.
If float code is added I would like to have a fixed point version also
available for regression testing.


> 3. Generic FSK device support. I have single FSK modulated device

> myself, driving my development. However there is a risk of
> optimizing too much for this single device. Do anyone have some FSK
> modulated devices they would like to have taken into consideration?
> Then please make some test data recordings, descriptions etc. and
> make a pull request to https://github.com/vestom/rtl_433_tests (I
> will push it upstream).

No Fsk modulation devices here.

>
> Comments welcome :-)
>
> /Tommy

MvH
Benjamin Larsson

[Tommy Vestermark]

On Thursday, August 13, 2015 at 9:33:40 PM UTC+2, Benjamin Larsson wrote:

The aliasing doesn't harm IIRC. The origins of the decimation code was
from when I used a 1024k sample rate. It should be fairly easy to re-add
decimation support if someone needs it. Thus feel free to remove it for now.

You are right. I can see it work for OOK signals and getting a higher BW for almost "free" :-)
Being an issue for FSK and currently hidden/broken, I will remove it for now.

Well these tables should fit in the data cache.  

Yes - when fetched. However even cache latency is significant these days, and operations on register variables are often "free", as they are masked by the load/stores. I have had my assumptions challenged a couple of times and now lean towards simple code first and no low-level optimization without measurements. However your mileage may vary :-)

 

RPi as main target is fine by me. But I'd rather not use float at all.
If float code is added I would like to have a fixed point version also
available for regression testing.

People are even using floats on their 8 bit Arduinos now ;o) On a modern i86, floats are not much more expensive than int - but that is not my target. I will avoid float in the signal processing hot path, but see no harm in using a single float calculation here and there (If in doubt: measure).
Just curious, do you have some other lower performance processor in mind?

I see you are proficient in fixed-point signal processing and scaling :-) Back in my DSP days, cycles were expensive, compilers bad and memory access single cycle. These days memory access and branches should be avoided and compilers can often optimize better when you don't.

/Tommy

[Benjamin Larsson]

On 08/13/2015 11:45 PM, Tommy Vestermark wrote:
> On Thursday, August 13, 2015 at 9:33:40 PM UTC+2, Benjamin Larsson wrote:
>
> The aliasing doesn't harm IIRC. The origins of the decimation code was
> from when I used a 1024k sample rate. It should be fairly easy to
> re-add
> decimation support if someone needs it. Thus feel free to remove it
> for now.
>
> You are right. I can see it work for OOK signals and getting a higher BW
> for almost "free" :-)
> Being an issue for FSK and currently hidden/broken, I will remove it for
> now.
>
> Well these tables should fit in the data cache.
>
> Yes - when fetched. However even cache latency is significant these
> days, and operations on register variables are often "free", as they are
> masked by the load/stores. I have had my assumptions challenged a couple
> of times and now lean towards simple code first and no low-level
> optimization without measurements. However your mileage may vary :-)

Well true, but it was fun code to write. Anyway working code is more
important then optimized code for this project.

> RPi as main target is fine by me. But I'd rather not use float at all.
> If float code is added I would like to have a fixed point version also
> available for regression testing.
>
> People are even using floats on their 8 bit Arduinos now ;o) On a modern
> i86, floats are not much more expensive than int

Floats are faster even.

> - but that is not my
> target. I will avoid float in the signal processing hot path, but see no
> harm in using a single float calculation here and there (If in doubt:
> measure).
> Just curious, do you have some other lower performance processor in mind?

Not really. I want to get it running on a dual threaded 400MHz mips
(OpenWrt router) eventually.

>
> I see you are proficient in fixed-point signal processing and scaling
> :-)

Ha, took me a long time to get that part of the code working ...

> Back in my DSP days, cycles were expensive, compilers bad and memory
> access single cycle. These days memory access and branches should be
> avoided and compilers can often optimize better when you don't.
>
> /Tommy

MvH
Benjamin Larsson

[Robert Terzi]

On 8/13/2015 3:08 PM, Tommy Vestermark wrote:

Do anyone have some FSK modulated devices they would like to have taken into consideration? Then please make some test data recordings, descriptions etc. and make a pull request to https://github.com/vestom/rtl_433_tests (I will push it upstream).

I can contribute the LaCrosse 915 Mhz temperature/humidity/wind/rain sensors which are FSK.  

How would you like the signals to be recorded?

If I've followed along correctly, rtl_433 -m 1 filename, should give 8 bit I/Q samples.  

If the recordings from rtl_433 -a -t are sufficient, because I think those also give 8 bit I/Q samples(?), See rtl_433_tests/tests/lacrosse/02.

@magellannh (github id) has code that uses rtl_fm https://github.com/magellannh/RPI_Efergy, rtl_wx https://github.com/magellannh/rtl-wx, decodes an Efergy device, but I'm not sure if that is the same FSK one or not.

Also @batilan (github id) posted some links in issue #134, about decoding an another energy monitor using gnu-radio: https://github.com/merbanan/rtl_433/issues/134

Hope this helps,
--Rob

[Tommy Vestermark]

Hi Robert,

On Friday, August 14, 2015 at 4:19:05 AM UTC+2, Robert Terzi wrote:

If the recordings from rtl_433 -a -t are sufficient, because I think those also give 8 bit I/Q samples(?), See rtl_433_tests/tests/lacrosse/02.

I/Q recordings from -a -t are preferred.

I looked into the files under rtl_433_tests/tests/lacrosse/02 with Audacity, and I am positively sure, they are not FSK encoded. It looks mostly like OOK PCM NRZ encoding. The pulses/bits are only 30 samples wide, so they were filtered away before we relaxed the low pass filter.
If you try the newest code, you will see it identify the pulses nicely. There seem to be some long runs of 1s, which may make it a bit tricky to decode. PCM coding should always include some frequent bit changes (e.g. start/stop bits) to ensure proper synchronization. But it may also be some sort of delimiter. The PCM RZ decoder should work with NRZ coding when short and long limit is set to same value. Try to set them both to 30 and see what happens. Have fun :-)

/Tommy

[Batilan]

Hi Tommy,

Thanks for all the great work you have bee doing recently, about your FSK endeavors:

Op donderdag 13 augustus 2015 21:08:45 UTC+2 schreef Tommy Vestermark:

Hi,

I am currently working on adding support for Frequency Shift Keying encoded signals in addition to On/Off Keying and is therefore making somewhat bigger changes to the baseband processing. Before going too far down the road, I would like to discuss some items:

1. Removal of hidden support for decimation. The code currently has an undocumented option (-c) for decimating the sample rate of the input signal. I believe this functionality is fundamentally broken, as the pulse timing in the decoders is not compensated for proper decoding. Furthermore the decimation is done without any anti-aliasing filtering and will result in excessive noise and artefacts. Instead I will recommend to just adjust the input sample rate, such that filtering, timing, CPU usage etc. scales correctly.
   Will anyone miss, if I remove the code for this?

If I read the above, probably not ;-) I once tried to use it to no avail. 

1. Target hardware. The sample-by-sample processing in the baseband code is somewhat CPU sensitive. Some of the current code has some explicit optimizations like envelope_detect() ("AM demodulator"), which uses a look-up table instead of integer arithmetic. As most CPUs capable of running a contemporary Linux kernel will have cheap integer arithmetic and expensive memory access I believe some of these optimizations are dubious. Even somewhat usable floating-point is becoming prevalent.
   My personal definition of a low-end target for this program is something like a Raspberry Pi (using it myself :-) and I would make its performance characteristics determine the optimizations. What do other consider an optimization target?

I agree with RPi also (also the original RPi1). Currently I have no other targets in mind myself. 

1. Generic FSK device support. I have single FSK modulated device myself, driving my development. However there is a risk of optimizing too much for this single device. Do anyone have some FSK modulated devices they would like to have taken into consideration? Then please make some test data recordings, descriptions etc. and make a pull request to https://github.com/vestom/rtl_433_tests (I will push it upstream).

I own a couple of "Conrad Energy Count 3000" aka "Velleman NETBSEM4” and “La Crosse Techology Remote Cost Control Monitor – RS3620”.

You might want to have a look at a blogpost I wrote: https://batilanblog.wordpress.com/2015/02/17/using-ec3k-with-raspberry-pi/ (skip the whole linux / installation bit but the references are interesting

The ec3k software is gnuradio based and is quite CPU intensive, probably also because it cascades several gnuradioblocks together which is probably not the most efficient way of decoding, also an interesting "artifact" is seen when tuning for the right frequency (described in the article at  the heading "Use it (and fix another issue…)")

BTW: these packets can be quite easily decoded by a Atmel based uP using an  ATmega328p processor and an RFM12B radio module (I'm using a Jeelink), I guess the RFM12B radio is taking care of quite some processing that when using an RTL-SDR dongle is done by the low level DSP part on the processor. 

However I wonder if we can somehow cooperate for the higher level decodings with the uP world (like Atmel, TI TM4C123 etc.), a lot is going on there and a lot of decoders are available already. If we can join forces that would probably benefit both "worlds". Probably this is not too relevant for the FSK decoding, but we might want t have a look at this aspect.

 

Comments welcome :-)

/Tommy

[Robert Terzi]

On 8/14/2015 1:25 PM, Tommy Vestermark wrote:

On Friday, August 14, 2015 at 4:19:05 AM UTC+2, Robert Terzi wrote:

If the recordings from rtl_433 -a -t are sufficient, because I think those also give 8 bit I/Q samples(?), See rtl_433_tests/tests/lacrosse/02.

I/Q recordings from -a -t are preferred.

I looked into the files under rtl_433_tests/tests/lacrosse/02 with Audacity, and I am positively sure, they are not FSK encoded. It looks mostly like OOK PCM NRZ encoding. The pulses/bits are only 30 samples wide, so they were filtered away before we relaxed the low pass filter.

Those files were created with -a -t a number of months ago.  So, I think you are saying that -a -t creates 8 bit files which are low pass filtered?

Does your latest code create 8 bit raw I/Q before filtering with -a -t?





 

[Tommy Vestermark]

No! The beauty is that the files in the test repository are raw, untouched, I/Q files and can be used to test ALL changes. It is really a stroke of genius by Benjamin to create the test repository. Without this, it would be difficult to test the code, and I wouldn't dare make the invasive changes I have made.
Had the files already been filtered, I wouldn't have been able to test changes to the filtering nor make a conclusion about the type of modulation of the Lacrosse sensor. I have attached the Audacity screenshot, where it is visibly an OOK signal. If in doubt, it is always very informative to look at the signals :-)
 

Does your latest code create 8 bit raw I/Q before filtering with -a -t?

I have not touched the -a -t code. It still makes raw I/Q files as it should. I have only cleaned up the -m 0, -m 1 modes and added a -m 2 mode for FM demodulated files for FSK (not merged yet).
 
/Tommy

[Tommy Vestermark]

On Friday, August 14, 2015 at 11:47:37 PM UTC+2, Batilan wrote:

I own a couple of "Conrad Energy Count 3000" aka "Velleman NETBSEM4” and “La Crosse Techology Remote Cost Control Monitor – RS3620”.

You might want to have a look at a blogpost I wrote: https://batilanblog.wordpress.com/2015/02/17/using-ec3k-with-raspberry-pi/ (skip the whole linux / installation bit but the references are interesting

 

That may be very useful! Is it possible you could make some recordings/captures with rtl_433 -a -t of these transmitters and make a pull request against
https://github.com/vestom/rtl_433_tests ?
 

However I wonder if we can somehow cooperate for the higher level decodings with the uP world (like Atmel, TI TM4C123 etc.), a lot is going on there and a lot of decoders are available already. If we can join forces that would probably benefit both "worlds". Probably this is not too relevant for the FSK decoding, but we might want t have a look at this aspect.

 

The rtl_433 code currently have 4 layers:

1. AM (and FM) demodulation. Baseband demodulation from I/Q samples to pulse samples.
   
2. OOK (and FSK) demodulation. Digitization and pulse packet detection. Outputs packages of pulse descriptions.
3. Digital demodulation (PWM, PPM, PCM, Manchester, ...) of pulse data. Outputs bit buffers.
4. Device decoding. Individual protocol decoding for each device. Outputs text to stdout.
   

It is mainly the two first layers, that are CPU intensive and it would be easy to add support for inputting hardware demodulated signals. For embedded use the device decoders should probably be reworked to output binary data instead of printing directly to text. There has been some previous work of introducing a 5th layer with generic data to JSON conversion, but I unfortunately don't know the staus of that.

At least I think there is huge benefit of sharing the knowledge regarding the devices. The https://github.com/merbanan/rtl_433_tests repository could be an excellent centralized source of knowledge. Currently it is very fragmented, and everyone makes their own web-pages for their 1-2 devices. Maybe someone could start a "wireless sensor wiki"?

/Tommy

[Batilan]

Op zaterdag 15 augustus 2015 13:32:37 UTC+2 schreef Tommy Vestermark:

On Friday, August 14, 2015 at 11:47:37 PM UTC+2, Batilan wrote:

I own a couple of "Conrad Energy Count 3000" aka "Velleman NETBSEM4” and “La Crosse Techology Remote Cost Control Monitor – RS3620”.

You might want to have a look at a blogpost I wrote: https://batilanblog.wordpress.com/2015/02/17/using-ec3k-with-raspberry-pi/ (skip the whole linux / installation bit but the references are interesting

 

That may be very useful! Is it possible you could make some recordings/captures with rtl_433 -a -t of these transmitters and make a pull request against
https://github.com/vestom/rtl_433_tests ?

Just posted some samples on rtl_433_test (tests/ec3k/01), hope it can be of some use to you.

If we can generate some bit-sequences from the FSK signal I would like to support the higher-level layer decoding (all credits go to Tomaz Solc for ec3k and "Thosch" on http://forum.jeelabs.net/comment/4972.html#comment-4972)

It should be quite trivial from the ec3k code.

Sorry, now see you requested the pull request against your fork of rtl_433_test, I did the request directly on Benjamin's master

[Tommy Vestermark]

Den søndag den 16. august 2015 kl. 23.04.09 UTC+2 skrev Batilan:

Just posted some samples on rtl_433_test (tests/ec3k/01), hope it can be of some use to you.

If we can generate some bit-sequences from the FSK signal I would like to support the higher-level layer decoding (all credits go to Tomaz Solc for ec3k and "Thosch" on http://forum.jeelabs.net/comment/4972.html#comment-4972)

It should be quite trivial from the ec3k code.

Thanks! The signals have a rather large frequency offset tripping up the current naive DC blocker filter (see screendump). It is also a rather difficult signal, as the pulses are only ~15 samples long. So very helpful...

Sorry, now see you requested the pull request against your fork of rtl_433_test, I did the request directly on Benjamin's master

That's OK, Benjamin has already merged :-)

[Batilan]

Hi Tommy,

I hope it helps indeed! Please let me know if I can help in any other way. 

I have a few questions regarding the signals: The ec3k software uses auto gain settings and it is clearly noticeable that when the gain is increased the number of correctly received samples increases, however it also seems that the gain that is finally reached seems not optimal. I wonder how the "default" gain settings of rtl_433 affect the samples, might it affect the usability of the captured samples?

Also I wonder if I can use the data files with  I/Q samples with software like SDR# or gnuradio (e.g. for waterfall or  FFT plots) to find out more about the signal (off course I can also do this in real-time with my SDR dongle, but inspecting someone elses samples might also be interesting).

What steps will you use to extract  bit-sequences from the samples? The info in this post http://forum.jeelabs.net/comment/4020.html#comment-4020 (from thosch) is probably helpful when trying to decode the signal. 

[Tommy Vestermark]

Hi,

Some news. I have worked out the complete FSK decoder stack and implemented stub drivers for a Danfoss Thermostat (my own) and Batilans E3Ck sensor.

• The baseband FM demodulation is currently a bit naive (only using imaginary part of phase difference and not complete angle difference), but seems to work OK with the two sensors, I have test data for. We can make it better later, if need be. The filtering is rather gentle due to the E3Ck having very short pulses (~12 samples!). It would probably work better on a higher sample rate, but I have tried to catch up on robustness in the quantizer instead.
• The FSK pulse quantizer/demodulator is combined with the OOK, as there is overlap in functionality (the FSK needs carrier detection), and we want to be able to determine, what type of modulation is used. The quantizer needs to be adaptive as the sensors carrier frequency offset can vary significantly to either side of the tuner frequency. Fortunately the two example sensors are on either side of the carrier :-)
  
• Both sensors use PCM NRZ modulation and I reworked the PCM RZ demodulator to be capable of both. Currently the E3Ck does not work very stable due to the very short pulses and the tolerance requirements of the demodulator. The Danfoss sensor seems stable. I know Robert would like to have the PCM demodulator more tolerant as well :-)
• Rework of rtl_433.c to include the new infrastructure and support for saving FM demodulated files for test/verification.

I simulated the baseband filtering and quantizer in Octave before implementation and have included graphs for both sensors with output from both Octave and rtl_433. They seem to mach well :-)

I will regression test and fine-tune a bit more before upstreaming - but it should be ready for tinkering and feedback is very welcome :-)

It is available here: https://github.com/vestom/rtl_433/tree/experimental

/Tommy

[Batilan]

Hi Tommy,

Thanks! I cloned your fork, checked out the experimental branch (git checkout origin/experimental; git checkout -b experimental origin/experimental)

and built it (cmake make). Then tried to run it to see what it makes out of my ec3k plugs. Unfortunately I'didn't see any useful data yet.

Any way we can proceed with this? Let me know if I can help (or if you can help me getting usefull packet decodes for ec3k :-)

Here is some output I receive on ./src/rtl_433 -a  -f 868255000 (this is the same frequency where ec3k receives most packets OK)

I already tried a few other frequencies.

p_limit: 6906

bitbuffer:: Number of rows: 0

*** signal_start = 8272086, signal_end = 8298988

signal_len = 26902,  pulses = 1

Distance coding: Pulse length 6901

Short distance: 1000000, long distance: 0, packet distance: 0

p_limit: 6901

bitbuffer:: Number of rows: 0

*** signal_start = 8426356, signal_end = 8496260

signal_len = 69904,  pulses = 3

Iteration 1. t: 6911    min: 6898 (2)    max: 6925 (1)    delta 25

Iteration 2. t: 6911    min: 6898 (2)    max: 6925 (1)    delta 0

Distance coding: Pulse length 6911

Short distance: 12056, long distance: 0, packet distance: 14591

p_limit: 6911

bitbuffer:: Number of rows: 3

[00] {0} :

[01] {0} :

[02] {0} :

*** signal_start = 8709783, signal_end = 8782014

signal_len = 72231,  pulses = 2

Iteration 1. t: 3997    min: 1090 (1)    max: 6904 (1)    delta 0

Pulse coding: Short pulse length 1090 - Long pulse length 6904

Short distance: 44236, long distance: 0, packet distance: 44236

p_limit: 3997

bitbuffer:: Number of rows: 2

[00] {1} 00 : 0

[01] {1} 80 : 1

*** signal_start = 9528282, signal_end = 9555203

signal_len = 26921,  pulses = 1

Distance coding: Pulse length 6920

Short distance: 1000000, long distance: 0, packet distance: 0

p_limit: 6920

bitbuffer:: Number of rows: 0

*** signal_start = 9681524, signal_end = 9772337

signal_len = 90813,  pulses = 3

Iteration 1. t: 6908    min: 6893 (1)    max: 6924 (2)    delta 1

Iteration 2. t: 6908    min: 6893 (1)    max: 6924 (2)    delta 0

Distance coding: Pulse length 6908

Short distance: 6756, long distance: 6756, packet distance: 43314

p_limit: 6908

bitbuffer:: Number of rows: 2

[00] {0} :

[01] {0} :

*** signal_start = 10005018, signal_end = 10031924

signal_len = 26906,  pulses = 1

Distance coding: Pulse length 6905

Short distance: 1000000, long distance: 0, packet distance: 0

[Tommy Vestermark]

Hi Batilan,

Beware! You will not get any useful data by using analyser mode (rtl_433 -a), as the analyser bypasses the demodulator infrastructure and does its own OOK only analysis. You should instead run it without parameters or with debug output (rtl_433 -D).

You should also really make a new check out of my experimental (git pull https://github.com/vestom/rtl_433/tree/experimental), as I have fixed lots of bugs. You should be able to get some raw data packages with that.

Finally the E3Ck sensor is really challenging, as its pulses are apparently only 12.5 samples long, which means errors will easily accumulate with longer runs of bits. Even if you run it with a higher sample rate, we cannot set the limits correctly in the device file.

/Tommy