Trouble decoding ERT messages that rtlamr finds

596 views
Skip to first unread message

Ethan T

unread,
Apr 22, 2020, 12:40:09 AM4/22/20
to rtl_433
I'm trying to monitor my neighborhood's water meter to watch for leaks, and as such I'm playing with ERT decoding in both rtlamr (https://github.com/bemasher/rtlamr) and rtl_433. Having set up rtl_433 at home to monitor several weather sensors, I like its interface and low CPU usage. I have been unable to get rtlamr running with sufficient speed on an RPi2, but rtl_433 has no problem on it ... except that it's failing to find ERT packets that I know it can decode.

My first test was at home: I ran rtlamr for a bit, then ran rtl_433 for a bit. rtlamr was consistently able to find more ERT/SCM traffic than rtl_433 in the same place, with the same rtl-sdr dongle and antenna. So I know it can decode the data that rtlamr is seeing, it just seems to be far less sensitive to the good data.

For my main tests, I collected radio data with an rtl-sdr dongle in two places: one spot about 6 feet from the water meter's transmit antenna, and one about 100 meters away near our clubhouse, which is where I'd eventually like to set up the logging station with a directional antenna pointed at the water meter.

This data, fed into rtlamr, shows several packets in both recordings. However, the same file fed into rtl_433, shows no decoded ERT (a.k.a. SCM) packets. In fact, it shows no packets at all. I recorded the files with rtl_sdr using the standard settings that rtlamr uses, in auto gain mode, with frequency 912,600,115 Hz and sample rate 2,359,296 sps. I'm running rtl_433 like so:

    rtl_433 -M level -r near_club.iq.cu8 -s 2359296 -f 912600155 -R 149

I tried "-Y classic" and "-Y minmax". I also played with "-Y level=" values. None of them got me any packets.

"-vv" yields "Test mode file issued 13 packets" and "Test mode file issued 10 packets", but "-A" shows no packets at all for one file, and gibberish for the other.

I can upload the IQ files somewhere but they're 377mb and 487mb respectively. I also have the sample files created by rtlamr, which only include the packets it's able to decode, and those are 2-3mb each.

Any ideas?

Paul Harrington

unread,
May 14, 2020, 1:00:05 PM5/14/20
to rtl_433
I modified the ert.c code slightly (diffs against e310efe; see below) to add in more debugging and also to make the output consistent wrt rtlamr (I think the ert-type processing is incorrect in rtl_433) 

I ran rtlamr and rtl_433 side by side against rtlmux so they are working on identical bytestreams and I noticed a lot of DECODE_ABORT_LENGTH (I live in a suburb of Boston and there are a lot of meters chirping regularly). Curiously, one a single meter id appears in the rtl_433 output. I will do some further experiments and report back if anything interesting comes up.

regards,
pjjH



phrrngtn@sdrpi:~/work/rtl_433/build $ git diff
diff --git a/src/devices/ert.c b/src/devices/ert.c
index 3d11ffe..fffc0b1 100644
--- a/src/devices/ert.c
+++ b/src/devices/ert.c
@@ -54,18 +54,27 @@ static int ert_decode(r_device *decoder, bitbuffer_t *bitbuffer)
     data_t *data;

     if (bitbuffer->bits_per_row[0] != 96)
+      {
+       fprintf(stderr, "DECODE_ABORT_LENGTH %d\n", bitbuffer->bits_per_row[0]);
         return DECODE_ABORT_LENGTH;
+      }
+

     b = bitbuffer->bb[0];
     if (crc16(&b[2], 10, 0x6F63, 0))
+      {
+
+       fprintf(stderr, "ERT failed checksum\n");
         return DECODE_FAIL_MIC;
+      }
+

     /* Instead of detecting the preamble we rely on the
      * CRC and extract the parameters from the back */

     /* Extract parameters */
     physical_tamper = (b[3]&0xC0) >> 6;
-    ert_type = (b[3]&0x60) >> 2;
+    ert_type = (b[3]&0x36) >> 2;
     encoder_tamper = b[3]&0x03;
     consumption_data = (b[4]<<16) | (b[5]<<8) | b[6];
     ert_id = ((b[2]&0x06)<<23) | (b[7]<<16) | (b[8]<<8) | b[9];

Christian Zuckschwerdt

unread,
May 14, 2020, 1:31:02 PM5/14/20
to rtl_433

+       fprintf(stderr, "DECODE_ABORT_LENGTH %d\n", bitbuffer->bits_per_row[0]);

Maybe this can help to get more insight what the bitbuffer looks like:

bitbuffer_printf(bitbuffer, "%s: DECODE_ABORT_LENGTH ", __func__);

Paul Harrington

unread,
May 14, 2020, 1:58:43 PM5/14/20
to rtl_433
Thanks for that. I have appended some of the output from rtl_433 and also some redacted output from rtlamr (to comply with the wishes of the author of rtlamr as per github) from in or around the same time (i.e. they are not directly correlated. I just wanted to include some output with timestamps to show how much traffic rtlamr is seeing). I notice in the code a comment that says that it is not using the preamble (F2A60) so I am unsure if these kind of failed decodes might be expected? In any case, I am having a difficult time reproducing the output of rtlamr using LuaRadio but I thought that was 100% due to this being my first SDR project. When I googled for rtl_433 and rtlamr, I found this post so I am going to study the rtlamr implementation more closely and see if I can understand what it is doing that rtl_433 is not.

BTW, rtl_433 is a really cool looking piece of software. I am looking forward to learning more about it as I debug this particular issue.

pjjH


[00] { 7} 06        : 0000011
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] {17} 08 00 80  : 00001000 00000000 1
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] {14} 03 3c     : 00000011 001111
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] { 3} 20        : 001
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] { 4} 30        : 0011
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] {10} 6f c0     : 01101111 11
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] { 5} 08        : 00001
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] { 4} 30        : 0011
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] { 5} 78        : 01111
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] {10} 00 c0     : 00000000 11
ert_decode: DECODE_ABORT_LENGTH bitbuffer:: Number of rows: 1
[00] { 1} 00        : 0



{Time:2020-05-14T13:43:06.621 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx64 CRC:0x7806}}
{Time:2020-05-14T13:43:08.894 SCM:{ID:1xxxxx22 Type:12 Tamper:{Phy:03 Enc:00} Consumption:  xxx06 CRC:0x96A2}}
{Time:2020-05-14T13:43:09.659 SCM:{ID:6xxxxx89 Type: 4 Tamper:{Phy:00 Enc:01} Consumption:  xxx04 CRC:0xB8E4}}
{Time:2020-05-14T13:43:09.894 SCM:{ID:6xxxxx33 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx09 CRC:0x2E41}}
{Time:2020-05-14T13:43:10.237 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx64 CRC:0x7806}}
{Time:2020-05-14T13:43:13.175 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx64 CRC:0x7806}}
{Time:2020-05-14T13:43:15.952 SCM:{ID:6xxxxx33 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx09 CRC:0x2E41}}
{Time:2020-05-14T13:43:17.334 SCM:{ID:6xxxxx87 Type: 4 Tamper:{Phy:00 Enc:01} Consumption:  xxx99 CRC:0x6F13}}
{Time:2020-05-14T13:43:18.007 SCM:{ID:6xxxxx33 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx09 CRC:0x2E41}}
{Time:2020-05-14T13:43:19.067 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx64 CRC:0x7806}}
{Time:2020-05-14T13:43:20.392 SCM:{ID:6xxxxx87 Type: 4 Tamper:{Phy:00 Enc:01} Consumption:  xxx99 CRC:0x6F13}}
{Time:2020-05-14T13:43:22.109 SCM:{ID:6xxxxx33 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx09 CRC:0x2E41}}
{Time:2020-05-14T13:43:23.882 SCM:{ID:1xxxxx22 Type:12 Tamper:{Phy:03 Enc:00} Consumption:  xxx06 CRC:0x96A2}}
{Time:2020-05-14T13:43:24.289 SCM:{ID:6xxxxx03 Type: 4 Tamper:{Phy:00 Enc:00} Consumption:  xxx64 CRC:0x7806}}
{Time:2020-05-14T13:43:25.379 SCM:{ID:6xxxxx89 Type: 4 Tamper:{Phy:00 Enc:01} Consumption:  xxx04 CRC:0xB8E4}}

Christian Zuckschwerdt

unread,
May 14, 2020, 2:25:36 PM5/14/20
to rtl_433
The bitbuffers look badly truncated. I'd say the demod wasn't able to find a good signal level. The demod in rtl_433 is very generic (i.e. not tuned to a particular bitrate or modulation) and needs a clean signal.
If we can hint the demod with some protocol information (e.g. a matched low-pass filter) we could get a better performance. We hope to implement that option if you pick only a single decoder someday.

Perhaps you can record some samples (-S all) and identify where the demod fails by looking at PulseView (-W filename.sr).

Paul Harrington

unread,
May 14, 2020, 3:03:33 PM5/14/20
to rtl_433
will try PulseView. I'm afraid that the last time I did anything like this was over 30 years ago using a logic analyzer to debug memory access on a wire-wrapped board with an  8-bit CPU!  I was very relieved when we got to writing the software.

wrt a more specific demodulator, from reading https://github.com/bemasher/rtlamr/wiki/Signal-Processing and doing some messing around, the baud rate of ERT is 32K (32768) so at a sampling rate of 2359296, the number of samples per bit is 72. I presume that in conjunction with (part of) the preamble, an incredibly specific filter could be consed up?

pjjH

Benjamin Larsson

unread,
May 14, 2020, 3:08:19 PM5/14/20
to rtl...@googlegroups.com
>      /* Extract parameters */
>      physical_tamper = (b[3]&0xC0) >> 6;
> -    ert_type = (b[3]&0x60) >> 2;
> +    ert_type = (b[3]&0x36) >> 2;
Can you send a PR for this?

MvH
Benjamin Larsson

Benjamin Larsson

unread,
May 14, 2020, 3:12:02 PM5/14/20
to rtl...@googlegroups.com
I think one issue is a problem with the signal segmenter. I have an idea
what could be used instead but I have no time to implement it. Any takers?

MvH
Benjamin Larsson

Benjamin Larsson

unread,
May 14, 2020, 3:39:47 PM5/14/20
to rtl...@googlegroups.com
Den 2020-05-14 kl. 21:03, skrev Paul Harrington:
> will try PulseView. I'm afraid that the last time I did anything like
> this was over 30 years ago using a logic analyzer to debug memory access
> on a wire-wrapped board with an  8-bit CPU!  I was very relieved when we
> got to writing the software.
>
> wrt a more specific demodulator, from reading
> https://github.com/bemasher/rtlamr/wiki/Signal-Processing and doing some
> messing around, the baud rate of ERT is 32K (32768) so at a sampling
> rate of 2359296, the number of samples per bit is 72. I presume that in
> conjunction with (part of) the preamble, an incredibly specific filter
> could be consed up?
>
> pjjH
>

In rtl_433 all decoding steps are generalized. First we have a signal
detector, then we do a pulse recovery. Then we pass the pulse array
through different decoding steps (PWM, PPM, etc). Then we send the
decoder bitbuffers to the decoders.

I think the part that is failing is the signal detector. To triage use
rtl_mux and use rtl_433 to save signals with -S all and rtlamr to decode
data. If rtl_433 saves signals at the same time as rtlamr then we need
to look at the framing of the signal. Using pulseview to look at the
signal will give a good view of the pulses. As example you can use the
following command:

src/rtl_433 -s 2400k -r
../rtl_433_tests/tests/ert/scm/01/g001_912.6M_2.4M.cu8 -W test.sr

In this sample file we see that one pulse in the beginning and one at
the end is not recovered. It looks like this does not matter for the
manchester decoder but maybe it actually does with more sensitive signals.

So first check the signal detection then the framing and then that the
pulses match. Depending on the output from this we know which step needs
fixing.

MvH
Benjamin Larsson

Paul Harrington

unread,
May 14, 2020, 5:06:48 PM5/14/20
to rtl_433
Thank you very much! This is very useful-looking advice.

I generated a PR for the ERT Endpoint Type masking.

pjjH

Steven Michael

unread,
May 14, 2020, 6:29:25 PM5/14/20
to rtl_433
I can also confirm that comparing rtlamr and rtl_433, rtlamr is consistently able to find 4 different SCM meters, while rtl_433 is only able to find one. Additonally, rtlamr has decoding for SCM+ which is "Similar to SCM, allows greater precision and longer meter ID's.", which would be great to add to rtl_433!

On Wednesday, April 22, 2020 at 12:40:09 AM UTC-4, Ethan T wrote:

Paul Harrington

unread,
May 15, 2020, 11:45:36 AM5/15/20
to rtl_433
I have worked on SCM+ and IDM frames based off description in https://github.com/bemasher/rtlamr/wiki/Protocol
They are both byte-aligned and pretty straightforward in structure. However, I am new to SDR/DSP and have not yet learned how to demodulate ERT so all I have done thus far with the protocol descriptions is shred them to more structured tabular metadata format. I presume once the demodulation is working correctly, the parsing part will be trivial/code-gennable. I have been trying to reverse engineer the DSP/demodulation from the rtlamr source code but have not made any progress so am excited to try out the techniques suggested by Benjamin Larsson upthread.  I will try to add them as flex decoders as per https://github.com/merbanan/rtl_433/wiki/Adding-a-new-remote-device once I become familiar with the rtl_433 toolchain. Alternatively, if stuff like this should live in ert.c, I can put it in there.

pjjH

Benjamin Larsson

unread,
May 15, 2020, 4:32:43 PM5/15/20
to rtl...@googlegroups.com
Den 2020-05-15 kl. 17:45, skrev Paul Harrington:
> I have worked on SCM+ and IDM frames based off description in
> https://github.com/bemasher/rtlamr/wiki/Protocol
> They are both byte-aligned and pretty straightforward in structure.
> However, I am new to SDR/DSP and have not yet learned how to demodulate
> ERT so all I have done thus far with the protocol descriptions is shred
> them to more structured tabular metadata format. I presume once the
> demodulation is working correctly, the parsing part will be
> trivial/code-gennable. I have been trying to reverse engineer the
> DSP/demodulation from the rtlamr source code but have not made any
> progress so am excited to try out the techniques suggested by Benjamin
> Larsson upthread.  I will try to add them as flex decoders as per
> https://github.com/merbanan/rtl_433/wiki/Adding-a-new-remote-device once
> I become familiar with the rtl_433 toolchain. Alternatively, if stuff
> like this should live in ert.c, I can put it in there.
>
> pjjH
>

I suggest you open a github ticket so we can track this issue there. And
if we get a SCM+ sample we can look into what is needed to add the
protocol.

MvH
Benjamin Larsson

Paul Harrington

unread,
May 15, 2020, 4:42:56 PM5/15/20
to rtl_433
will piggyback on "Add support for SCM+" #1378

Paul Harrington

unread,
May 18, 2020, 5:45:42 PM5/18/20
to rtl_433

opened https://github.com/merbanan/rtl_433/issues/1386 for the problems with rtl_433 not finding the same messages as rtlamr. rtlamr has an option to record the raw samples so I did that and uploaded the data to dropbox. 
Reply all
Reply to author
Forward
0 new messages