ODR-AudioEnc: TAI-UTC offset always defaults to 37 seconds

160 views
Skip to first unread message

Felix Hofmann

unread,
Feb 6, 2022, 12:28:30 PM2/6/22
to mmbtools
Hello,

I just configured my test setup to use EDI in order to transport data from the ODR-AudioEnc to ODR-DabMux. While this works well, for some reason, the delay is huge. After checking the log of ODR-AudioEnc, I discovered that it states:

"Initialised TAI-UTC offset to 37s"

Changing the value of -T does not appear to have any effect, it always defaults to 37 seconds. Has anyone else encountered this before? Does someone know a solution? Any help would be greatly appreciated.

Thank you and kind regards

Matthias Brändli

unread,
Feb 6, 2022, 2:14:45 PM2/6/22
to crc-mm...@googlegroups.com
Hello Felix,

The TAI - UTC offset is currently 37s, and changes when a leap second is
inserted in UTC to compensate for changes in earth rotation rate.

For more details see https://en.wikipedia.org/wiki/Leap_second


When you say "the delay is huge" do you mean end-to-end audio latency?

Cheers
mpb
> --
> You received this message because you are subscribed to the Google
> Groups "mmbtools" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crc-mmbtools...@googlegroups.com
> <mailto:crc-mmbtools...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crc-mmbtools/61b1fa3d-32bb-4c13-b78f-1c0e2c543c3an%40googlegroups.com
> <https://groups.google.com/d/msgid/crc-mmbtools/61b1fa3d-32bb-4c13-b78f-1c0e2c543c3an%40googlegroups.com?utm_medium=email&utm_source=footer>.

Felix Hofmann

unread,
Feb 6, 2022, 3:02:12 PM2/6/22
to mmbtools
Dear Matthias,

thank you for the quick reply and extensive information. Yes, the end-to-end latency is very large, it is well over 30 seconds. This only occurred when I switched from zmq to EDI. In this test, ODR-AudioEnc was running with the following parameters:

odr-audioenc -v rtp://@:8365 -D -C 200 -L --audio-resampler=samplerate --aaclc -c 2 -p 64 -b 144 -r 48000 -g -3 -s 60 -e tcp://localhost:9003 -T 0 -P PROG1
directory=/home/dab/dab/config/mot/PROG1

and ODR-DabMux was configured like this:

sub_radio3 {
        type dabplus
        bitrate 144
        id 3
        protection 4
       
        inputproto edi
        inputuri "tcp://*:9003"
        buffer-management timestamped
        buffer 40
    } 

I was originally trying to optimize my setup because I have issues with the end-to-end audio delay increasing indefinitely when the setup is running 24/7. I thought that switching to EDI and setting the buffer management to "timestamped", I would be able to resolve this issue. Is that assumption correct?

The following graph shows the end to end audio of one of the programs while still running using zmq (the station's direct studio output is compared to the audio signal of a DAB+ receiver tuned to that same station). A workaround is to restart all services every night at 3:00 AM using crontab, but that is not very elegant.

Delay.bmp

Cheers
Felix

Matthias Brändli

unread,
Feb 6, 2022, 3:53:11 PM2/6/22
to crc-mm...@googlegroups.com
Hi Felix,

ah wonderful, someone is using buffer-management timestamped :-D I'm
happy to get feedback!

What are your mux output settings?

Do you confirm an ntp client (ntpd or chrony) is properly running on
both encoder and mux machine?

mpb
> <https://groups.google.com/d/msgid/crc-mmbtools/61b1fa3d-32bb-4c13-b78f-1c0e2c543c3an%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/crc-mmbtools/61b1fa3d-32bb-4c13-b78f-1c0e2c543c3an%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "mmbtools" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crc-mmbtools...@googlegroups.com
> <mailto:crc-mmbtools...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crc-mmbtools/1af14402-acb0-4383-a2ca-f58595ad5a36n%40googlegroups.com
> <https://groups.google.com/d/msgid/crc-mmbtools/1af14402-acb0-4383-a2ca-f58595ad5a36n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Felix Hofmann

unread,
Feb 6, 2022, 4:33:10 PM2/6/22
to mmbtools
Hi Matthias,

yes, the "timestamped" option seems like the most logical way to go and I'm really happy that it has been implemented, so thank you very much for your work :-). For ODR-DabMux, the output is configured as follows:

outputs {

    zmq "zmq+tcp://*:18081"

    test "simul://" 

}

The audio encoders and the multiplexer are running on the same machine. The output of the multiplexer is then fed to a EasyDABv2 exciter board. The output of timedatectl on the encoder/multiplexer is:

Local time: Sun 2022-02-06 22:24:07 CET
Universal time: Sun 2022-02-06 21:24:07 UTC
RTC time: Sun 2022-02-06 21:24:07
Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: yes
NTP service: inactive
RTC in local TZ: no

so everything is looking good. Chrony is also running. I just did a few more tests: When I stop ODR-DabMux, the audio is gone immediately. That tells me that the issue is not between the transmitter and the multiplexer. When I stop ODR-AudioEnc, there is a delay of exactly 37 seconds which is why I thought that the TAI-UTC offset has something to do with this. When I use zmq instead of EDI, the delay is much lower (only about 2 seconds) but the delay keeps growing indefinitely. If you would like to take a look at the configuration, I would be happy to enable SSH access for you. This is only a test setup in my lab, the production unit that is actually broadcasting is still running with zmq and the buffer option "prebuffering" but will be switched over to EDI with "buffer-management timestamped" once the cause of this strange phenomenon is identified. :-)

Cheers

Felix 


Matthias Brändli

unread,
Feb 9, 2022, 5:32:29 AM2/9/22
to crc-mm...@googlegroups.com
Hi Felix,

yes that's the whole point of the timestamped operation: control the
end-to-end latency!

I'm surprised by

> NTP service: inactive

and you telling me

> Chrony is also running

This looks contradicting to me. What does `chronyc tracking` output?


Regarding the buffering, what do you see in the max min columns in the
stats? Those show min and max buffer size in the last 30s. (Measured in
bytes, which is actually rather meaningless for the end-user, I know...)
Use this script: ODR-DabMux/doc/show_dabmux_stats.py

You'll need managementport 12720 in your mux config


It's not excluded that the 37s TAI-UTC clock offset has some influence
here, but it shouldn't!

Before we look further: are you running releases (master branch) or code
from the "next" branch? Please run "next", maybe there are some
improvements that I didn't release yet, and more importantly we'll look
at the same version!

mpb
> <https://groups.google.com/d/msgid/crc-mmbtools/1af14402-acb0-4383-a2ca-f58595ad5a36n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/crc-mmbtools/1af14402-acb0-4383-a2ca-f58595ad5a36n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "mmbtools" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crc-mmbtools...@googlegroups.com
> <mailto:crc-mmbtools...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crc-mmbtools/9e4fd9b1-17cc-4255-8124-58b9db78e546n%40googlegroups.com
> <https://groups.google.com/d/msgid/crc-mmbtools/9e4fd9b1-17cc-4255-8124-58b9db78e546n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Felix Hofmann

unread,
Feb 19, 2022, 12:14:28 PM2/19/22
to mmbtools
Dear Matthias,

thank you and apologies for the late reply, I was extremely busy so I didn't get around to do further testing until today. I just updated all ODR components to the "next" branch, the issue currently still persists.

chronyc-tracking returns:

Reference ID    : 81468425 (stratum2-4.NTP.TechFak.NET)
Stratum         : 3
Ref time (UTC)  : Sat Feb 19 17:10:14 2022
System time     : 0.000284899 seconds slow of NTP time
Last offset     : -0.000129834 seconds
RMS offset      : 0.000627513 seconds
Frequency       : 11.001 ppm slow
Residual freq   : +0.006 ppm
Skew            : 0.917 ppm
Root delay      : 0.021726156 seconds
Root dispersion : 0.002185798 seconds
Update interval : 129.7 seconds
Leap status     : Normal

The Python Script returns:

Statistics from ODR-DabMux v4.0.0


id                       max     min   under    over audio L audio R  peak L  peak R           state                                         version  uptime  offset

sub_radio1             43200   42768    1511       0      -6      -6       0       0    Unstable (2)                 ODR-AudioEnc v3.1.0-3-g137828b      381       0

sub_radio2             15552   13824     197      33      -7      -7       0       0    Unstable (2)                                                       0       0

sub_radio3             15552   13392     812     132     -90     -90     -90     -90    Unstable (2)                                                       0       0

sub_radio4             15552   13824     197      33      -7      -6      -1      -1    Unstable (2)                                                       0       0

sub_radio5             15552   13824     196      33      -7      -7      -1      -2    Unstable (2)                                                       0       0

sub_radio6             15552   13824     196      33      -8      -6      -4      -4    Unstable (2)                                                       0       0

sub_radio7             15552   13824     196      33      -8      -8      -4      -4    Unstable (2)                                                       0       0

sub_radio8             15552   13392     196      33      -8      -7      -3      -3    Unstable (2)                                                       0       0

sub_radio9             15552   13824     196      33     -11     -12      -5      -5    Unstable (2)                                                       0       0


Currently, only sub_radio1 is configured to use the timestamped option, the other channels are configured to use pre-buffering and have a normal latency that, however, keeps increasing continuously.

Greetings and have a great weekend :-)

Cheers
Felix



Felix Hofmann

unread,
Feb 19, 2022, 1:17:53 PM2/19/22
to mmbtools
Just a quick update: After running for approximately 70 minutes, the statistics now look like this:

Statistics from ODR-DabMux v4.0.0

id                       max     min   under    over audio L audio R  peak L  peak R           state                                         version  uptime  offset

sub_radio1             42768   42768    1511       0      -5      -5       0       0   Streaming (4)                 ODR-AudioEnc v3.1.0-3-g137828b     4171       0

sub_radio2             15552   13392     197      33      -9      -9       0       0   Streaming (4)                                                       0       0

sub_radio3             15552       0   12075    1947     -90     -90     -90     -90    Unstable (2)                                                       0       0

sub_radio4             15552   13824     197      33      -3      -4      -1      -1   Streaming (4)                                                       0       0

sub_radio5             15552   13824     196      33      -5      -5      -1      -2   Streaming (4)                                                       0       0

sub_radio6             15552   13392     196      33      -8      -7      -3      -4   Streaming (4)                                                       0       0

sub_radio7             15552   13392     196      33      -7      -7      -4      -4   Streaming (4)                                                       0       0

sub_radio8             15552   13392     196      33     -11     -13      -3      -2   Streaming (4)                                                       0       0

sub_radio9             15552   13392     196      33     -13     -12      -4      -5   Streaming (4)                                                       0       0


Jan de Vries

unread,
Feb 19, 2022, 5:46:42 PM2/19/22
to mmbtools
>> the other channels are configured to use pre-buffering and have a normal latency that, however, keeps increasing continuously.  <<
That is what i am seeing too, sometime's up to 2 minutes!
Restarting the encoder to get on-time again.

Op zaterdag 19 februari 2022 om 19:17:53 UTC+1 schreef djho...@gmail.com:

Matthias Brändli

unread,
Feb 25, 2022, 3:26:54 PM2/25/22
to crc-mm...@googlegroups.com
Hi Felix,

I haven't been able to reproduce the issue yet. There is improvement
potential in the statistics script because sometimes it's not clear if
the input is not receiving any data or if the data has late timestamps.
But at the moment the cause of this isn't clear to me...


Could you test without the EasyDABv2?

e.g. install dablin, compile mmbtools-aux/zmqtest/zmq-sub and run

zmq-sub 127.0.0.1 18081 | dablin_gtk


mpb

On 19/02/2022 18:14, Felix Hofmann wrote:
> Dear Matthias,
>
> thank you and apologies for the late reply, I was extremely busy so I
> didn't get around to do further testing until today. I just updated all
> ODR components to the "next" branch, the issue currently still persists.
>
> chronyc-tracking returns:
>
> Reference ID    : 81468425 (stratum2-4.NTP.TechFak.NET)
> Stratum         : 3
> Ref time (UTC)  : Sat Feb 19 17:10:14 2022
> System time     : 0.000284899 seconds slow of NTP time
> Last offset     : -0.000129834 seconds
> RMS offset      : 0.000627513 seconds
> Frequency       : 11.001 ppm slow
> Residual freq   : +0.006 ppm
> Skew            : 0.917 ppm
> Root delay      : 0.021726156 seconds
> Root dispersion : 0.002185798 seconds
> Update interval : 129.7 seconds
> Leap status     : Normal
>
> The Python Script returns:
>
> Statistics from ODR-DabMux v4.0.0
>
>
> id                       max     min   under    over audio L audio R
> peak L  peak R           state
> version  uptime  offset
>
> sub_radio1 43200 427681511 0-6-6 0 0Unstable (2) ODR-AudioEnc
> v3.1.0-3-g137828b381 0
>
> sub_radio2 15552 13824 19733-7-7 0 0Unstable (2) 0 0
>
> sub_radio3 15552 13392 812 132 -90 -90 -90 -90Unstable (2) 0 0
>
> sub_radio4 15552 13824 19733-7-6-1-1Unstable (2) 0 0
>
> sub_radio5 15552 13824 19633-7-7-1-2Unstable (2) 0 0
>
> sub_radio6 15552 13824 19633-8-6-4-4Unstable (2) 0 0
>
> sub_radio7 15552 13824 19633-8-8-4-4Unstable (2) 0 0
>
> sub_radio8 15552 13392 19633-8-7-3-3Unstable (2) 0 0
>
> sub_radio9 15552 13824 19633 -11 -12-5-5Unstable (2) 0 0
> <https://groups.google.com/d/msgid/crc-mmbtools/9e4fd9b1-17cc-4255-8124-58b9db78e546n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/crc-mmbtools/9e4fd9b1-17cc-4255-8124-58b9db78e546n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "mmbtools" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crc-mmbtools...@googlegroups.com
> <mailto:crc-mmbtools...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crc-mmbtools/3d3a8335-c7c7-4535-9502-2b474bb51700n%40googlegroups.com
> <https://groups.google.com/d/msgid/crc-mmbtools/3d3a8335-c7c7-4535-9502-2b474bb51700n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Felix Hofmann

unread,
Jun 6, 2022, 11:13:09 AM6/6/22
to mmbtools
Dear Matthias,

apologies for not getting back to you for so long, I lost track of the project for a while and have not tried to diagnose the issue any further. I will try to verify if the phenomenon is related to the EasyDABv2.

Cheers
Felix

Reply all
Reply to author
Forward
0 new messages