EDI transport issues
81 views
Skip to first unread message
Nick Piggott
Jul 7, 2020, 9:22:40 AM7/7/20
to mmbtools
Hi
MUX:
debian jessie
odr-dabmux 3.1.0
MOD 1:
debian jessie
odr-dabmod 2.4.1
uhd-lea-m8f-v3.14.1.0
MOD 2:
debian jessie
odr-dabmod 2.4.1
uhd-lea-m8f-003_009_00
I've configured odr-dabmux to provide ZMQ and EDI. The SFN has a 6 second time offset set in it at each MOD box. All the boxes are using ntp, and have a jitter of below 100ms.
Using ZMQ, the connection between the MUX and MOD boxes seems reliable, and I see virtually no errors.
The EDI configuration has PST enabled, FEC set for 5 blocks and interleaving over 240ms. There are three EDI destinations as unicast UDP - MOD1, MOD2 and local-loop (on three different orgination ports)
Using EDI, the stream is very unstable. I see a huge number of UHD warnings and errors about underruns and late packets:
OutputUHD status (usrp time: 1594127126.318571): 0 underruns and 1584 late packets since last status.
I also see packet sequence errors:
pseq 20442 timed out after RS
WARN EDI AF Packet sequence error, 20443
ETI FCT discontinuity, expected 192 received 193
and timestamp warnings
2020-07-07 14:15:33,946: WARN OutputSDR: Timestamp in the past at FCT=154 offset: -96.149871 (1594127733.943871) frame 154, tx_second 1594127637, pps 0.794000
2020-07-07 14:15:33,972: WARN OutputSDR: Timestamp in the past at FCT=158 offset: -96.079097 (1594127733.969097) frame 158, tx_second 1594127637, pps 0.890000
2020-07-07 14:15:33,996: WARN OutputSDR: Timestamp in the past at FCT=162 offset: -96.007393 (1594127733.993393) frame 162, tx_second 1594127637, pps 0.986000
2020-07-07 14:15:34,021: WARN OutputSDR: Timestamp in the past at FCT=166 offset: -95.936666 (1594127734.018666) frame 166, tx_second 1594127638, pps 0.082000
2020-07-07 14:15:34,047: WARN OutputSDR: Timestamp in the past at FCT=170 offset: -95.866243 (1594127734.044243) frame 170, tx_second 1594127638, pps 0.178000
2020-07-07 14:15:34,071: WARN OutputSDR: Timestamp in the past at FCT=174 offset: -95.794892 (1594127734.068892) frame 174, tx_second 1594127638, pps 0.274000
The combination of underruns, late packets, and packet sequence errors means the transmission either glitches out or causes the modulator to completely reboot.
I've looked at the incoming UDP traffic using nc -ul -p XXXX, and it's there and seems consistent. There's no obvious reason why UDP should be so dramatically less reliable than TCP.
Has anyone experience similar dramatic differences between using TCP (ZMQ) and UDP (EDI)?
Thanks
Nick
Nick Piggott
Jul 9, 2020, 6:45:21 AM7/9/20
to mmbtools
To follow up:
With PFT disabled, I'm seeing the multiplexer generate UDP packets of 3 196 bytes length every 24 ms
With PFT enabled (even if FEC and interleaving are not configured), the multiplexer generates UDP packets of 148 bytes every 20ns (!)
So I think something is wrong in the PFT code - there's no reason for the UDP packets to be so short and thus need to be sent so rapidly. I'll open an issue on odr-dabmux to track
Nick
Matthias Brändli
Jul 9, 2020, 7:24:19 AM7/9/20
to crc-mm...@googlegroups.com
ODR-DabMux sends the EDI fragments in quick succession, not in
equidistant time intervals. That might explain the interval, but you
should see bursts.
About the packets being short:
This is done in lib/edioutput/PFT.cpp Line 144 and on. I now wonder if
the scenario "Fragment only, no protection" could be handled better.
There's a maximum chunk length of 207 for reasons I don't remember.
mpb
> --
> You received this message because you are subscribed to the Google
> Groups "mmbtools" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crc-mmbtools...@googlegroups.com
> <mailto:crc-mmbtools...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crc-mmbtools/482e0ffc-8138-4118-b1d4-17f87362da26n%40googlegroups.com
> <https://groups.google.com/d/msgid/crc-mmbtools/482e0ffc-8138-4118-b1d4-17f87362da26n%40googlegroups.com?utm_medium=email&utm_source=footer>.
equidistant time intervals. That might explain the interval, but you
should see bursts.
About the packets being short:
This is done in lib/edioutput/PFT.cpp Line 144 and on. I now wonder if
the scenario "Fragment only, no protection" could be handled better.
There's a maximum chunk length of 207 for reasons I don't remember.
mpb
> You received this message because you are subscribed to the Google
> Groups "mmbtools" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crc-mmbtools...@googlegroups.com
> <mailto:crc-mmbtools...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crc-mmbtools/482e0ffc-8138-4118-b1d4-17f87362da26n%40googlegroups.com
> <https://groups.google.com/d/msgid/crc-mmbtools/482e0ffc-8138-4118-b1d4-17f87362da26n%40googlegroups.com?utm_medium=email&utm_source=footer>.
Nick Piggott
Jul 9, 2020, 7:35:25 AM7/9/20
to mmbtools
Yes, that's a better description of what I'm seeing. I've put more details in an issue on odr-dabmux.
To unsubscribe from this group and stop receiving emails from it, send an email to crc-mmbtools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crc-mmbtools/86437455-a4c1-a9cc-280e-10d4ce0f54f4%40mpb.li.
Nick
pswiatki
Jul 9, 2020, 8:23:41 AM7/9/20
to mmbtools
But wait, Nick.... How is that possible that ~20 times shorter packets get sent roughly one million times more frequently?
Somethings seems to be seriously wrong here....
Nick Piggott
Jul 9, 2020, 9:41:58 AM7/9/20
to mmbtools
Hello,
I've put a fuller description of what's happening on the issue ticket on Github.
What's happening is that it's sending 15 "smaller" UDP packets every 24 ms, instead of 1 large one. So you see a burst of packets gapped out at 15ns or so, which I assume is as fast as the computer can send them.
The problem is that the modulators are complaining about late and missing packets, which suggests the appliances in the network in between aren't passing them quickly (or in some cases dropping them).
I'll try some other changes (in default TTL, and potentially tunneling the packets in a VPN, so they appear to be crossing fewer devices) but I think I'm still going to be fighting against the number of packets per second (625 per second).
Nick
--
You received this message because you are subscribed to the Google Groups "mmbtools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crc-mmbtools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crc-mmbtools/b6b9a15e-3ef3-4e6e-9cce-6341c72eb71eo%40googlegroups.com.
Nick
pswiatki
Jul 9, 2020, 11:10:18 AM7/9/20
to mmbtools
Oh... I see.... packet bursts. Now it makes perfect sense. Still, why bursts (?).
Perhaps data could be aggregated into bigger blocks before being sent in UDP packets.
To unsubscribe from this group and stop receiving emails from it, send an email to crc-mm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crc-mmbtools/b6b9a15e-3ef3-4e6e-9cce-6341c72eb71eo%40googlegroups.com.
--Nick
Reply all
Reply to author
Forward
0 new messages