HL2 Remote TX audio issue

[Will VA3JWO]

I recently (weeks ago) purchased a new HL2 from MF. My TXRX work is remotely across a Wireguard VPN.

Facts:

RX audio works fine and I can widen it right up to maximum and pull 5mbps down the pipe with zero issues. TX mic audio is a different story on ALL voice modes..

Softwares tried are SDRC, Thetis, SparkSDR, and Quisk with all the same TX audio issues and across different OS's and different microphones. I have spent time digging through forums and made changes to the TX Buffer settings (max 75ms based on previous posts) along with PTT hang time both on the computer software side and the Python software side of the HL2 with no luck. I have ran a debug script successfully but cannot interpret what its telling me.

What I'm seeing is lots of TX audio micro stuttering which looks like distortion on the TX signal at the HL2 when listening directly from another receiver, and also looks like wideband garbage on the SDR software screen administering the HL2 when audio is applied. NO PTT relay chattering is occurring. Audio source is both mic audio and two tone from the software directly. I would've thought the TX buffer could pad this out but it doesn't seem to be doing anything at all. Changing from 10ms to 75ms in SparkSDR yields no difference.

Topology:

The HL2 is positioned at Point A with a 100/50mbps LTE connection and nominally 25-35ms latency, LTE tower site within 1.3mile (2km) of the location. VPN is Wireguard with the server at a VPS cloud.

Remote PCs to administer and connect to the HL2 are considered "Point B" everything on the Wireguard VPN and my latency across the VPN from point A to point B is nominally 60-75ms with 150/50mbps throughput availability.

MTU size is fixed at 1300 for VPN network traffic.

Any suggestions for my TX audio? My Icom 7300 RSB-1A does not have this issue across the same termpoints on the network, I understand the protocols are different.

I can send recordings and debug scripts if needed.

Software version is 73, firmware Patch 3, Board Id 5 as per SparkSDR.

[Steve Haynal]

Hi Will,

Thanks for the detailed post!

First, does your HL2 work as expected with a local ethernet LAN connection? I'd like to distinguish if there is a problem with your HL2 or the LTE link. 

Second, you mentioned running the Python debug script. Did that generate a .vcd file? If so, can you share it?

Third, SparkSDR reports EP6/EP4 drops in the top right of the screen. Do you see any drops? Are there more drops when you are transmitting?

The HL2 needs just over 3Mbps of low jitter connection for TX to operate properly. Since receive is working, we know that some host2radio packets are getting through. (There is a watch dog timer which will disconnect if not enough host packets are seen.) Some people experience problems over wifi which probably has better QOS than your LTE link. Some have success by setting up a small computer local to the HL2, so that there is enough buffering and TCP can be used.

73,

Steve

kf7o

[ron.ni...@gmail.com]

Remote Tx with an HL2 only works if the maximum latency jitter or latency variation is less than the size of the transmit buffer (which defaults to 10 mS).  Otherwise, the FPGA's small transmit buffer will empty, and the HL2 transmit state will time out, which will cause the relays to click or chatter, and the Tx signal to break up.  

So it's very possible that the problem is that your network is occasionally losing, delaying, or holding on to UDP packets for more than 10 mS longer (or longer than your configure delay) than the average network latency duration, before forwarding them to the HL2.  

Test this hypothesis by doing a long series (say a minutes worth) of pings to the HL2, and noting the min and max ping latency over that entire test time.  If the difference is more than 10 mS (or 75 mS ?), then a bad network is the problem; and that will prevent the HL2 from being used for clean transmit signal using that network. 

An alternative is to co-locate a computer near, or directly connected to the HL2, and using a remote desktop to operate that computer.

73, Ron, n6ywu

On Monday, February 19, 2024 at 3:12:06 PM UTC-8 Will VA3JWO wrote:

[Will VA3JWO]

Hi team, 

Thanks for your replies so far and thank you for the support. 

Steve, Ron, over the VPN I can confirm that there are plenty of EP6 drop counters (150 / sec) racking up steadily in both TX and RX over LTE, not really seeing "more" when I transmit, the counters are racking up at a consistent rate both TX and RX and the bursts of EP6 counters sync perfectly when each stutter and voice drop most times with 48k sampling selected on SparkSDR. I trialed with the TX buffer setting adjustments but thinking its more of an issue with packet loss on my end. My round trip time (latency) from HL2 to PC remotely is typically 65-110ms variable factoring worse case jitter.

I ran a flood ping test from a host on my HL2 network to a host on my remote network I would be typically trying to access the HL2 from across the VPN on LTE, and out of 9570 packets sent at 1200 bytes in size I received 9565 (0.0522% loss) with an RTT of 85ms avg which seems odd as it isn't losing a tun of data at such a large size but the min and max was 76ms and 104ms during (28ms jitter). My MTU across my VPN is capped at 1200 I also tested adjusting MTU sizes on my VPN with no change. The audio issue appears as what looks like wideband garbage on the Panadapter on TX. Also attached in a picture here. You can see when it comes and goes. This is only on remote. 

Locally on a PC in the same room on a GigE connection, the counters sit at 0 and thus TX RX audio is perfectly clean and excellent quality.

So looking at a possible other idea, I have a client/server audio program that has worked well for me before and seemed to be pretty resilient to an LTE connection. I understand based on previous threads that the AK4951 companion board in the "HL2+" is a tough topic here, but is there known to be any other way to pipe audio directly into this unit bypassing the delicate transport protocol so a more versatile one can be used? Audio delay/ latency is not an issue for me, I just need 90-95% TX audio to make it through undamaged for communications grade. If the AK9451 only adds latency and not loss, this is a solution I can accept unless there is another one out there. I have done this before when I needed an audio transport method during the HamRadioDeluxe days and the audio client/server program I used was perfect. I can outfit a computer with soundcard output directly into the HL2 some way.

Could another method be to utilize HL2_TCP? does this support AM and SSB mode within it's audio transport? 

I would like to use Remote Desktop as a very last resort only as it really isn't ideal for my use case. Graphical sampling like Teamviewer in my findings is cut down and unpleasant and the overall user experience was such I would just stick to my 7300 RSB-1A which works fine. 

Steve, the debug.vcd file is attached for your review. This is approx. 10 or so seconds of remote TX and audio in AM mode and 75ms TX buffer selected in SparkSDR. Let me know if it doesn't show up.  

Will

[Steve Haynal]

Hi Will,

Thanks for the debug.vcd. I have a very busy week. I will study it this weekend end get back to you.

The audio still goes back to the host and then to the HL2 with the AK4951 board. It just adds to the problem if you then increase the traffic with another audio pipe.

73,

Steve

kf7o

[James Ahlstrom]

Hello Will,

I am currently working on software that runs on a single board computer such as a Raspberry Pi. It reads HL2 traffic from PC software on one network interface, usually WiFi, and copies it to the Ethernet port attached to the HL2. It adds buffering to the Tx path. This is meant to make the HL2 work on a WiFi connection with large latencies.

In your case, WiFi is not used, but the WiFi connection can be replaced with your VPN connection. The software should work in this way too.

The trouble is that the software is not finished. It seems to work but I had planned to release it after I get back from my next ski trip. I leave next Monday for two weeks. If you want, I can release it now for testing, but I won't be able to fix anything for another two weeks. But you may be able to gain some insight from it. And who knows, it may work.

Jim

N2ADR

[Ryan Jennings]

This would be really helpful in using VPN connection without maintaining a full desktop.  I like it!

Ryan

--
You received this message because you are subscribed to the Google Groups "Hermes-Lite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/1dbfd6a4-5b48-4dda-81c0-b43690e34a10n%40googlegroups.com.

[Will VA3JWO]

This sounds great Jim! I would love to try it. At this point I am game for trying anything as I can only use my HL2 for RX in the present state.

Thank you for the support on this along with Steve as well for the reply and insight to the AK companion board interface. Hopefully my debug file shared and information on my working conditions can be of help to the community for future developments as an example. 

Just so I'm clear, the software your developing Jim, sits static on my VPN network that both the HL2 and PC Software are talking on, and acts as an intermediary IP device for buffering all protocol data between the PC Software remotely and the HL2 itself locally? I'm imagining setting up the Rpi with your software to talk to the IP of the HL2 directly on the local side of the network (my HL2 is on a static internal IP) and the PC Software data reaches remotely from the VPN the Rpi and gets processed and redirected by your software to the HL2, do I have this correct? So setting up for example SDRC or Thetis, one would set the software up on the remote PC to contact the HL2 via the IP address of the Rpi with your software instead of the HL2 directly. 

I would love to test it!

Will   

[ron.ni...@gmail.com]

Hi Jim,

You beat me to it.  Nice work!  I recently posted an emulated HL2 server as a Gist on my GitHub page.  I was planning on using that code in conjunction with my HL2 client connection code in hl2_tcp (also on GitHub), as end-caps on a buffering HL2/Metis/UDP relay.  The thing that delayed me from finishing this was trying to figure out what error concealment algorithm(s) I should try in the case of missing or out-of-sequence packets.  Otherwise even a small rate of missing UDP packets will eventually cause the FPGA FIFO to underflow.

As to the question about using hl2_tcp for this: hl2_tcp uses a slightly modified  rtl_tcp protocol. This is an asymmetric protocol designed for SDR receivers.  Thus, it that only sends small control messages upstream; so is not suitable for SSB or AM, which requires streaming IQ data in both directions.  I have been experimenting with modifying it for transmitting CW and slow data rate FSK for use over even lower bandwidth networks than would support transcoded HL2 UDP.

73, Ron, n6ywu

[James Ahlstrom]

Hello Will,

As an alternative solution you could try the AC2YD extension to Quisk. A version of Quisk runs locally on the PC and another runs remotely next to the HL2. The bandwidth requirements are minimal.

Jim

N2ADR

[Will VA3JWO]

Hi Jim,

I spun up a Debian 12 VM instance on my server today with Wireguard running for the 2nd interface and have both Ethernet and VPN interfaces presently up and available to use now. I am ready to test HL2WifiBuffer but the Git page for it doesn't seem to have any files yet for me to pull and make the software. https://github.com/jimahlstrom/hl2_wifi_buffer also doesn't seem to exist as per the readme file. I would love to test it in your absence on your ski trip!

I will also have a look at the AC2YD Extension to Quisk this weekend.

Kindest Regards,

Will 

[James Ahlstrom]

I pushed my software to my github site . Please post results to the new group "HL2 Wifi Buffer".

Jim

N2ADR

[Will VA3JWO]

Hi Guys, just an update after using the Wifi buffer software I posted my findings in Jim's thread. Unfortunately, I am still encountering the same audio distortion issues as before. There is no Relay chatter at all at all anymore with Jim's Buffer, and as you can hear from my attached recording I'm sending a 1K tone remotely from Thetis through the Wifi buffer which is local to the HL2, but the audio is still so badly distorted it sounds like a fax modem. This is during remote TX only, everything wired to the ISP both ends. Locally on the network its fine and no distortion when tried. Adjusting Jim's Wifi buffer to max in the txt file yields the same results as without, just removes all the relay chattering. I feel the buffer is working fine as it removes totally the relay chatter on TX but you can hear the 1K tone reproduction from remote computer to HL2 TX is still distorted and looks almost modulated on the SDRC panadapter I also provided screenshot of.

Based on the recording attached, are all of these micro cuts in the tone dropped audio packets? Each time a micro cut happens this might be counted as an EP6 counter in SparkSDR? I'm thinking there is something happening in the datastream of this commercial LTE network that is just incompatible with the transport method and encountering loss that I am unaware of. Good for web browser and app use for mobile phones but full of datastream holes.

I was unsuccessful in getting Quisk to run as a server client instance. It works and controls the HL2 but with audio issues and no TX, etc. 

I will try the RDP method to a local PC and see if I can pipe good quality audio to a local computer near the HL2.

[Steve Haynal]

Hi Will,

I did take a look at the .vcd debug file you sent, but I see no problems. You had TX buffer latency set to 51ms which is good. The total buffer is 85ms so you want TX buffer latency set to about half. The "fill" of the TX buffer ranges between 20ms and 60ms. There are no signs of drop or overflow. Is this for your remote failing setup or your good local setup? I am a bit suspicious as you do remote relay clicks. This .vcd debug doesn't have full precision but is based on sampling the buffer fill.

If Jim's program eliminates relay clicks but you are still seeing garbled transmit, then I'm starting to agree with you that there is dropped data on the LTE link. The HL2 uses udp which has no guarantee that all packets must go through or even arrive in order. It would be helpful if you could provide a wireshark trace at the remote site. That would tell us if packets are dropped or reordered.

I didn't look closely at Jim's code, but I suspect he has a added extra UDP buffering in a small software service. This won't fix dropped or reordered UDP packets but will help if the problem is high jitter. If dropped/reordered UDP packets is the problem with your LTE link, then you'll have to switch to TCP as done in:

https://github.com/hotpaw2/hl2_tcp

Note that you can't fix this problem by using local audio with the AK4951 companion card. Take a look at the protocol, in particular the USB protocol document (originally USB but same packets now go over ethernet). 
 https://github.com/softerhardware/Hermes-Lite2/wiki/Protocol

You will see there is LR mic data and which is sent from the HL2 to the host as well as LR audio data which is sent from the host to the HL2. This is because there is no modulation/demodulation done in the HL2. This is a software-defined radio so that is done on the host. The host must always send IQ TX data to the HL2 and receive IQ RX data from the HL2. Using local audio just adds dependencies on this audio data in both directions too, so makes things twice as bad for you. In your case, you'd actually be better off reducing the size of the packets and just eliminating the mic and audio data. In a future more elegant solution, that is what I'd like to do.

73,

Steve

kf7o

[Will VA3JWO]

Hi Steve,

This was from the remote failing setup during a transmission I think with 1K tone on AM observing the audio issue.

Please find attached my screenshot sample of (I think) lost UDP packets in Wireshark during another remote AM transmission and 1K tone generated running through Jim's software. The packets typically come in with frame ID increments of 1 but I'm seeing some jumping incrementally as high as 4 or 5 or more intermittently as per my second screenshot. Am I reading this right? I know UDP has no sequence but is jumping frame IDs potentially indicative of loss? 172.16.30.4 is the remote PC running Thetis and 192.168.50.13 is Wireguard client running Wifi_HL2 talking to the HL2.

I do confirm that Jim's software fixed the relay clicks perfectly. I can actually re-create the issue and drop the buffer to 0 and see the relay chatter away intermittently on the HL2 and the TX reset on the data dump, then set it to 100ms and not get a single click or buffer refill for minutes. 

I will try out the HL2_TCP but was unsure if it supported SSB/AM transmit but will give it a go.

Your last paragraph makes perfect sense. Thanks for clarifying on this and citing the links.

[Reid Campbell]

Hi Steve,

Your last comment about removing the mic and audio data is something I have been interested in doing for a while. Would it be a big job to update the Gateware with an option to not send mic data and not receive audio data? A switch for this could be added to the protocol.

I think the transmit band with could be halved, so reducing requirements of a WAN link. If this was available, I could modify Thetis to not send the audio or expect the mic.

Cheers

Reid
Gi8TME/Mi0BOT 

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/bc68b35f-7581-4a6b-8b78-0708e06f8f61n%40googlegroups.com.

[ron.ni...@gmail.com]

Not sending the mic and audio data would involve creating a completely new UDP protocol, incompatible with the current Metis wrapped Protocol 1, which may impact compatibility with multiple OpenHPSDR hardware devices (Radioberry, Red Pitaya, SquareSDR, HL2, older Apache Anans, and etc.) as well as a half dozen or more SDR applications.  IIRC, a decision has already been made not to evolve the current Hermes Lite 2 project to support the newer Protocol 2.  So this proposed change might represent a complete new OpenHPSDR  protocol and project.  

BTW, hl2_tcp already can reduce HL2 network bandwidth by around a factor of 4 by optionally truncating receive samples to 8 bits, and by not sending IQ transmit data samples at all.

73, Ron, n6ywu

[Reid Campbell]

No, I don't think so. We already have a variable protocol as you can specify the number of receivers. The more receivers, the more information packed in the packets. I use this feature to reduce the bandwidth for WAN links by reducing the number of receivers to two.

What I'm suggesting is to over load some of the bits in the protocol, like that done for Band Voltages or PS Sync. This would inform the Gateware not to expect audio data or send mic data. If you don't set the bit, then the protocol stays the same, so not affecting any legacy software.

Cheers

Reid
Gi8TME/Mi0BOT

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/987003fe-fff6-44e7-b95d-fe235aba1bf4n%40googlegroups.com.

[ron.ni...@gmail.com]

That is not a correct description of Metis or Protocol 1. The size and format of the UDP Metis packets and the size of the 2 Protocol 1 (USB) packets inside the Metis UDP wrapper stays the exactly same no matter how many receiver slices. So the SDR network stack stays identical, irrespective of the number of receiver slices in use.  Only the interleaving of the IQ data varies.  The UDP packet rate does vary with both the sample rate and the number of receiver slices, but not the packet format.  The band volts feature did not change the protocol but used optional fields already documented in the protocol.  It's not just your preferred software and legacy software that supports Protocol 1 as used by the HL2, but lots of other current software and SDR hardware, that one should be careful not to break.  73, Ron, n6ywu

[Reid Campbell]

As the mic and audio data are interleaved like the receivers, should it not be possible to not interleave them. The SDR software just has to signal that reduction with the Gateway, just like it does for the number of receivers. If the Gateway doesn't see the remove mic and audio signal bit, it just treats it like the normal Protocol 1 for the HL2, which in itself is already different from the default Protocol 1.

The format and packet size would stay the same but the packets, for transmit, would only contain 16 bit I&Q transmitter data. That would reduce the effect bandwidth by half, as the 16bit L&R mic data wouldn't be sent.

Cheers

Reid
Gi8TME/Mi0BOT  

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/38ab9c78-dd31-4274-9548-e4649ef83b34n%40googlegroups.com.

[Reid Campbell]

Just realised that the mic would be mono so only one channel of 16 bits, reducing bandwidth by a 3rd.

Cheers

Reid
Gi8TME/Mi0BOT

[Steve Haynal]

Hi Reid and Group,

For PC->HL2 data, I propose we create a "skip audio data" mode by changing a bit in the: 

<0xEFFE> UDP Preamble

<0x01> Upper 4 bits of byte command which immediately follows the 0xEFFE

<End Point> Upper 4 bits of end point which is second byte after 0xEFFE

or

<Sync bytes> Bits in one of the 3 sync bytes

See the Metis documentation for more details about these 4 bytes.
https://github.com/TAPR/OpenHPSDR-Firmware/blob/master/Protocol%201/Documentation/Metis-%20How%20it%20works_V1.33.pdf

See the Protocol documentation for details about the 3 sync bytes.

https://github.com/TAPR/OpenHPSDR-Firmware/tree/master/Protocol%201/Documentation

This would be an extension to the protocol and would be backwards compatible. For example, suppose we chose End Point bit 6 to represent data which has no audio. Then instead of sending 0x02 here as is normally sent by software for standard data, software would send 0x22. This would tell the HL2 that there is no LR audio data included. The protocol would remain the same except the 16-bit audio samples are skipped (no bytes in udp packet) by the HL2. Skipping is easier than rearranging or trying to pack anymore data into the existing udp packet. This would apply per packet sent and could actually be dynamically changed from one packet to the next if there is a need.

Likewise, for TX, I propose we assign a control register in the base memory map which contains a setting specifying what data the HL2 should return. HL2->PC data uses the same 0xEFFE preamble, End Point and Sync Bytes. When sending data, the HL2 would make an identical change to the specified bit depending on the control register. For example, again assuming End Point bit 6 signals no audio data, then software writes 0x20 to the control register, the HL2 starts sending 0x26 instead of 0x06 and skips sending any audio data in the sent packet. 

For those who are worried this breaks the protocol and backwards compatibility, note that you just continue sending the same bits you already are and everything will continue to work as before.

I would like to reserve at least 4 bits for this type of functionality as I can imagine other variations we may want in the future.

Please let me know what you think and which bits you would prefer to use. This is simple to implement and I can probably do it this weekend.

73,

Steve

kf7o

[Steve Haynal]

Thinking about this a little more, I'd prefer to use bits in the first byte of the 0xEFFE preamble. This is early in the protocol. For possible future others uses of these bits, like sending raw ADC samples, we could have less overheads.

73,

Steve

kf7o

[Will VA3JWO]

Hi Steve,

I am unfortunately not terribly experienced with the eningeering internals of the data, but am certainly willing to test with my worst case scenario setup! 

Thanks again for the hard work. 

Will

You received this message because you are subscribed to a topic in the Google Groups "Hermes-Lite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hermes-lite/-qjywnphDYE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/e584ea53-2eda-4105-bff3-38152cb8ea51n%40googlegroups.com.

[Reid Campbell]

Hi Steve,

This sounds fine so far. If we don't packet the audio or mic data into the packet and not add more I&Q, that should make the packets smaller? I was wondering if that would have an affect on dropped packets as there would be more transported. For most efficient transport, you normally go for as large packets as possible.

I also noticed that there was packet stuffing for different receivers which would have to be thought about as well.

Cheers 

Reid
Gi8TME/Mi0BOT

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/70443e0c-5bdf-496a-bc43-b5982fd3f4afn%40googlegroups.com.

[ron.ni...@gmail.com]

In my experience, the cause of broken-up transmit is rarely a network bandwidth issue, but more likely either network latency, latency variation or jitter, and/or loss of some UDP packets from the SDR host to the HL2. Normally the transmit network bandwidth requirement is several times less than receive network bandwidth, unless receiving only one slice at the lowest sample rate. When a symmetric network path provides enough bandwidth for HL2 receive, there' usually far more than enough bandwidth for HL2 transmit packets.  Thus, solving broken transmit issues involves, not finding a faster network or reducing transmit packet bandwidth, but either (1) switching to a network (or network stack) with higher reliability and lower transit jitter, or if that's not possible, then (2) using a buffering server to absorb latency jitter, and/or either a more reliable transport (such as TCP), or some form of error concealment.  So while optionally decreasing UDP packet size might provide some benefit in some situations, it's unlikely to solve the broken-up transmit issue. 73, Ron, n6ywu

[ron.ni...@gmail.com]

Thinking in reverse... There is one other possibility to increase transmit reliability, and that is to not reduce bandwidth, but increase it by sending redundant transmit data, perhaps multiple UDP packets with the same sequence number and Tx data, and have the HL2 discard duplicates.  If 4 copies of the Tx data were sent, then 1 to 3 UDP packets in a row could be lost by the network without any break-up of data needed for the HL2 to continuously transmit.  That duplication would only increase Tx network bandwidth to the same as receiving one slice at 192 ksps, or two slices or 96 ksps.  73, Ron, n6ywu

[Reid Campbell]

That's quite an interesting thought but I would be inclined to throw out he unnecessary data and fill it with redundant packets.

Cheers

Reid
Gi8TME/Mi0BOT

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/c43e5446-b689-4174-b881-b85e813b9771n%40googlegroups.com.

[Steve Haynal]

Hi Ron and Reid,

Adding redundancy or retries is an interesting idea which would also benefit from reducing the host-to-hl2 data. Also, I suspect a good case can be made that other QoS factors (jitter, lost UDP packets, out of order UDP packets) may be improved if the link is less utilized.

There is an experimental gateware release here:

https://github.com/softerhardware/Hermes-Lite2/tree/experimental_protocol_changes/gateware/bitfiles/experimental/20240303_74p100

to experiment with these two ideas. Note you must be on the experimental_protocol_changes git branch. I've only made changes in the host-to-hl2 direction for now. I am repurposing the lower 2 bits of the first 0xEF in the 0xEFFE to be:

audio_included_next = eth_data[0];

relax_sequencing_next = eth_data[1];

For the default 0xEF, both audio_included and relax_sequencing will be set to 1'b1 or enabled.

If audio_included is set to 0, then the HL2 will not expect any L1,L0,R1 or R0 audio data. This data or the bytes taken by this data should not exist in the packets sent. Everything else should remain the same.

By default, the HL2 doesn't pay any attention to the sequence number as described in the Metis How it works document. This is considered relaxed sequencing and is enabled by default. If you set relax_sequencing to 0, then the HL2 will ignore duplicates or older packets. Only the least significant 4 bits of the sequence number are monitored. Here is the code.

if (((eth_data[3:0] - seqno) > 4'h0) & ((eth_data[3:0] - seqno) < 4'h8)) begin

seqno_next = eth_data[3:0];

enable_fifo_push_next = 1'b1;

end else begin

enable_fifo_push_next = relax_sequencing;

end

The effect is that an arriving sequence number must be 1 to 7 greater than the last seen sequence number for the HL2 to accept it. This is modulo 16 since only 4 bits are used.

With relax_sequencing disabled, software can send duplicate packets (same sequence number) and the HL2 will ignore them if already seen. In fact, software can send a series of duplicate packets of up to 8 incrementing sequence numbers and still have them ignored. This does nothing for reordered packets and it is possible for the HL2 to skip ahead.

The code changes are in the experimental_protocol_changes github branch.

https://github.com/softerhardware/Hermes-Lite2/tree/experimental_protocol_changes

73,

Steve

kf7o

[Clifford Heath]

Excellent changes, Steve.

It would be good to keep a count of missed packets (sequence from 2 to 7 ahead of last) and out of order packets (sequence from -1 to -8 behind last) to help knowing when there are network issues. Similarly for buffer underflow, in case the HL2 is too far away to hear the relays click.

We'd need to find a place in the protocol to report such things, of course. Is there any existing reporting like this?

Clifford Heath

[ron.ni...@gmail.com]

Hi Steve,

Just to clarify, does zeroing the experimental "audio included" bit mean that the Metis UDP packets will be shorter than 1032 bytes, and the internal Protocol 1 segments will no longer be offset by 512 bytes?  Thus, that any code that expects or checks for that format needs to be optionally disabled if this bit is cleared?

Does the optional "audio included" bit imply that the rate of UDP packets will be the same in both cases?

Also, what does the default configuration of the current release gateware do if duplicate sequence numbered packets are sent?  e.g. Are duplicate sequence number UDP packets ignored by the current release gateware, and thus can be sent to HL2's running current gateware?  Or is this a experimental change in behavior to allow duplicate packets to be sent to increase Tx reliability?

73, Ron, n6ywu

[James Ahlstrom]

I think that removing mic and audio data is a useless modification that complicates the protocol and results in no benefits. For my reasoning please reread the comments by Ron, N6YWU who has explained the issues better than I could.

If we are trying to fix this issue let's measure the lost and misordered packets first and then find a solution. I can modify my buffer software to do this when I get back from skiing.

Meanwhile you can use my buffer software and watch the buffer fill level on its web page. It the utilization falls the issue is lost packets.

Jim

N2ADR

[ron.ni...@gmail.com]

Instrumenting and measuring the Tx stutter problem a good idea.  My suspicion is that, since leaving out the mic and audio data does not change the UDP packet rate, the rate of lost UDP packets, and thus skipped sequence numbers that are seen in some problem networks, will not be significantly improved.  Router and switch equipment and network software stacks usually handle complete IP packets, not raw byte streams.  Thus reducing the raw byte rate but not the UDP packet rate might not change the issues seen much at all.  

There is even a small possibility that smaller UDP packets might even increase latency jitter issues, due to more packets fitting (and thus waiting too long) within some router's buffers.  Sending more UDP packets though by sending duplicate sequence Tx data, might actually push data out of some router's buffers sooner, thus reducing latency jitter, and fixing two problems.

73, Ron, n6ywu

[Reid Campbell]

Hi Steve,

I tried the experimental build but had a few problems. I flashed it and everything was normal. I then set the second header to 0xed which should have turned off relaxed sequencing. When I tried to transmit with a normal transmission, I got a bad spectrum with lots of sidebands.

Looking at your changes, I think you were trying to catch out of sequence and duplicate packets but to be honest, subtraction of limited bit integers was doing my head in. I decided to simplify it and made it such that any duplicates would be discarded.

Now when I set the header to 0xed and transmitted, the spectrum was OK. I then tried to send duplicated packets and I was back to a wide spectrum with lots of side bands.      

Now, I could have been doing some wrong but it appear to not be disposing of the duplicated packets. 

Cheers

Reid
Gi8TME/Mi0BOT

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/255aeebf-f8a5-4a01-b054-3f924edf2d28n%40googlegroups.com.

[Steve Haynal]

Hi Group,

I don't think it is useless to eliminate audio samples in the host2hl2 direction. You eliminate almost 50% of the data. This should help with low bandwidth links, especially if you also want to send redundant packets. Less data should also make it easier to achieve QOS for links with higher bandwidth too, where we've seen packet jitter can cause problems.

If people don't like the changes in the packet structure, what about just replacing the audio samples with additional IQ samples? The packet structure would remain the same but contain twice as many IQ samples.

Reid, feel free to check your changes into the experimental fork. I'd like to see them. I wasn't sure if duplicates would always appear in order and hence the attempt at a small window. Did you experiment with eliminating audio samples?

I agree that we need more data and analysis of the real problem some see. Is it dropped UDP packets? Is it out of order UDP packets? Is it UDP packet jitter? I never see the problem so people with LTE and WIFI links should collect and share data. It sounds like Jim's program can help with that.

73,

Steve

kf7o

[Steve Haynal]

Hi Ron,

Sorry, forgot to answer these questions. The current experimental change will make the UDP packets shorter but the same rate. If we replace audio data with IQ data, the packet lengths will be the same but the send rate would be half of what it currently is. A sweet spot might be to send a redundant packet so the send rate stay the same as it is currently.

The default gateware ignores the sequence number. Any (including redundant) and all packets received are sent. The experimental change added functionality to the gateware to ignore redudant packets by requiring the sequence numbers to be monitonically increasing to at least 3 bits.

This is all experimental stuff and in its own experimental fork.

73,

Steve

kf7o

[Will VA3JWO]

Hey Steve would it help if I shared a data dump of some PTT TX intervals across the network while experiencing the packet loss using Jim's software? It seems to display data during the connectivity session.

I can also try and capture another debug file too that's longer in capture time during PTT. Its heard as constant lost audio so wouldn't be that intermittent to capture and share for informative purposes. Let me know. 

Will

[Steve Haynal]

Hi Will,

If you are using Jim's software already, I think it would help if that software provides some debug output regarding missing or reordered udp packets as well as min/max/jitter times between packets it sees. Then you could share those numbers. Maybe the software can already do that. Maybe Jim would be kind enough to add that. I haven't looked at it yet.

73,

Steve

kf7o

[Reid Campbell]

Hi Steve,

I made the simple mod of

//      if (((eth_data[3:0] - seqno) > 4'h0) & ((eth_data[3:0] - seqno) < 4'h8)) begin
      if (eth_data[3:0] != seqno) begin

which meant that a header mod'ed to 0xed didn't cause wide Tx spectrum. Once I started to send duplicate packets, the spectrum went bad.

I haven't tried the removal of the audio data as I have to be sure I get the right loop in the software which inserts the audio samples. I was going to use the protocol 1 simulator to validate the new packet layout. I though the duplicate packets would have been the low hanging fruit but it didn't work out that way.

Cheers 

Reid
Gi8TME/Mi0BOT

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/173812b2-9bda-4f6f-bff9-6f26c40cfe26n%40googlegroups.com.

[James Ahlstrom]

Hello Group,

I published a new version 1.1 of my WiFi buffer at https://github.com/jimahlstrom/HL2WifiBuffer. It isn't finished but I wanted to release it because it adds a counter for missing or out-of-sequence Tx packets. If Will could run the buffer it may give some insight into what is happening on the VPN. Both the web page and the terminal output would be useful. The software can correct out-of-sequence UDP Tx packets.

Jim

N2ADR

[Reid Campbell]

Hi Steve,

I had a quick look at this again. I'm assuming that "enable_fifo_push_next" will allow the packet through if '1' and drop it if '0'. As a test I forced "enable_fifo_push_next" to '0' and though there would be no output, there was. Maybe I'm not understanding it fully.

Cheers 

Reid
Gi8TME/Mi0BOT

[Steve Haynal]

Hi Reid,

The value of enable_fifo_push_next set in state SEQNO0 should then determine if data is pushed in states PUSHL1-PUSHQ0 via enable_fifo_push.

I really should test this code. I made the changes but didn't have the time to test. I also want to add a flag which repurposes the audio data as IQ data at least in the host to HL2 direction. Let me try to test this week and I will get back to you although I won't promising anything. My day job is very demanding at the moment.

73,

Steve

kf7o

[Steve Haynal]

Hi Jim,

Thanks for this enhancement. Hopefully now Will and others experiencing this problem can report exactly what the cause is.

73,

Steve

kf7o

[Will VA3JWO]

Hi Steve and all,

Agreed on the day job struggles. Im hoping also to get back on this around midweek when it quiets down some, specifically with Jim's software to run it through its paces and I will report back my findings! Thanks again.

Regards

[Reid Campbell]

Hi Steve,

No problem, I know what it can be like when dead lines have to be met.

I will be travelling in a couple of weeks and will be able to do some remote testing. I wonder if I could do a firmware update using SparkSDR remotely? That really does open up the possibility of your HL2 being hacked.

Cheers

Reid
Gi8TME/Mi0BOT

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/9bcb1371-7a89-42d0-8380-c1589eeee890n%40googlegroups.com.