Acceptable Buffer Depths in FPGA?

[Steve Haynal]

Hi Group,

I've been revisiting the buffering in the HL2 firmware. When we send data to the PC, we send at regular intervals and the burden is more on the software on the host to buffer and be able to save all incoming data. When we receive data from the PC, we must have enough buffering to hide packet jitter on a home network and hide packet timing variation introduced by the software. If we don't, there will be stuttering during transmit. In the openHPSR version 1 firmware I started with, the 2 TX buffers were run almost always at empty which I believe led to some infrequent stutters during FT8 transmit when packets from the PC were late. I now have a single TX buffer set for both late and early packets. I'd like to hear what others think is a good amount of packet jitter tolerance, especially since my home network may not be representative.

In an ideal world, packets arrive from the PC every 2.625ms. This is for the 126 IQ samples in every UDP packet (Two 512  byte frames, each frame with 63 IQ samples). 

The new HL2 TX buffer will tolerate variation of +/-21ms. This works well on my network. I could double this, but would spend more RAM blocks on the FPGA. The blocks are available, but I'm trying to save them for some future use. According to this, jitter should be below 30ms, which the total buffer size of 42ms covers. For those writing software, what quality of service and jitter is the software introducing?

I also added a watchdog timer which stops sending packets and clears the run bit if not enough packets are received. Taking into account number of receivers and the receiver bandwidth, there should always be an integer relation n:m of packets set to packets received. In my first attempt, if 8 expected packets in sequence were not received, then the watchdog would fire and stop the radio. I was surprised to see that this condition happens with Quisk almost every time one switches from TX to RX. This is a jitter introduced by Quisk of over 18ms since it occurs at the TX to RX switch. Is that expected? Can software do better? I have increased the watchdog count to 256 or 672 ms, which I've never seen reached.

73,

Steve

KF7O

[Alan Hopper]

Hi Steve,

It would be useful to have some sort of feedback from the radio of buffer level, then we could monitor it and get a measure of jitter over a range of networks and setups. I believe  protocol 2 has this.  The 1.0 releases of Spark use windows 'Pro Audio' thread priority for the realtime dsp path so the jitter should be considerably lower than +-21ms , I've not set the priorities in the linux version yet so it could be much worse.  It is possible that the the wspr,jt9 and psk modes can affect the jitter as they currently expose the realtime threads to the risk of being delayed by the .net garbage collector, this does not apply to FT8 and I mean to back port the improvements in FT8 to the other modes.

I've often wondered how to best treat the tx buffer, should a few extra packets be sent at startup? What happens when packets are lost? 

The watchdog should save me a lot of walking to the shed (which I have to do whenever I stop my code in the debugger and don't stop the radio properly).  

73 Alan M0NNB

[James Ahlstrom]

Hello Steve,

I am in Dayton, Ohio for the ARRL hamvention. This has been on my bucket list for years. Too bad it is raining. So I am working from memory here.

I measured jitter years ago when I wrote my firmware. It was usually about 10 msec, but could occasionaly be 50 msec or more. The cause of the jitter was the PC, not the network. This may be different now with multicore processors. A single core is subject to conflicting demands from other processes. But I think some ARM processors are single core, and ARMs are important.

I do not understand why there would be two Tx buffers. My firmware has a single large Tx buffer, but I don't remember the size. Say it is 100 msec. To transmit, we assert PTT and Quisk starts sending samples, but the firmware will not start sending them until the FIFO buffer is half full. Then Tx starts. If the buffer overflows, samples are thrown away. If it underflows, Tx stops and will not resume until the buffer is half full again. So the expected latency is half the buffer size. There is no need for a watchdog. If PTT drops, Tx stops and the buffer is cleared.

With my firmware, no samples were sent unless Tx was active. The HL2 protocol sends Tx data at all times, so that is different. But the above algorithm is simple, and it is easy to change the latency by just changing the buffer size without changing the logic.

I am afraid you will see a wide variation in jitter at different sites. WiFi networks can have high jitter. I will look into the 18 msec pause in Quisk when switching from Tx to Rx when I get back.

Jim
N2ADR

[Alan Hopper]

Hi Steve,

another thing that would be nice to feed back to the pc is some count of missed packets based upon the sequence number. Combined with the buffer level software could then minimize latency for the local environment.  I've never really tackled cw in my code but have put some thought into it, the old buffers were certainly part of the problem of making it work with low latency over the network, I had thought of bypassing the buffers for cw, just some food for thought.

73 Alan M0NNB

[Graeme Jury]

Hello Steve, Alan, Jim et al,

Not quite on the level that you guys are discussing but related. For me the holy grail is to be able to operate my SDR anywhere in range of my wifi network. At present the system largely goes but every minute or so it hunts and pulses for a few seconds until the buffers sort themselves out and then runs normally. Using Quisk this happens both on my HL1.4 and HiQSDR and is better on pulse than Alsa. Even with Play latency set to 500 and Graph refresh rate set to 3 it still happens although less frequently. The same effect occurs with piHPSDR only it is even more pronounced. My throughput is normal for an 802.11-n system but I can't tell you what the worst case packet delay would be until I learn more about it. I don't have a windows computer so have not tried it on other radio software.

The ghpsdr3-alex project worked well as a remotely operated system with a computer dedicated as a server and any simple computer as a client. It was possible to wander around the house with my netbook and enjoy a QSO with no drop outs. Of course there was a downside which was mainly due to arranging the buffers for very long packet arrival (mSecs) to allow operation over the Internet and CW operation was not possible at break-In or contest speeds. This is not what I am after though, just a normal phone or CW QSO from my laptop in whatever room I am in.

I have spoken to many people and all are agreed that to be able to set up the SDR radio in a fixed location with power supplies, linear and antenna tuner and operate it from anywhere in wifi range with a laptop is just about as good as it can get. In my own case I have built a little unit using an Arduino and an ESP1 wifi unit which will switch on the SDR, Amp and tuner from a small utility written in QT5 and from there Quisk can take over. Quisk is so cleverly done that you can control your station by adding buttons to it for things like starting an auto-tune process so remote operation is great except that you need to be tethered by an ethernet cable to avoid dropouts and pulsing. A fix to use wifi would put the system in a league of it own.

73, Graeme zl2apv

[Graeme Jury]

Just a bit more info and some numbers which may be helpful to those determining buffer sizes. I set up my laptop on wifi as a tcp server and then as a udp server and measured some data throughput between it and my desktop. It explains why wifi does not work well compared to an ethernet wired connection and I guess is typical of a home lan. Others may wish to duplicate the experiment and see how their figures compare as it is quick and easy to do.

gvj@gvj-M92p ~ $ iperf -c 192.168.1.54
------------------------------------------------------------
Client connecting to 192.168.1.54, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.50 port 34868 connected with 192.168.1.54 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.2 sec  48.2 MBytes  39.6 Mbits/sec
gvj@gvj-M92p ~ $ iperf -c 192.168.1.54 -u -b 1000m
------------------------------------------------------------
Client connecting to 192.168.1.54, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.50 port 51663 connected with 192.168.1.54 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   114 MBytes  95.8 Mbits/sec
[  3] Sent 81470 datagrams
[  3] Server Report:
[  3]  0.0-10.1 sec   107 MBytes  88.9 Mbits/sec   0.343 ms 5404/81469 (6.6%)
[  3]  0.0-10.1 sec  1 datagrams received out-of-order
gvj@gvj-M92p ~ $

I had two wishes Jim, one was to go to Dayton and the other was to go to Oshkosh.

73, Graeme zl2apv

[Steve Haynal]

Hi Alan,

I agree that it would be nice to provide feedback to the host regarding queue status. As you know, there are the 3 sync bytes 0x7f sent with every frame, twice in a UDP. We could repurpose those to be various queue status. I wouldn't want to break compatibility with other software, so we could assign one of the bits in the run/stop bandscope on/off byte to turn this status feedback on or off. When off, the default, the sync bytes would still be 0x7f. The easiest for the firmware would to just report the number of entries currently in the TX queue. This would be an 11 or 12 bit number. What do you think? Any other ideas?

The new TX queue tries to remain at half full. If the queue underflows, the transmit DSP will be sent zeros until the queue is half full again and then normal TX data will begin again. If the queue overflows, samples from the host are not put into the queue and dropped until the queue has drained to half full again. I don't think a few extra packets at startup are required now, but that may help with the older HL2 and HL1 firmware.

73,

Steve

KF7O

[Steve Haynal]

Hi Graeme,

Thanks for the feedback. How old is your wifi router? It may be that newer wifi routers will provide better performance. Since there is currently unused memory resources in the FPGA, I will double the TX queue in the next HL2 firmware release. This should help mask wifi variability. But when we need some memory resources for something else in the future, I may steal these back.

The HL2 firmware is now quite different from the HL1 firmware.

73,

Steve

KF7O

[Steve Haynal]

Hi Jim,

Have fun in Dayton. I too want to attend the Hamvention some day.

The latest firmware does run a single TX queue as you describe. It was the older firmware that inherited two TX queues from openHPDSR, I think because of migrating from initial USB support. 

The watchdog timer is not related to queue underflow/overflow management. There have been times when software crashes or is disconnected without ever telling the HL to stop sending packets. In the past, the HL would continue to send packets to nowhere. TX could even remain on. The watchdog timer requires that we hear from the host every so often, and if not, automatically go to the stop state to prevent these runaway conditions.

73,

Steve

KF7O

[Alan Hopper]

Hi Steve,

I used the sync bits in the past to send back the actual number of receivers being used and can confirm that it breaks powerSDR compatibility so being able to turn it on is worthwhile (although it is very easily fixed in the powerSDR source).  If we add this, can we add the number of receivers and my 'add receivers without stopping' patch at the same time? As you are actively managing the buffer in the fpga we probably only need the top few bits of the count and maybe a bit to indicate an underflow.  Your new way of managing the buffer sounds like a great improvement.

73 Alan M0NNB 

[James Ahlstrom]

Hello Graeme,

Maybe you can try a newer router and adapter combination. Newer units can operate at both 2 and 5 GHz, and the 5 GHz band may be better. That would mean a new router or access point and probably a new WiFi adapter at your laptop. Do you get complaints about dropouts on transmit, or is this a receive problem only?

Jim

N2ADR

[Steve Haynal]

Hi Alan,

Remind me of you 'add receivers without stopping' patch again? The changes in the latest firmware should just allow this now. You should be able to change the number of receivers on the fly and the HL2 will accommodate. There may be one 512 byte frame were the new alignment takes place so there are some crackles due to the misaligned audio, but the frame will never exceed 512 bytes and will be properly aligned for the selected number of receivers in the next 512 byte frame. Please let me know if this isn't working.

Do you want to know the maximum number of receivers the firmware supports, the current number of active receivers, or both? It is a bit hard to precisely synchronize sending the active receiver count with a frame that is properly formatted for that number of receivers because the stuffing of the data into a queue occurs in a different clock domain. I don't want to expand the queue width to always include that information. My thinking is that it is acceptable for a few receive crackles as the queue drains when changing the number of receivers. Software could also apply a period of silence.

By the way, the queue for LR data sent to the HL2 has been separated from the IQ queue. Before they were one queue but we rarely used the LR data. Splitting them into two allows us to save wasted space. You use this data to program the distortion tables, and the firmware can be built with a LR data queue as an option to support your distortion table programming.

73,

Steve

KF7O

[Alan Hopper]

Steve,

Good to hear receivers can be added without stopping.

I want to know the number of receivers encoded in each packet so that on change I correctly decode the packets, because of the buffering there can be many packets worth of delay from sending the command, so it is currently a guessing game which can result in a block of wrongly decoded packets.  In the ideal world it would be nice to have sample rate, gain and frequency as well.

73 Alan M0NNB

[Steve Haynal]

Hi Alan,

I think I can tell you the number of receivers but synchronized maybe one or two 512 byte frames late. Assume you have 2 receivers and everything is running fine. You switch to 3 receivers but the software continues to operate assuming 2 receivers because it hasn't been notified. At some later point, the firmware will switch to 3 receivers, maybe right in the middle of a 512 byte frame. It will not exceed the 512 bytes when doing this. The software will still receive garbage numbers assuming 2 receivers for one or two 512 byte frames, but these can still be decoded as software just resulting in noise. Starting the next frame, or maybe next next frame, the firmware will start identifying that 3 receivers are running and software should be able to switch with no glitches for the 3 receivers. Is this acceptable to you? It becomes very hard to not avoid one or two 512 byte frames at the wrong rate. The penalty for this in low in my mind as it is just a few clicks on the receive audio.

For the repurposed sync bits as additional optional status, I have:

4 bits number of active receivers

4 bits max possible receivers?

8 bits TX queue depth

1 bit TX queue overflow flag

1 bit TX queue underflow flag

2 bits current receive bandwidth

Anything else? We have 4 bits left.

I'm currently building bit files for a beta release. These changes will be in a future release.

73,

Steve

KF7O

[Alan Hopper]

Hi Steve,

that sounds good to me, we don't need the max possible receivers as that is in the discovery packet.  If we can fit lna gain in that would be nice.  6 bits would probably be enough for queue depth for anything I can think of doing with it.

73 M0NNB

[Steve Haynal]

Hi Alan,

Okay. I will update the protocol wiki page with a plan.

Regarding the discovery packet, we are also sending the string "HERMESLT" one byte at a time. I think this was for CW skimmer. Do you or anyone else remember the details of this? I'd like to remove this or reduce it to the minimum requirement for CW skimmer. I think maybe N1GP added this?

73,

Steve

KF7O

On Sunday, May 20, 2018 at 12:47:09 PM UTC-7, Alan Hopper wrote:

Hi Steve,

[Alan Hopper]

Steve,

I think it was N1GP, it is used here https://github.com/n1gp/HermesIntf/blob/master/HermesIntf/Hermes.cpp .  I don't use it in my code.

73 Alan M0NNB 

[Steve Haynal]

Hi Alan,

Okay. Thanks. I've communicated with N1GP and K3IT off list and I think we can drop sending that string.

73,

Steve

KF7O

[Graeme Jury]

Hello Steve and Jim,

I guess I have not kept up with wifi technology and see that my 802.11n system is woefully short of the possible current throughput. I will be shifting QTH in a few months and will redo the wifi then. Yes my laptop will need an eternal wifi adapter and I think that this will solve the problems. I am not too sure about transmit dropouts as I don't QSO with it only listen apart from Whisper where I do Tx but some dropouts would probably not show in the overall picture.

Yes I see HL1 and HL2 growing apart but HL1 is still a fine little radio and just perfect for general band monitoring and Whisper and always will be. I am looking forward to getting an HL2 though.

73, Graeme zl2apv

[James Ahlstrom]

Hello Steve and Alan,

On Sunday, May 20, 2018 at 1:25:35 AM UTC-4, Steve Haynal wrote:

I agree that it would be nice to provide feedback to the host regarding queue status. As you know, there are the 3 sync bytes 0x7f sent with every frame, twice in a UDP. We could repurpose

Changing the sync bytes will break PowerSDR and also Quisk. And we only get three bytes. We are going to want more bytes than three.

Another possibility is to leave the standard 1024 port and protocol unchanged, and open an additional UDP port, maybe 1025. This port can carry all the extra information we want. The firmware already uses UDP ports, and branching on the port number should be easy. My firmware uses three UDP ports, and it was no problem.

Jim

N2ADR

[Alan Hopper]

Hi Jim and Steve,

for the extra info I'd really like (number of receivers, sample rate, gain and frequency) they really need to be in the same packet as the iq samples otherwise it is still a guessing game.  For the buffer level it would be fine in another packet.  I am not keen on using multiple port numbers and would prefer just a different endpoint id in the hpsdr header if we were to go down this route.  I believe Steve's suggestion is to add a bit in the start command to tell the radio to repurpose the  sync bits so as not to break old software.  At some point it would be nice to escape from the current protocol which we are starting to run out of tweaking options,  my feeling is this is best left a little way down the road when there are a significant number of HL2 about so the motivation is there for other software developers. Another alternative is to append data to the end of current packet if the extra start bit is set, I don't know the cost of doing this in the fpga but it might be more flexible.

73 Alan M0NNB

[Dieter Kedrowitsch]

Just to add my $0.02 -- and Steve and the rest of the developers are obviosuly entitled to steer this project any way they wish, but...

I for one would be disappointed to give up PowerSDR...and believe it would be a real shame if it ever came down to that.  In fact I'd go so far as to say breaking PowerSDR support would likely drive me away from the HL platform altogether.  And I'd be surprised to hear if I were alone in this sentiment.  

One of the fantastic things about HL is how it "plays ball" not only as a great all-around SDR transceiver, but because it also supports the same software as it's mature HPSDR bretheren.

Since PowerSDR is open source, has anyone considered contributing full Hermes Lite support into it? 

73 de W3DRK

[James Ahlstrom]

Hello Dieter,

I also believe that support for PowerSDR is essential.

Jim

N2ADR

[John Marvin]

I agree, and I have volunteered to do some work on PowerSDR for HL.  However, at the rate things keep changing, I might wait a bit. :) Actually that is an excuse, because I'm fairly bogged down right now, and have not even really tested my HL2 Build 6 board, other than to power it on and connect it to the LAN and verify connectivity with PowerSDR. I haven't tried attaching an antenna to it yet.  I've started ordering extra hardware (e.g. I don't like relay T/R switching, so I ordered the pin diode switcher from HobbyPCB).

The way I see it, there is a lot of good SDR software for receiving, and some good SDR software for digital transmit, but PowerSDR is still the best SDR software for SSB transmit support. But I don't like having to use the VAC "remote hack" to do audio in/out, so one thing I'd like to do is add support to PowerSDR for direct audio card support using ASIO. However, I'm also hoping to not have to maintain an HL2 PowerSDR fork, so I may limit my changes to what the current PowerSDR folks will allow in their tree.

73,

John
AC0ZG

--
You received this message because you are subscribed to the Google Groups "Hermes-Lite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Ahlstrom]

Hello Dieter,

My last post was not very useful, so let me try again.

Everyone wants to maintain compatibility with PowerSDR. When we need to add a new feature, we are trying to fit it into the existing protocol so it doesn't break PowerSDR. The latest discussion was setting a bit somewhere and then changing the sync bytes. Changing the sync bytes would break PowerSDR and also Quisk. But for PowerSDR the bit would never be set, the sync bytes would remain unchanged, and PowerSDR would still work.

My concern is that there are not many free bytes in the protocol, and in fact there are fewer than we need. Using them is becoming complicated. So maybe we should consider using a second UDP port or a different USB end point. This would help maintain compatibility with PowerSDR, not harm it.

Jim

N2ADR

[Steve Haynal]

Hi Dieter and Group,

I'd like to reassure people that there are no plans to drop or diminish PowerSDR support. A primary goal of the mainline firmware is to maintain good core compatibility with the openHPSDR protocol 1 and software that supports that protocol. Not everything the HL2 does or we want it to do fits nicely into this protocol, so we have already added and plan to add various extensions. The existing firmware has extensions to set the bias, read temperature, etc. that do not hamper PowerSDR use. The guiding philosophy is that any extensions do not hamper existing protocol 1 compatibility, and if they might, the extensions must be explicitly activated by hidden/extra protocol switches if desired by other software.

One benefit of a FPGA-based architecture is that different firmware images may be loaded on to the HL2 and used. So when Alan says "escape from the current protocol" I interpret this as at first an alternate experimental firmware. Users have the choice whether to install the alternate firmware or stick with the mainline firmware. You can even switch back and forth. This experimental aspect of the HL2 is what appeals to me the most. And who knows, maybe someday there will be a better protocol, firmwaer and software that people prefer over PowerSDR.

I am not a PowerSDR user. I use only digital modes with SparkSDR and Quisk. Consequently, I am not actively looking for ways to improve PowerSDR use with the Hermes-Lite 2.0. I'd appreciate any help from John AC0ZG and other users to make the Hermes-Lite/PowerSDR experience better and bug free. Please share any bugs, improvement suggestions and code patches you have and I will do my best to work with you to ensure good protocol 1 compatibility in the mainline firmware. For example, I currently need people to test the latest firmware with PowerSDR. 

73,

Steve

KF7O

[Steve Haynal]

Hi Jim and Alan,

I agree with Alan that this little bit of extra receiver-specific information really belongs in the same packet as the corresponding receiver data. This information will fit in less than 3 bytes, and will only be enabled if the software desires it. Otherwise it will remain 0x7f sync bytes to maintain backwards compatibility. I will try to document these ideas on the protocol wiki page soon so that we can have a clearer and more focused discussion.

Your suggestion to use a different port for extra data is also good. We may have to do that in the future when we have a need for the additional data. My one worry is the required FPGA resource requirements, and I'd have to experiment and convince myself that it can be done without significant resource usage. We are at 95% utilization and I am currently cleaning up some bloat in the I2C blocks. I'd like to have resources available for more interesting DSP tasks.

Alan's suggestion to just (optionally) increase the length of a 512 byte frame is also interesting. I think our packets are below the 1500 byte MTU threshold so we do have some room here. From the FPGA firmware point of view, it is just some additional states in an FSM. Again, I'd want to experiment with the resource usage and keep it low.

73,

Steve

KF7O

[Alan Hopper]

Hi Steve,

just thought of something else for the extra bits, the ptt bit to indicate a packet contains  data captured whist transmitting, this is should simplify preventing audio agc and display agc being disturbed on tx/rx transition.

73 Alan M0NNB