ADC Clipping Indicators

[Steve Haynal]

Hi Group,

There was some discussion about decreasing the HL2 LNA dramatically to avoid clipping at some active QTHs on a Quisk list. I looked into this and found a bug in the firmware. For background, here is how the 4 LEDs on the HL2 currently work:

1. The left most LED indicates connection/no connection to software. On is connected.

2. The next LED indicates TX active. On is transmit active.

3. The next LED is the "good ADC range" indicator. It flashes on when the ADC is using 75% or more of the available codes.

4. The right most LED is the clip indicator. It flashes when the ADC reaches the most positive or negative code, not necessarily a clip.

The current firmware sets the ADC overload flag when either 3 or 4 above is true. This is a bug and leads to software indicating ADC overload even when only 75% of the ADC codes are used.

I will fix this in a future firmware release, but for now I'd recommend setting the LNA as described below when you are in an active environment.

1. Ignore the software ADC overload indicator.

2. Adjust the LNA gain so that the "good ADC range" LED is probably mostly on, but the clip indicator is mostly off. The clip indicator can flash or flicker occasionally without much degradation.

Even if the firmware is updated to only set the ADC overload flag for condition 4, Quisk keeps the indicator on for at least 1 second so that you can see it. Other software is similar. This means that if you average just one corrupted sample (true clip) every 76.8 million samples, software will keep the indicator permanently on. This is less than  0.000002% corruption which I don't think will matter. I need to think about this more, but I'd like the firmware to only set the ADC overload flag when enough corrupted samples have occurred over a fixed period of time that noticeable degradation may occur. Maybe there are some ideas from list members? Since we only have 12-bits with the HL2, we need to make the most use of every bit and tolerate the rare clip.

73,

Steve

kf7o

[Alan Hopper]

Steve,

my preference would be to extend the protocol with a peak adc value reached since last sent and a count of adc overload events, this could just be left to wrap round and the software could work out the % overload as it can estimate the number of samples per update. 

73 Alan M0NNB

[James Ahlstrom]

Hello Steve and Alan,

I don't think we need a drastic change here. I can change Quisk to indicate the relative number of packets with the clip bit set (somehow). I am worried about changing the protocol. I am getting requests to add further Hermes protocol items to Quisk. I just added the Alex high pass and low pass filter bytes. I don't want to collide with an existing Hermes protocol item. Anyway, the best indicator of excessive clip is the graph, which widens and degrades with clipping.

Jim

N2ADR

[in3otd]

Hello,
we briefly discussed ADC clipping also a couple of years ago and I also tried to compute how many samples per seconds can be clipped before seeing an increase in the ADC noise floor - the math is the same as for the Noise Power Ratio test - but still I'm not convinced the numbers there make sense.
I any case I think the best general strategy for setting the LNA gain is to increase it until the noise coming from the antenna is higher than the internal RX noise or until the ADC starts clipping, whichever comes first.
I think the FW should just report the ADC overload as usual and any filtering/processing of this information should be done in the SW.

73 de Claudio, IN3OTD / DK1CG

[Alan Hopper]

Hi all,

the problem with the current solution, even without the bug, is that it gives a very poor measure of the actual clipping level, it will give the same information to the software for 1 in 60,000 (ish) samples clipping or every sample clipping so both software and human agc will tend to over react.  Some versions of SparkSDR have an agc mode called 'min' which tried to do what Claudio describes, I had to play statistical guessing games with the adc data to do this as the current data from the radio does not really give enough information.  In side by side tests with two HLs with one running max gain before clipping and one running just enough gain to be above internal noise I did not see any real difference in number of wspr/jt spots but I keep meaning to re run these tests as I have a feeling there were differences at the start and end of the day that were swamped by the number of spots during the better part of the day.  I have not put the min mode in later versions of Spark simply because it relies on a noise floor calibration that I came to realize was not valid for all HL1 firmware, now the HL2 has settled down I'll look at putting it back in.

73 Alan M0NNB 

[Steve Haynal]

Hi All,

Thanks for the feedback!

Alan, I would like to extend the protocol to provide more clip information to software, provided it doesn't interfere or break core openhpsdr protocol 1 compatibility. We've discussed similar extensions in the past and the ball is in my court. I just need to motivate myself and find the time!

Jim, thanks for the willingness to improve Quisk, however I would like to have a (partial) solution independent of software as there are users who use PowerSDR and linHPSDR, and these packages may be harder to make changes to. I'm thinking of setting the ADC overload flag only when X number of clips occur within a time period Y. I'll make some experiments to determine a conservative X and Y yet improve over what exists currently. For this basic usage, there are no changes to software or the openhpsdr protocol but it should provide more accurate ADC overload indication.

Claudio, I agree with your noise floor method of setting the LNA gain. But Sid and Graeme have been reducing the LNA to -2 dB gain during the most active periods and have been relying on the ADC clips to make this setting, yet the ADC code use percentage reported in the bandscope is typically in the 25 to 55% range for them. I think this LNA setting is too low and they haven't even reached the point of antenna noise being higher than internal noise. If we relax the ADC clip indication, maybe they can reach this point or at least come closer. Maybe they can report on what happens when they set the LNA as you suggest.

Also, there is some trade off between better incidental dither with higher LNA gain versus more noise from the higher LNA setting. With my Z-match filter, I often see the ADC code usage in the less than 5% range, which indicates very little dither. What are your thoughts about this?

73,

Steve

kf7o

[Alan Hopper]

Steve,

I wonder if sending back an overflow count might be an good thing to do first as it makes it much easier to then test and set good X and Y for the basic hpsdr overload indicator.  I very much want to get back to playing with the firmware and hope to once the x-platform spark is ready.

73 Alan M0NNB

[in3otd]

Happy new year to all!

Regarding the clipping indicator, to know how many samples per second we can afford to clip we have to assume, at least implicitly, a certain input signal distribution, to know the additional noise power due to the error coming from the clipped samples. Equivalently, we could compute how many dB we can increase the gain past the point where we have 1 clipped sample per second (i.e. clipping indicator in Quisk always on) to have that number of clipped samples per second.
I mean, there is no need to change when the ADC overload reporting, we can get the same result just by telling the users "it's safe to increase the LNA gain by x dB past the point where the ADC overload starts to occur".

Out of curiosity I've added a screen in Quisk showing the bandscope samples histogram, not sure which information we can get from this, besides the ADC non-linearities already noticed by Alan some time ago, but it's interesting to see the samples distribution changing rapidly over time, sometimes to shapes which are clearly far from gaussian.



I wonder if the FPGA could be fast enough to compute the samples histogram in real time and send that back instead of the bandscope samples, in that way we won't miss any overload event.

73 de Claudio, DK1CG / IN3OTD

[Steve Haynal]

Hi Claudio,

The histogram looks very interesting. Can you share your modified version of Quisk?

At first thought, the histogram might be difficult to do in the hardware, mainly due to memory requirements. 

73,

Steve

kf7o

[in3otd]

Hello,
enclosed is the patch for adding the histogram screen to Quisk 4.1.30. It's a bit rough but seems to work; the histogram screen is activated pressing twice the Bscope button. The Ys slider controls the Y scale and currently there is no control to change the X scale (so you cannot see the whole ADC range...).
You can choose between a log or a lin graph by changing the sources, look for the line
self.ytype = "log" # change histogram type
in the __init__() method of the HistogramScreen() class.
The histogram is averaged, the time constant for averaging is changed in quisk.c, line
double alpha = 0.05;
at the beginning of get_histogram().
I've also added a fitted gaussian distribution graph, to make it easier to compare with the actual signal distribution.

73 de Claudio, DK1CG / IN3OTD

[Steve Haynal]

Hi Claudio,

Thanks for the patch. It is quite interesting to play around with the histogram view. I can definitely see the non linearity where the peak moves away from zero as the LNA gain is increased. For the HL2 I was using, it moves off in the negative direction. Your HL2 appears to move more slightly in the positive direction.

I like the idea of characterizing the input signal distribution. I'm thinking of simple ways to make use of this. Suppose we pick some value, maybe +/-2000 to be a cut off point. X is a count of samples with absolute value greater than the cutoff, near clipping. Y is a count of samples with absolute value less than the cutoff. We consider the ratio X/Y and if it becomes too large we know clipping is degrading and we set the ADC overload flag. We can also send X as an extension to the protocol as Alan requested.

73,

Steve

kf7o

[Steve Haynal]

This is pretty much equivalent to just counting the number of max values seen in a certain time period and setting the ADC overload at a limit...

73,

Steve

kf7o