WSPR SNR accuracy

[KE6HDU]

For the past month a friend was staying in my ham shack (den) so I couldn't be on the air (much).  To keep WSPRing, I decided to experiment to understand how WSPR reports SNR.  (My real goal is to see how often WSPR fails to decode or misdecode as the signal strength is linearly decreased, but this SNR analysis was a simple place to start.)  To generate a controllable noise level, I created a 2 minute noise file with Audacity and a 2 minute signal file using WSPR.  I then mixed the two tracks after attenuating the signal track by various amounts.  The resultant wav file was then fed back into WSPR and decoded.  Attached are the WSPR displays showing decreasing signal strengths.  For no noise at all, WSPR reported 12db SNR.  For equal signal and noise, it displayed 9db.  Reducing the signal from that equal point by 10db gave a 0db SNR.  A 20db reduction produced -9, 30db gave -19, 40db yielded -29.  41 and 42db reductions, though clearly visible on the waterfall, did not decode.

I know where the limit test occurs in the WSPR code and will see if the code will detect weaker signals if that threshold is changed.  It's also possible that we are out of bits since the wav files are 16bits, which only allows a 65000:1 signal ratio.  I purposely was staying at about 1/2 of the maximum, so we only had about 32000:1, while 40db is 10000:1 all by itself, so we will begin to have excessive bit quantization in the signal.

Attached are two displays... the first shows a no-noise signal, which is the sort of thing one sees when transmitting and receiving (on a different antenna) at the same time.

The second shows a 42db reduced signal at the far right.  The 40db next to it decoded but the 42 didn't (which would have corresponded to a -31db SNR on WSPR).

It's interesting that WSPR is quite linear in its SNR reports even though they seem not to be actually an SNR number, since I'd assume that equal S & N should display as 0, rather than 9.  Perhaps I'm wrong in that assumption??

Steve

[Glenn Elmore]

Steve,
Interesting test. I guess I'm confused though. I would have counted the
dynamic range of 16 bits at 96 dB - 6 dB/bit, rather than the 3 dB you
suggest. If doubling the value doubles the output voltage in the same
impedance, power is quadrupled and it's 6 dB. How am I thinking wrong
about this?

Also, parity at "equal" levels assumes that you are comparing with a 2.5
kHz noise bandwidth, I think. If one compares the ratios of bandwidths,
you get 10*log(2500/6) ~26 dB which seems about as WSPR is advertized,
guessing the effective WSPR bandwidth to be about 6 Hz, which may be an
errant guess. Also there may be an extra 3 dB for phase continuous
detection too, I'm a bit foggy on this. In any case it's probably
pushing 30 dB better than an SSB bw which seems about right.

How did you generate your noise file and how did you calculate it's
signal level in a 2500 Hz bw?

Glenn n6gn


It seems that the noise floor must be calculated based on signal level
in regions excluding WSPR signals. Is this perhaps a smart calculation?

> --
> You received this message because you are subscribed to the Google
> Groups "2 Meter WSPR" group.
> To post to this group, send email to 2-mete...@googlegroups.com.
> To unsubscribe from this group, send email to
> 2-meter-wspr...@googlegroups.com.
> Visit this group at http://groups.google.com/group/2-meter-wspr?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[KE6HDU]

Glenn, you're quite right about my bit limit error, but first let me explain exactly what I did:  I told Audacity to operate at 44.1Khz and create  0.45 maximum amplitude white noise using the Generate noise option.  Then I captured a WSPR transmit into a file and placed that in another track, also normalized to 0.45 maximum amplitude. (This amplitude choice was made so that the two tracks added together prior to scaling wouldn't clip.) 

I then applied various levels of attenuation to the WSPR track prior to selecting Mix and Render from the Tracks option.  This created the composite, which sounded good and analyzed correctly.  (At the time, I wondered if I should bandwidth limit the white noise prior to decoding, but the WSPR documentation said that there was no need to restrict the bandwidth since the transforms being performed didn't care about the extra noise.  I would like to test that theory by performing this same test, with bandpass filtration on the composite signal as it would seem that additional noise spectrum must have at least some impact on decoding efficiency.)

Notice that each additional 10db of attenuation did cause WSPR to report a 10db drop, which indicates that both the generation and decode were tracking exactly.  You are very right about my mistake in db... I was thinking about voltage rather than power.  After checking Audacity's manual, I see that they say that 0.5 maximum amplitude is 6db down, not 3db, so, yes, the dynamic range is the square of what I thought it was, so we are nowhere near running out of bits.

Do you think that this was an adequate test, or should I have approached it differently?  I considered taking the output of a 2 meter HT and mixing it directly with the generated WSPR audio, but I didn't want to deal with making an exact analog fader or trying to analyze the spectrum of the white noise coming from the receiver headphone jack.  I do want to do a statistical analysis of the decode efficiency and correctness as the SNR drops, but I don't want to do it if I've failed to consider something important.  You already pointed out one error. (Thank you!)   Are there others?

Steve, KE6HDU

[Glenn Elmore]

Steve,
I think this all makes sense. I'm not sure of the goal of the first test, except to see how well WSPR measures signal strength. One would think it could do it pretty well, since it's all related to digitizing audio and then measuring in the digital domain.  One question I still have is "What is WSPR reporting, minimum S/N, maximum or some kind of average over the 2 minute interval?"
Since you are interested in watching decode characteristics as a function of SNR with constant SNR, I would think this is doesn't matter, but I've always wondered.

WRT the "extra 9 dB", maybe that makes sense too.  If your Audacity tool was generating truly white noise with a 44.1 kHz clock, I would think the result would be flat noise power density out to half the sampling frequency - per Nyquist.  That would say that your noise power bandwidth, rather than being 2.5 kHz as WSPR is "specified", is actually 22.05 kHz  and the power in a 2.5 kHz bandwidth would be 10 log (2.5/22.05) = -9.45 dB down from the total. This would put the "equal" level situation you describe SNR at +9.45 dB relative to the WSPR "specification", and match your measurement.

Glenn n6gn




I agree that trying to generate an analog signal and then digitizing is not going to give you quite the precision of the approach you took.

[Bob W7PUA]

Hi Steve and Glenn,

First your generator project sounds great and potentially useful to both do measurements and to gain insight.

But, I have a concern.  I went and looked at Audacity, a program i like a lot.  But, I was concerned by the way that they described the noise generator---it says that it is white noise and it has a voltage (of say 0.45).   In the way of a caution, noise time series need a description of the probability distribution, such as Gaussian.  You can have other distributions that are "white."  If indeed the Audacity generator is the same as a well behaved 2-meter SSB receiver, and Gaussian, then it cannot be described by a peak "Voltage." but rather something like the "standard deviation," often referred to as sigma.  Once you know that, the value that would correspond to "power" would be sigma squared. (I am putting these items in quotes because we don't know, but also don't care about the impedance level---the answer is the same for any impedance because you only want the S/N ratio).

If that Voltage is one sigma (I doubt it) the values in the file would be overloading often.

I looked around the Internet, but couldn't ind any details on the Audacity noise generator.

Steve, maybe you know what the distribution is?  If not, it is possible to take the noise file, analyze it for distribution and power, nd answer the question.  Maybe easier is to find somebody that is familiar with the function and knows the answer.  The source code is available,as well, and might tell us.

Once you have the noise "power" then you also need the signal power.  From your description, you have used the WSPR to generate the multi-tone sine wave.  The signal "power" is 0.5 times the center to peak value of the sine wave squared.  That value divided by sigma squared is the S/N. 

It is a bit tricky to not lose 3 or 6 dB in all this, but when you get all done, your generator sounds like a neat and useful tool.  Good luck with that project.

Bob  W7PUA

[KE6HDU]

Bob, thanks for your ideas and concerns, especially concerning the noise generator.  

A noise generator can be constructed by making calls to a random number generator algorithm, which in turn can be limited to a maximum value by only using a specific number of bits in the random number.  All other values will be less than that maximum, so we don't have the situation of huge excursions.  You are right that this isn't truly random, but only random within a range.  I haven't looked at the Audacity source yet, but I suspect that's what it does. (My transceiver is also random only within a range.  No matter how much signal it receives, the output power to the speaker isn't going to exceed a few watts.)   On the Audacity spectrum analysis display, whatever this random generator is doing, the signal was uniformly spread from a few dozen hz up to 22khz, so I don't see any issues with what is being created, though the "random" numbers could fail more advanced random number tests.

To address Glenn's comment, I applied a mathematical filter to the noise and constrained the frequency range to less than 3.2khz (-40db by 4khz) and less than 1.6khz (-40db by 2khz).   The signal display bumped around a lot more as we didn't have so many high frequency spikes to fill in the spaces.  This was pretty much in accordance with what Glenn had suspected.  Amazingly enough, however, WSPR was unimpressed.  All 3 bandwidths, 44khz, 3.2khz, and 1.6khz, produced an SNR of -9 db when I applied a -20db attenuation to the signal part of a 1:1 signal to noise mixture.  Looking at the code in mept162, and sync162, it's clear that there are a lot of normalization and empirical constants being used that make the math not so straightforward.  I'm not explaining why this is happening, but only reporting on it.  (At one point in the sync code we see that the SNR is computed

as this:  snrx=db(max(psavg(ipk),0.0001)) - 26.5 !Empirical        The "Empirical" comment is in the code.)

As one final test for tonight, I fed a signal of what would have been reported as -30db (-41db attenuation) to WSPR at several different signal levels to see if this made a difference in decoding.  It didn't, neither a -4db level or 19db input level would decode, despite the signal being clearly visible against the noise.  1db more signal does consistently decode, which implies that we have hit a threshold somewhere.  There are several level tests in this code so I'll add some print statements to see which one has decided not to allow the decode to be done.

Have a great evening and thanks again for your comments,

Steve, KE6HDU

[KE6HDU]

Bob, the white noise generator in Audacity is in its EffectNoise::MakeNoise routine.  It simply calls the rand() function and then scales the random number to the desired maximum, so there will be no points that are outside the specified amplitude.  Rand() is generally a fast algorithm but not necessarily a great one.  It's quality depends on how the particular operating system's function was written.  Nevertheless, I think that the numbers we're getting should be adequate for what we're doing, given the spectral distribution that the analyzer showed.  By the way, Audacity's pink and brown noise generators use the same rand() function, just with different weighting.

Steve

[Bob W7PUA]

Steve -  If they use a single call to C's rand() they are getting a sample of a uniform distribution.  It really is quite different from the Gaussian distribution that comes from the receiver. The uniform distribution will provide a number between 0 and 32767 (usually), with all values equally probable. Not to say that this is not a useful as a rough approximation.  But it is rough!

Interestingly, by a property called the Central Limit Theorem, you can produce a really good approximation to the Gaussian distribution by adding a number of uniform samples together, and dividing by the square root of the quantity of numbers added.  Quantities like 5 or 10 are getting close to Gaussian for many purposes.  If you are interested, I could get you more info.  It is also easy to figure the noise power to get S/N. That would be an interesting function for Audacity.

There are various ways to prevent overload problems with the Gaussian, all centered around the idea that samples at, say, 4 sigmas are very infrequent.

Again, we are talking about the probability of taking any value, not the frequency spectrum.  They are independent in many respects.

Thanks for taking the effort to find that.  And thanks for putting up with this  ;-)

73, Bob  W7PUA