DS1054Z Spectrum Analyzer

[Steve Haynal]

Hi List,

I'm ~90% there with being able to use my DS1054Z as a spectrum analyzer. So far I am pretty happy. After processing gains, I am seeing about 85 dB of useful dynamic range. Although sampling at 1 GHz implies FFT up to 500 MHz, I think anything higher than 150 MHz will be hampered by the probes. Everything works with pyvisa and pyvisa-py. I've written my own scripts that use pyfftw. Although not realtime, I prefer the flexibility of this scripted method of performing measurements. For example, printing out the dBm of all harmonics I'll be able to do my own harmonic analysis for faster turnaround time and get V2 and V1.2 in better shape.

With the addition of a Si570-based exciter, one should be able to make VNA measurements with a DS1054Z and PC. One port would be for a sync. Another 1 or 2 ports for ADC. This may be a better approach than trying to use a Hermes-Lite, as you can make things specific for VNA, have multiple ports, and higher sampling frequency. I see the Hermes-Lite VNA as mainly useful for some antenna analysis and low frequency filters.

The figure below is of 40M with a 5W amplifier and QRP labs filters. The fundamental is at 38 dBm, the second harmonic at -23 dBm  and the third harmonic at -14 dBm. I'm surprised that the harmonics keep going and increase slightly between 80 and 100 MHz. This may be due to the QRP labs filter design, or something to improve with my physical measurement setup for higher frequencies.  There is also this unexpected emission at 62.5 MHz. All the emission are more than 50 dBc. I only plotted out to 100 MHz as I am having problems plotting larger graphs with up to 12 million FFT bins and need to do some smoothing, especially for on screen display.

73,

Steve

KF7O

 

[John Laur]

Great job, Steve.

Regarding using the scope as a rudimentary VNA keep in mind that when engaging multiple channels the sampling is interleaved as all channels share the same ADC. You therefore get only 500msps or 250msps per channel running multiple channels. This is not so bat WRT nyquist as the frontend is only good for 100-150mhz as you say, however it will affect your process gain and reduce dynamic range by 3-6dB or perhaps more.

Also, the scope offsets each channel automatically in the view to account for the sampling offset, but I do not know if you receive the same luxury if you export the raw data.

Before you become too trusting of the data you are seeing it would be very useful to compare in realtime against a spectrum analyzer the next time you have access to one. I suspect for instance some of the artifacts you are seeing may not be quite as bad as you think -- products of ADC jitter etc.

73, John K5IT

--
You received this message because you are subscribed to the Google Groups "Hermes-Lite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Haynal]

Below is the latest iteration of the budget spectrum analysis output. Instead of smoothing or reducing the bins, I clip off the lowest levels of mostly noise for a displayed range of 90 dB. I keep all the data in original format so can regenerate plots in whatever form I like. Hopefully tomorrow night I will start a series of measurements to tune the Hermes-Lite output and post to the wiki.

Found 30 peaks

  0.064208 MHz   -64.3 dB

  0.064375 MHz   -64.9 dB

  0.128375 MHz   -69.9 dB

  0.128708 MHz   -68.0 dB

  2.064000 MHz   -67.7 dB

  7.120042 MHz     0.0 dB

 14.240083 MHz   -61.3 dB

 21.360125 MHz   -52.5 dB

 28.480167 MHz   -59.0 dB

 35.600208 MHz   -58.1 dB

 42.720250 MHz   -68.0 dB

 49.840333 MHz   -63.2 dB

 56.960375 MHz   -61.4 dB

 62.500000 MHz   -51.8 dB

 64.080417 MHz   -58.6 dB

 71.200458 MHz   -60.1 dB

 78.320500 MHz   -56.6 dB

 85.440542 MHz   -53.7 dB

 92.560583 MHz   -54.0 dB

 99.680625 MHz   -56.4 dB

106.800667 MHz   -60.5 dB

113.920708 MHz   -67.3 dB

117.879958 MHz   -66.9 dB

125.000000 MHz   -57.7 dB

132.120042 MHz   -65.6 dB

133.216833 MHz   -67.9 dB

135.280833 MHz   -64.7 dB

140.336875 MHz   -59.5 dB

147.456917 MHz   -63.6 dB

154.576958 MHz   -67.5 dB

[John Laur]

Steve,

It might make the plot look nicer, but clipping samples is not a good thing to do. By doing that you are drastically and haphazardly reducing your ENOB. You should average multiple FFT results (so long as your signal is continuous through the period) in order to improve the display.

The only ways to improve your measurement are with process gain (which you get both from oversampling and increased FFT length/decreasing bin width) and improve the display are with averaging. This can't work if you are trying to capture transients or realtime measurements, but fortunately you are not trying to do that at all -- you are trying to measure responses of pure sinusoids, so you can get very good results. If you are using the full 24mpoint memory depth this will be your limiting factor depending on how much measurement accuracy (bin width) you need. But I estimate you can break 100dB SnR in this case. Let's find out---

Your ideal 8bit ADC has only 50dB of dynamic range and the particular ADC in the 1054z is the HMCAD1511 so it's really pretty dang close to perfect: https://www.hittite.com/content/documents/data_sheet/hmcad1511.pdf

If you want to run an FFT with a 250MHz  nyquist frequency with Fs=1gsps (2x oversampling) and you want a bin size of 1khz you will have to use an FFT length of 500,000 samples. So with 24mpoint memory you will be able to average 48 FFTs. pretty good. What is our expected noise floor?

Assuming ENOB of 7.9 for our ADC from the datasheet, Fs=1Gsps and FFT length (M) of 500kpt we would get the following:

SNR ADC = 6.02N + 1.8 = 49dB

FFT gain = 10 log (M/2) = 54dB

Oversampling 2x = 3dB

Noise Floor = 106dB below full scale! 48 averages should clean that up very nicely. Dont forget to experiment with the windowing function.

Here are some references for further reading:

http://www.analog.com/media/en/training-seminars/tutorials/MT-003.pdf

http://cp.literature.agilent.com/litweb/pdf/5988-4368EN.pdf

I am looking forward to seeing the python code and/or the data you have captured if you choose to release it. Looks like wonderful progress--

73, John K5IT

[Steve Haynal]

Hi John,

Thanks for your advice but I think I am on the right track. I am doing a full 24000000 point FFT. This results in an average noise floor of close to 120 dB, better than the 106 dB you arrive at. By using a large FFT, I have narrow bins of ~42 Hz per bin. This provides a more accurate value for the power within a bin than a wider bin, which is important if the goal is to best measure harmonic power. The "clipping" is to facilitate plotting as there are issues with Matplotlib and plotting 12000000 points on some machines. Since for this application I don't care if some harmonic is more than 95 dBc, I throw away anything lower than that for visualization. This drastically speeds up visualization while full results are still available programmatically. 

My plots look very different from other spectrum plots posted on this list because of these choices and some may wrongly conclude that something is broken. The harmonic peaks in my plots are very narrow because of the narrow bins and high FFT resolution. Although the average noise floor is quite low, the noise variance is high because I am not averaging. That is why you see a big band of "noise" in the first plot. As the two references you give should attest to, averaging does not improve dynamic range or change the average noise floor. It is a trade-off between bin size and noise variance. For this application, I am favoring bin size. I think the right thing to do is to lower the sampling rate to 500 MHz so that I have even narrower bins for 24M points as I can't rely on results above 250 MHz anyway. 

The nice thing about this approach is that you have complete programming freedom and can also average as you suggest. I'm a bit leery about averaging separate 24M point captures as it takes 10-20 seconds to download a capture over USB and that time difference is relatively large. Breaking up the 24M points should work. Also, I have yet to get all the codes (0-255) out of the scope in RAW (or any other) mode. The best I see is a low of 22 to a high of 184. This is reducing the dynamic range by about 5 dB. I've tried many ideas to increase this range. I suspect this is by design of the scope and not related to the ADC.  

I will add the scripts to the git repository, but don't plan to document or polish much for now. 

I am a student of the FFT. My wife and I published some theoretical research regarding the computational complexity of certain FFTs that I am very proud of. The result is even mentioned on wikipedia. Search for my last name, Haynal.

73,

Steve

KF7O

[Steve Haynal]

I forgot to mention that the second plot in this thread does use windowing. The code applies a Hanning window, although any window can be specified. I correctly rescale after applying any window so that power measurements should be correct. 

[Alan Hopper]

Steve,

great work, I think a ds1054z will be added to my xmas list.  I've just been adding a proper zoom fft to my radio software to reduce memory usage when zooming in so my head is also full of fft spectrum stuff.

73

Alan M6NNB

[Steve Haynal]

Hi List,

I was thinking a bit more about this and think a hybrid approach may be best. There is no way you can properly display 12M bins on most monitors, and people don't like to see large variance in the noise. So, for visualization one can compute multiple shorter FFTs and average. But behind the scenes, for better frequency and amplitude precision, the 24M FFT can still be computed. Final numbers would be based on that more accurate FFT. The averaged FFT could also be used to identify general locations of true peaks.

[Alan Hopper]

Steve, as a sanity check have you tried feeding a clean artificial signal into your script, I found this a great help with my spectrum display.

73 Alan  M6NNB

[Steve Haynal]

Hi Alan and John,

I added synthetic signal generation where the signal is an 8-bit data capture set to match the DS1054Z. Gaussian noise can be added. I also generalized the code so that multiple spectrum can be computed for the same dataset, and you can specify the number of equal length partitions to average. Setting this number to 1 defaults to a single long FFT with no averaging. It is interesting to look at the various plots and they illustrate why I'm not that excited about averaging.

Below is the synthetic signal (3 signals: 20dBm at 7.123456MHz, 15dBm at 18.987654MHz, 10dBm at 3.989898MHz) with no noise added. The FFT is averaged 240 times. Bin width is 500Hz. There is noise due to 8-bit quantization and computation.  I found that the flattop window is the best to preserve power, and the power readings are right on the money. There is about 85 dB range before the signal start to become visually lost in the noise.

Here is the same exact input but with no averaging. I clip everything less than 105 dB for display as I can't visually distinguish a signal that low in this plot. There is about 95 dB range before a signal starts to become visually lost in the noise. (Note that I am using careful language here about visually lost in the noise, not the average noise floor which is also better without FFT averaging.) This is 10 dB better than with averaging, and the frequency resolution much better at 21 Hz per bin. With the flattop window, the power estimates are the same and just as accurate.

  Now if we add Gaussian noise to the 240 average we see that the noise variance is small as expected when averaging a random distribution 240 times, but signals start to get visually lost at about 57 dB. 

For the same signal + same noise with no averaging, we get the better frequency resolution and an additional ~14 dB until the signal get visually lost.

  

There are a couple of issues I am still working on. First, my setup with the scope is picking up FM radio signals and other EM noise! I think I can remove this noise by making a sample set near the same time without the TX signal under test and doing simple spectral subtraction. There are many denoising algorithms and I might try one or two. Second, I have to think carefully about the probe, probe compensation and if there is a LPF behavior of the scope. So far things look good up to 250 MHz, but the probes are rated up to 150 MHz. Third, I think some of my measurements with the scope are clipping even though I do not reach full codes (0 or 255). Something is going on in the scope. This results in splat across the spectrum. I can see the same splat if I clip the synthetic signal.

I hope to start making and recording real measurements of the Hermes-Lite this weekend too.

73,

Steve

KF7O

[Alan Hopper]

Steve,

that all looks very convincing, I think my now unused flex 1500 will be traded in for one of these scopes.  It was your comments about the different behaviour of various sdr software with your original hermes lite test signal that made me create my emulator which in turn has greatly eased development of my radio software.

Is this running with the 100Mhz scope firmware?

73 Alan M6NNB

[Steve Haynal]

Hi Alan,

Yes, I am running the 100MHz scope firwmare on the DS1054Z. This is a brilliant marketing ploy by Rigol in my opinion.

BTW, I have updated a home Windows install to Windows 10. I tried your software (not the latest version) and it doesn't start. Nothing happens or shows up even when run from a command prompt. It appears to try and do something for 1 or 2 seconds but then just returns with no message via GUI or command prompt. Any ideas? I'm probably missing something that has already been discussed on this list somewhere... I will also try the version you just released but am away from that computer at the moment.

73,

Steve

KF7O

On Saturday, October 10, 2015 at 5:11:15 AM UTC-7, Alan Hopper wrote:

Steve,

[Alan Hopper]

Steve,

have you got .net 4.5 and the 32 bit visual studio 2013 redistributable installed?  You could look in C:\Users\@@your name@@\AppData\Roaming\m6nnb\radio\settings there might be a log.txt file that might give a clue.  It works here on windows 10.  You could try deleting the settings directory just in case there is an issue between versions but I'm not aware of any.

73 Alan M6NNB

[John Laur]

Hi, Steve.

The results look great! I now understand properly that when you said clipping low level signals you were just trimming the Y axis of the FFT. I had taken your statement to mean that you were clipping samples in the time domain (!) so you can imagine why I thought that was a bad idea. I am really sorry about that thanks for the careful clarification. I did not mean to come back with a reply that was far too basic. The actual algorithms of the FFT are for the most part largely over my head; I am very glad there are people like you who understand it and can help to optimize it.

I have also been doing some experiments (I am using perl/PDL instead of python and matlab). There are two things I am looking at:

First, even if you use a smaller FFT for display purposes, I do not think that you necessarily have to rely on the FFT bin size to get a precise measurement of frequency. For these harmonic measurements when you can be reasonably sure there is a single sinusoidal signal peak in the bin, you can use the phase information of the bin between two FFTs taken at a known interval to determine instantaneous frequency more precisely than looking at the bin energy alone. I believe this is how most sampling SA's must do it because they will provide for peak measurements that are not centered on the bin. There may also be other ways to do it as well. The power measurement should be able to be made fairly precise too even with larger bins; maybe consider subtracting the expected noise power in the bin if it ends up being significant  enough. I have not looked into these ideas very much or run any research on them, so could be very wrong on these assumptions...

Second, I am looking at running overlapping FFTs to compute a spectral density plot similar to how the data is displayed on real-time spectrum analyzers or in gr-fosphor. So for an FFT length of 1 megapoint, with an overlap of 1 sample, up to 23 million transforms can be aggregated to examine spectral density. This may also allow for an oversampled 1gsps source to be sliced every 4 samples to compute FFT from 0-125MHz which is probably more reasonable given the capabilities of the scope. Slicing every 3 samples might be even better since 167mhz is a more reasonalbe approximation of the range of the frontend, and the scope cannot be made to run at this rate natively. Granted you get only a couple ms of time with this samplerate, so it might not be terribly useful, but it might be enough to establish the fundamental technique for future projects.

This document has some good theory of operation stuff in it for sampling spectrum analyzers. In fact, it is oddly comprehensive while not getting too terribly technical. http://circuitslab.case.edu/manuals/Real_time_Spectrum_Analyzer_Fundamentals_-_Tektronix.pdf I havent read it all yet, but I did notice some interesting bits related to how they measure power - how it is computed, what windowing function is used (They default to Kaiser) and how the windowing function is adjusted for the desired RBW.

I have not tried talking with the scope yet. I have been working all with synthesized data. Since I can't find anything off the shelf for perl, I was going to give it a shot over LXI/ethernet as it seems like it would be easier to implement than USB. It might also allow faster transfers which would be extremely welcome.

Picking up FM is interesting; I figured it would be possible on purpose; in fact I had planned on trying it, but it kind of sucks that it's getting in there unwanted.

Regarding the raw values not going to full scale, the range of 22-184 you are seeing does not seem great -- that implies that when the scope is AC coupled the dc bias into the ADC's not well centered in the dynamic range of the ADC (0V at ~103 instead of 128). Not much to do about that.I guess, but you should be able to eek out a bit more dynamic range if you can either boost the input signal into the scope by a few dB or adjust the frontend to the next smallest v/div scale (which might have a more centered bias point too) and use an adjustable attenuator to try to get the ADC to run out to full scale. All of that is basically manually accounting for the fact that the scope does not have the step attenuator or preamps on the frontend like most SA's have to deal with the problem of getting the signal pushed to ADC full scale. Have you run the self-cal? I doubt it would affect the raw ranges but it might be doing some internal scaling. If you haven't, give it a shot but be aware it takes a good amount of time on this scope. A final thought is that there are clamp diodes on the frontend that you might hit -- you might hit clamping before you actually get the ADC to full scale (Rigol may do this to avoid clipping), and you may not be able to see it on the scope itself since it will be way off screen.

Alan, I am really looking forward to seeing what you have done w/ your software. If you have a chance, you might check out gr-fosphor and consider a similar plotting mode using overlapping FFT. Have you had a look at the DirectX 11 FFT API? I wonder if it is as fast or capable as the CUDA FFT? Fosphor includes a basic OpenCL FFT kernel as well -- license may determine what of those you could use...

73, John K5IT

[Alan Hopper]

John,

gr-fosphor looks very interesting, I put a overlapping option in my zoom fft but have not used it yet.  My current display annoys me on wspr and jt9/65 transmit as random timings can make one transmission look different to another depending on whether the symbol transitions happen near the middle or edge of the hopping window, the fosphor approach should prevent this.

The last time I looked at the directx fft(a long time ago) I think it was limited to floats and my software currently uses doubles for almost everything, I shall have another look..

73 Alan M6NNB