RTL-SDR Data File Histograms

[Michael Mruzek]

Hi, I am using RTL-SDR dongles for hydrogen line data acquistion. One of my observing goals is to record the raw I-Q data to a file during specific observations to create a record. I am using the rtl-sdr command line tools to record the data. ($ rtl_sdr -f 1420400000 -g 10 etc.) I know this will create large files, and have a 2 TB drive for the purpose.

Data from the RTL-SDR dongles is a stream of I/Q values expressed as bytes. As you know a byte can range from 0-255. The I/Q data is stored in the data file as offset integers, which can be thought of as having a zero point of 127. 

I have been using a hex file editor to examine the bytes in the data files while I vary gain of the SDR. The data can be plotted in a histogram. The histogram conveniently shows the distribution of the bytes.

Attached are plots for a gain of 1, 10, 20, and 30. I believe we want to use a reasonable fraction of the dynamic range of the ADC, but avoid clipping/pileup as can be seen for high gain.  I'm wondering if anyone has thoughts or comments about what would be the optimum gain setting in this situation?

Michael

[Marcus D. Leech]

The optimal gain setting will depend very much on your local environment, and what the rf "gain chain" looks like ahead of
  the receiver.

It will also depend on your observations.   For example, if you set it so that the Sun just barely causes clipping occasionally, it
  may not be quite enough for some weaker objects.   The ADC in the RTL-SDR has roughly 40dB of dynamic range at the end
  of the day.  If the quiet sun represents perhaps 200K of antenna temperature rise in your setup, then even an object that
  causes an only 1K Tant rise is still squarely within your dynamic range.

--
--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/e5ee0db0-49f5-4bb7-b967-b22f05596883n%40googlegroups.com.

[Alex P]

Hello Michael,

Just wondering why you are considering recording all of the IQ Data ?

With smaller dish systems ( 1m-2m ), the Hydrogen RF is barely above the noise level of the sky .. meaning that  1-5 Minutes of data

may be required to create a well defined, clean  spectrum .. That equates to say 50,000 to 800,000 samples... and probably towards the high side

of that range. 

Also, with smaller dishes, the beam-width is so broad that nothing much changes in less than say 0.5 to 1 degree of RA drift .. ( 2min - 4 min of time )

1.2 m dish  5 minutes of data per saved file  ( ~ 850,000 averages per saved file )

And as Marcus commented, you need to optimize  the gain of the LNA during the acquisition of actual H_Line data

Alex

[Jim Abshier]

Michael - Attached is a family of plots showing ADC quantization noise
as a function of input signal level. Gaussian noise-like signals are
assumed. In general the SNR increases as the input level increases due
to finer quantization (more bits involved) and eventually an optimum
point is reached. Beyond this optimum point the SNR decreases rapidly
due to clipping. As shown, the curves are different for different ADC
resolutions.

Jim Abshier

> --
> --
> You received this message because you are subscribed to the Google
> Groups "Society of Amateur Radio Astronomers" group.
> To post to this group, send email to sara...@googlegroups.com
> To unsubscribe from this group, send email to
> sara-list-...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sara-list?hl=en
> ---
> You received this message because you are subscribed to the Google
> Groups "Society of Amateur Radio Astronomers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sara-list+...@googlegroups.com

> <mailto:sara-list+...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sara-list/e5ee0db0-49f5-4bb7-b967-b22f05596883n%40googlegroups.com

> <https://groups.google.com/d/msgid/sara-list/e5ee0db0-49f5-4bb7-b967-b22f05596883n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Lester Veenstra]

Two points: (1) are all RTL0SDr 8 bit devices. Some indicate higher.

(2):  Percent time in clip;  The percent of time (Assume 8 bit a/D)  in clip should be down in  0.01 percent.

Above this the intermod distortions (NPR)  goes up quickly.

 

PTIC  = (Number of zero values observed+Number of 255 values observed)/total number of sables observed

 

Lester B Veenstra  K1YCM  MØYCM  W8YCM   6Y6Y W8YCM/6Y 6Y8LV (Reformed USNSG CTM1)

les...@veenstras.com

 

452 Stable Ln

Keyser WV 26726 USA

 

GPS: 39.336826 N  78.982287 W (Google)

GPS: 39.33682 N  78.9823741 W (GPSDO)

 

 

Telephones:

Home:            +1-304-289-6057

US cell          +1-304-790-9192

Jamaica cell:    +1-876-456-8898

--

--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/e5ee0db0-49f5-4bb7-b967-b22f05596883n%40googlegroups.com.

[Marcus D Leech]

All devices based on the RTL2832 are 8-bit. 

There are devices that use the same tuner chip like the AirSpy that are 12-bit. 

Sent from my iPhone

On Oct 4, 2023, at 9:40 AM, 'Lester Veenstra' via Society of Amateur Radio Astronomers <sara...@googlegroups.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/00cd01d9f6c8%2458a300f0%2409e902d0%24%40com.

[Michael Mruzek]

Hi. Just a quick update on analyzing the raw IQ data from an RTL-SDR dongle taken with my 3M TVRO satellite dish. In my previous post I mentioned that I capture the raw IQ data using rtl_sdr on a Linux machine.  A 20 minute data file is about 5 GB.  My goal is to analyze the raw data using homebuilt tools that I can tweak for different purposes.  I am using the C programming language and the gcc compiler. My analysis program converts the single-ended byte data to complex data with real and imaginary components.  From there I am applying a Discrete Fourier Transform from the FFTW library. (Fastest FFT in the West at fftw.org ). Using FFTW is a little bit tricky and there was a learning curve.  After some averaging I plot the data using gnuplot, in some cases I subtract a background calibration run.  Attached is an example from 4 minutes of a galactic center transit.   I find it pretty amazing that this can be done with a small C program. I plan on writing this up in a webpage with more details after fixing the plot to display frequency on the horizontal axis.  Michael

[Lester Veenstra]

Always infested in code that can capture raw IQ from STL-SDR.

We can put it up in the drop box repository if you are willing.

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/059661e8-5f3f-4e51-a472-5ff7c9976ef7n%40googlegroups.com.

[Marcus D. Leech]

On 26/10/2023 12:47, 'Lester Veenstra' via Society of Amateur Radio Astronomers wrote:

Always infested in code that can capture raw IQ from STL-SDR.

We can put it up in the drop box repository if you are willing.

The "rtl_sdr" utility that comes with the library does this.

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/086401da082c%241f3f0d20%245dbd2760%24%40com.

[Lester Veenstra]

URL  please

 

Lester B Veenstra  K1YCM  MØYCM  W8YCM   6Y6Y W8YCM/6Y 6Y8LV (Reformed USNSG CTM1)

les...@veenstras.com

 

452 Stable Ln

Keyser WV 26726 USA

 

GPS: 39.336826 N  78.982287 W (Google)

GPS: 39.33682 N  78.9823741 W (GPSDO)

 

 

Telephones:

Home:            +1-304-289-6057

US cell          +1-304-790-9192

Jamaica cell:    +1-876-456-8898

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/7bb2e56d-0b69-4655-99d9-dc8bd4cbcfd7%40gmail.com.

[Marcus D. Leech]

On 26/10/2023 13:16, 'Lester Veenstra' via Society of Amateur Radio Astronomers wrote:

URL  please

I assume you're asking about windows.   I'm not a Windows guy, but here's a guide to
  building from source:

https://www.onetransistor.eu/2017/03/compile-librtlsdr-windows-mingw.html

On my Ubuntu system, the "rtl_sdr" command got installed when the library got installed.

rtl_sdr, an I/Q recorder for RTL2832 based DVB-T receivers

Usage:     -f frequency_to_tune_to [Hz]
    [-s samplerate (default: 2048000 Hz)]
    [-d device_index (default: 0)]
    [-g gain (default: 0 for auto)]
    [-p ppm_error (default: 0)]
    [-b output_block_size (default: 16 * 16384)]
    [-n number of samples to read (default: 0, infinite)]
    [-S force sync output (default: async)]
    filename (a '-' dumps samples to stdout)

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/087801da0830%24388f0380%24a9ad0a80%24%40com.

[James Abshier]

Here is some code in octave for testing RTL-SDR receiver units. It uses rtl_sdr to collect 5000 samples and then parses it into complex data, computes a spectrum and plots the spectrum along with the real part of the complex data. This particular script repeats this process 5 times. Octave is available for both Linux and Windows.

Jim Abshier

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/7bb2e56d-0b69-4655-99d9-dc8bd4cbcfd7%40gmail.com.

[Lester Veenstra]

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/3bb0d4d9-922a-4150-9285-6f070e0b9233%40gmail.com.

[Lester Veenstra]

Thanks

 

Lester B Veenstra  K1YCM  MØYCM  W8YCM   6Y6Y W8YCM/6Y 6Y8LV (Reformed USNSG CTM1)

les...@veenstras.com

 

452 Stable Ln

Keyser WV 26726 USA

 

GPS: 39.336826 N  78.982287 W (Google)

GPS: 39.33682 N  78.9823741 W (GPSDO)

 

 

Telephones:

Home:            +1-304-289-6057

US cell          +1-304-790-9192

Jamaica cell:    +1-876-456-8898

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/890f1e37-6baa-44ae-b10f-999cd2ab1f25%40ameritech.net.

[Michael Mruzek]

Hi. Here is a link to the full package that contains rtl_sdr with instructions for installing/building based on your operating system.  I don't mean to start anything, but I have been a Windows guy most of my life, and I have to say using the terminal command line in Linux now is really fantastic for doing the scientific stuff. FWIW: I am writing a brief tutorial on how to build a Linux box using a junk computer.  I will post a link when I finish it. Michael

https://osmocom.org/projects/rtl-sdr/wiki/Rtl-sdr

You received this message because you are subscribed to a topic in the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sara-list/_k4rxceIRTY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sara-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/890f1e37-6baa-44ae-b10f-999cd2ab1f25%40ameritech.net.

[Captain Anne Flint]

Looking forward to your tutorial post - Wende 

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/CABR_EQOOgfknUxcDUzXC45AsOmkFPospNkX-0a0_gOpwZF-WRA%40mail.gmail.com.

[Marcus D. Leech]

On 26/10/2023 13:46, Michael Mruzek wrote:

Hi. Here is a link to the full package that contains rtl_sdr with instructions for installing/building based on your operating system.  I don't mean to start anything, but I have been a Windows guy most of my life, and I have to say using the terminal command line in Linux now is really fantastic for doing the scientific stuff. FWIW: I am writing a brief tutorial on how to build a Linux box using a junk computer.  I will post a link when I finish it. Michael

THIS.

One of the problems with using computers entirely through a GUI, is that you lose a lot of power as a user.  Anything that is
  kind of "procedural" across different applications is kind of hard to express using just point and click--humans are inherently
  users of language, and GUIs don't actually fit that very well in many cases.

Folks like Steve Jobs had the "you should never want to do that" attitude.  Which just dumbs everything down.

The Unix shell was conceived to allow non-programmers the ability to kind of synthesize new things by stringing existing
  tools together--either directly on the command line, or encapsulated into a shell script.  I guess Windows Power Shell now
  provides this, but, sweet Cthulu, Unix has been around since 1969, and pipeline functionality was added around 1973...

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/CABR_EQOOgfknUxcDUzXC45AsOmkFPospNkX-0a0_gOpwZF-WRA%40mail.gmail.com.

[Marcus D. Leech]

On 26/10/2023 13:46, Michael Mruzek wrote:

Hi. Here is a link to the full package that contains rtl_sdr with instructions for installing/building based on your operating system.  I don't mean to start anything, but I have been a Windows guy most of my life, and I have to say using the terminal command line in Linux now is really fantastic for doing the scientific stuff. FWIW: I am writing a brief tutorial on how to build a Linux box using a junk computer.  I will post a link when I finish it. Michael

https://osmocom.org/projects/rtl-sdr/wiki/Rtl-sdr

Also, tangentially related:

https://wiki.gnuradio.org/index.php/InstallingGR

Apparently, "Radioconda" makes it super-easy to install and use GnuRadio and friends on Windows these days.

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/CABR_EQOOgfknUxcDUzXC45AsOmkFPospNkX-0a0_gOpwZF-WRA%40mail.gmail.com.

[bsn...@gmail.com]

Michael,

Been running Linux (Slackware) on a "junk computer" since the mid-1990s that was assembled from separate motherboard, video & sound cards, ethernet, including a 5 1/2 floppy drive. Installed MatLab on it in the early 2000s and did years of productive academic work.

Don't know if the more modern Linux distros can do that but may depend on your definition of a "junk computer". :-)

Have fun and good luck....bill....

[Michael Mruzek]

Hi, I wrote a short tutorial on how to install Linux (Zorin Lite) on a garage sale computer.  It's what I have been using to record and analyze data files captured with an inexpensive RTL-SDR dongle.  If you have a junk computer laying around and have been tempted to try Linux this may help get you going.  This tutorial uses a junk 64 bit computer, I will be adding additional information for 32 bit machines. Michael

https://www.mtmscientific.com/linux.html

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/1428c5d3-a3a4-402e-958a-2c6fc39108fbn%40googlegroups.com.

[Captain Anne Flint]

Thanks Micheal! Wende 

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/CABR_EQMW_tHwz1H08mR0s55c6QsqvYhikNrx2FdKVSztoH_-Lg%40mail.gmail.com.

[Marcus D. Leech]

On 27/10/2023 09:00, bsn...@gmail.com wrote:

Michael,

Been running Linux (Slackware) on a "junk computer" since the mid-1990s that was assembled from separate motherboard, video & sound cards, ethernet, including a 5 1/2 floppy drive. Installed MatLab on it in the early 2000s and did years of productive academic work.

Don't know if the more modern Linux distros can do that but may depend on your definition of a "junk computer". :-)

Have fun and good luck....bill....

How deep you can go into computer paleontology to secure a system depends very much on what you want to do.
  If your approach is to record data in complex-baseband first, and then analyse later, you can get by with something
  fairly junky--as long as it can sustain recording from an RTL-SDR at 2.5Msps, then you just need enough disk space to
  handle those recordings.

If, however, like me, you tend to prefer real-time signal-processing (possibly from an SDR that produces more than 2.5Msps),
  you'll need a beefier computer.

My approach, for many years, has been to buy computers at the "trailing edge" -- my current laptop is an off-lease
  i5 6200U system that is about six years old.  I routinely use it for work with various SDRs that have higher
  sample rates than RTL-SDRs.

Our main receiver computer at the observatory is literally over 10 years old, but it's a server class XEON system with 8
  cores, and 16GB of memory.   It handles 4 channels at 10Msps per channel going into 2048-bin FFTs, which are then
  integrated, decimated, and recorded at a leisurely pace.  We're currently doing an observation that will last a couple of
  days.  If we were recording 40Msps of baseband samples, we'd run out of disk space (and disk bandwidth) fairly
  quickly.   But having the DSP processing chain do some of the "heavy lifting" up-front means you have less of an
  overwhelming "lump" to analyze later.

I long maintained that if you were the kind of ham-radio guy who would happily go out and spend $$$$$ every couple of years
  on the latest HF rig, surely spending a few hundred on a not-quite-bleeding-edge computer to do real-time astronomy
  signal processing isn't out of the question.

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/1428c5d3-a3a4-402e-958a-2c6fc39108fbn%40googlegroups.com.