Python script for RFI filtering in ezRA

[Miguel Ríos Martín]

Python script for RFI filtering in ezRA

Hello.

In my battle against RFI, I created a small Python script to filter the raw data obtained via ezCol.py.

The script processes the raw data file and applies the `savgol_filter` function with `WINDOW_LENGTH = 21` and `POLYORDER = 3` to smooth out RFI spikes (it is a very simple script).

Once smoothed, the file is processed using the standard tools (ezCon, ezPlot, etc.), and I was surprised to find that the results are quite good. The graphs appear smoother and more defined.

I am pasting a sample comparison of both.

[Stephen Arbogast]

Very  cool!

I have  used ezCon260214a.py to  remove  RFI ,  truncate  hi and lo  ends of spectrum  and  choose  a  reference.   Here  are my  commands...

Configuration

Discovery Dish
Disovery HI feed

AirSpy -> ezColS250524a.py
Sampling rate = 2.5 Meg samples/sec
Integration Count = 200,000, about 84 sec

Processing Command Lines

Note: -ezConRefMode31 11.5 REF = the one antB spectrum closest to Right Ascension 11.5 hours

python3 ../../ezRA/ezCon260214a.py data/Durango260527_16.txt  data/Durango260528_00.txt -ezConInputdB 1 \
-ezConRawFreqBinHideL 346 346 -ezConRefMode31 11.5  -ezConAntXTFreqBinsFracL 0.2 0.7 \
-ezConUseVlsr 1 -ezConAntXInput 6  -ezConAntXTVTClipL 1.3 999

Note: -ezConRefMode    10: REF = last REF sample marked in data, if none will use sample 0

python3 ../../ezRA/ezCon260214a.py data/Durango260527_16.txt  data/Durango260528_00.txt -ezConInputdB 1 \
-ezConRawFreqBinHideL 346 346 -ezConRefMode 10  -ezConAntXTFreqBinsFracL 0.2 0.7 \
-ezConUseVlsr 1 -ezConAntXInput 6  -ezConAntXTVTClipL 1.3 999

python3  ../../ezRA/ezSky.py  .  -ezSkyInput 18

Stephen

[Miguel Ríos Martín]

Where can I find that new version of ezRA—the one that includes ezCon260214a.py to remove RFI?

--
--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sara-list/xGWfCeKeeA8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sara-list+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sara-list/b0eb4445-32b3-4c02-82b1-8af61a27cb81n%40googlegroups.com.

--

Miguel Ríos Martín

[Miguel Ríos Martín]

This is the effect of the filtering function on any given line of the ezCol.py output file.

[Alex P]

Hi Miguel, 

That looks good.

Add a bit of Mean filtering after the Median processing.

[andrew....@googlemail.com]

You need to email Ted Cline directly as that file is unpublished – latest published version is available on Ted Cline’s Github page – search for “Github ezRA”.

 

Email me on and...@thornett.net if you don’t have Ted’s email address and I will send it back to you.

 

Andy

You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sara-list/CAFfVMO%2BFQLG_LvFonEXNjF_v3%2BPHUGC6tu%2B9fNO6Sddav2GopQ%40mail.gmail.com.

[tedcl...@gmail.com]

Hi Miguel,

Thanks for exploring new analysis methods.

Your method smoothes data before the ezRA ezCon analysis.
The posted   ezCon250711a   explores smoothing data at the end of the ezCon analysis.
I do not know which method is better.


After ezCon optionally
      hides troubled (RFI) frequencies, and
      trims off unneeded frequencies, and
      trims off troubled (RFI) sample spectra in time, and
      enhances the Galactic hydrogen signal, and
      chooses which enhancement method worked best, and
      and finally calculates the AntXTVT signal,
then ezCon provides several choices to smooth that   ezCon087   plot,
as if it was a photographic image, with the   -ezConAntXTVTSmooth   option:
      -ezConAntXTVTSmooth
            (smooth antXTVT heatmap data as an image, with a choice of filters)


      -ezConAntXTVTSmooth  1
            does a   cv2.blur(antXTVT,(5,5))   once.
      -ezConAntXTVTSmooth  11
            does a   cv2.blur(antXTVT,(5,5))   twice.
      -ezConAntXTVTSmooth  111
            does a   cv2.blur(antXTVT,(5,5))   3 times.

      -ezConAntXTVTSmooth  2
            does a   cv2.GaussianBlur(antXTVT,(5,5),0)   once.
      -ezConAntXTVTSmooth  22
            does a   cv2.GaussianBlur(antXTVT,(5,5),0)   twice.
      -ezConAntXTVTSmooth  222
            does a   cv2.GaussianBlur(antXTVT,(5,5),0)   3 times.
   
That   ezConAntXTVTSmoothDo()   code notes that you will need to install python3-opencv:
    # Ubuntu22 cv2 install:
    #   sudo apt-get install python3-opencv
    # Windows10 cv2 install (may work but not tried):
    #   https://docs.opencv.org/4.x/d5/de5/tutorial_py_setup_in_windows.html
    #       https://github.com/opencv/opencv/releases
    #           opencv-4.11.0-windows.exe
    # Windows10 cv2 install (provided too much):
    #   https://docs.opencv.org/4.x/d3/d52/tutorial_windows_install.html
    #       https://sourceforge.net/projects/opencvlibrary/files/
    #       https://sourceforge.net/projects/opencvlibrary/files/4.11.0/
    #       https://sourceforge.net/projects/opencvlibrary/files/latest/download
    #           ran that opencv-4.11.0-windows.exe
    # Windows10 cv2 install (worked on Win10):
    #   https://github.com/opencv/opencv-python?tab=readme-ov-file#supported-python-versions
    #       py -m pip install opencv-python
    #
    # https://opencv24-python-tutorials.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_filtering/py_filtering.html


I attach some after-smoothing   ezCon098   plots: 0, 1, 111, 2, 222.


I have not posted    ezCon260214a   because I consider it experimental and do not fully trust it yet.

Ahh, so many distractions !

Helpful ?

---
Ted Cline   N0RQV
TedClineGit at gmail.com

On Thursday, October 8, 2026 at 10:33:00 AM UTC-6 Miguel Ríos Martín wrote:

[Miguel Ríos Martín]

Hi.

I’ve slightly modified the pre-filtering script following Alex's suggestions. The result is a plot with even fewer spikes (see attachment).

I’m also attaching the script. It is used as follows (where `fichero_datos` is the text file generated by ezCol):

python3 Clean_RFI.py fichero_datos

This generates a new file named `RFI_clean_fichero_datos`. You then delete `fichero_datos` (so it doesn't interfere) and process the result using the ezRA graphical tools (ezCon, ezSky, etc.).

Andy, of course you can use the filter—you're more than welcome to.

Ted, how can I get the file you mentioned, `ezCon250711a`? I’d like to try it out, if you don't mind.

--

--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/sara-list/e5462ba7-e3f2-4958-ba44-208227601f95n%40googlegroups.com.

--

Miguel Ríos Martín

[b alex pettit jr]

RFI is typically very narrow band, these peaks seem to be 2-3 chs wide as if there has been

low pass filtering Before the Median Filtering....

the method will work better if the spikes are very narrow

( the orange correction is being shifted by the amplitude of wide spikes )

  HL3D  

Alex Pettit

=============================================

[b alex pettit jr]

 HL3D post-processing allows adjustment of the RFI filtering to best fit the noise.

( "Filter Width" implements Median Filtering using a moving window of specified width & "Smooth"  is a moving window Mean Filter )

                      very simple     very effective

https://github.com/AP-HLine-3D/HLine3D/blob/main/Info_HL3D_processing_RFI%20Eraser01.pdf

Alex Pettit

=============================================