polynomial fitting

[Eduard Mol]

Hi all,

During last Sunday's Drake's Lounge there was some discussion about bandpass correction and polynomial fitting. For those of you who have not followed Drake's Lounge here is a bit of context.

When trying to do spectroscopy on faint and difficult targets (e.g. other galaxies in HI, masers...) the signal is often completely obscured by the uneven bandpass shape of the SDR receiver/ radio telescope system. Proper background subtraction to get a good flat spectrum is really the key to success for such observations. Typically, we take on-target spectra and an equal number of off-target spectra (with the telescope pointed away from the target), then subtract or divide the off-target spectra from the on-target "dark" spectra. This way we (hopefully) get rid of the system response leaving only the faint spectral signal of the target. Unfortunately, it is common that the bandpass shape slightly changes during recording due to factors like temperature variation, leaving a residual slope in the spectrum after background subtraction. This is especially problematic with long observation runs, which are needed to detect the faintest targets. If you are lucky it is a straight slope that can simply be removed by fitting a line, but sometimes the curve is more complex and a (higher order) polynomial may be needed. However, that comes with its own problems.

During the Drake's Lounge I showed an example of this effect in a recent ~2 hour long recording of Orion KL at the 22.2GHz water line. Due to variations in temperature, atmospheric moisture etc there was a strong residual curve in the spectrum after background subtraction, which could be reasonably approximated with a second order fit:

However even after subtracting this curve the spectrum was still not as good as I would like to see:

For comparison here is a spectrum taken under much better conditions last winter (low humidity and stable temperature). As you can see there are some stronger narrow spikes but also some broader, weaker features between 0- 10 km/s and 15-20 km/s, which are actually composed of numerous weak maser features. 

Today I tried a 5th order polynomial fit on the data from June 2 to see if I could get a result that is closer to the spectrum from last winter shown above. As you can see the spectrum is a lot flatter but it is also a bit overcorrected. The broader features are mostly lost and instead there appears to be an absorbtion feature at 8- 15 km/s, which does not really exist! I think it is just the gap between the two broad emission features in the spectrum. 

Anyway, I think the lesson here is that while polynomials could be useful for spectral data processing, one has to be careful with applying higher (>2nd) order polynomials to weak signals in noisy data since that could easily introduce artifacts. 

Eduard. 

[Alex P]

Nicely Done

Another idea, wait a few months till the area of interest is in a more linear part of the LSR correction curve

[Marcus D. Leech]

I have also experimented with fitting polynomials to the baseline, and had, as you had, mixed results. 

[b alex pettit jr]

High order polynomial curve fit solutions can be Amusing..

They can also be well behaved and Correct.

This is a 6th order Matlab polyfit of LSR correction data calculated at 1/2hr intervals ( Blue Circles ) vs the polyfit solution ( Orange Line )

The 1/2 hr interval data was from a Matlab routine provided by F1EHN and compares with this calculator

F4KLO - Dimension Parabole

F4KLO - Dimension Parabole informations sur le radiotelescope de la villette

===================================================================================

( the X axis is scaled in acquisition samples per day )

Alex

[Nathan Towne]

A suggestion is to exclude the two sharp peaks near the bottom of the curve as the outliers that they are - they are not background.  This will improve the fit of a second-order polynomial to the background at least a little.

-- 

  Nathan Towne

  tow...@ownmail.net

On Tue, Jun 18, 2024, at 6:24 AM, Eduard Mol wrote:

Hi all,

During last Sunday's Drake's Lounge there was some discussion about bandpass correction and polynomial fitting. For those of you who have not followed Drake's Lounge here is a bit of context.

When trying to do spectroscopy on faint and difficult targets (e.g. other galaxies in HI, masers...) the signal is often completely obscured by the uneven bandpass shape of the SDR receiver/ radio telescope system. Proper background subtraction to get a good flat spectrum is really the key to success for such observations. Typically, we take on-target spectra and an equal number of off-target spectra (with the telescope pointed away from the target), then subtract or divide the off-target spectra from the on-target "dark" spectra. This way we (hopefully) get rid of the system response leaving only the faint spectral signal of the target. Unfortunately, it is common that the bandpass shape slightly changes during recording due to factors like temperature variation, leaving a residual slope in the spectrum after background subtraction. This is especially problematic with long observation runs, which are needed to detect the faintest targets. If you are lucky it is a straight slope that can simply be removed by fitting a line, but sometimes the curve is more complex and a (higher order) polynomial may be needed. However, that comes with its own problems.

During the Drake's Lounge I showed an example of this effect in a recent ~2 hour long recording of Orion KL at the 22.2GHz water line. Due to variations in temperature, atmospheric moisture etc there was a strong residual curve in the spectrum after background subtraction, which could be reasonably approximated with a second order fit:

However even after subtracting this curve the spectrum was still not as good as I would like to see:

For comparison here is a spectrum taken under much better conditions last winter (low humidity and stable temperature). As you can see there are some stronger narrow spikes but also some broader, weaker features between 0- 10 km/s and 15-20 km/s, which are actually composed of numerous weak maser features. 

Today I tried a 5th order polynomial fit on the data from June 2 to see if I could get a result that is closer to the spectrum from last winter shown above. As you can see the spectrum is a lot flatter but it is also a bit overcorrected. The broader features are mostly lost and instead there appears to be an absorbtion feature at 8- 15 km/s, which does not really exist! I think it is just the gap between the two broad emission features in the spectrum. 

Anyway, I think the lesson here is that while polynomials could be useful for spectral data processing, one has to be careful with applying higher (>2nd) order polynomials to weak signals in noisy data since that could easily introduce artifacts. 

Eduard. 

--

--

You received this message because you are subscribed to the Google

Groups "Society of Amateur Radio Astronomers" group.

To post to this group, send email to sara...@googlegroups.com

To unsubscribe from this group, send email to

sara-list-...@googlegroups.com

For more options, visit this group at

http://groups.google.com/group/sara-list?hl=en

---

You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/CAFcm1Q4PSVq_ash%2BoFgx60PWw4X0zA3aKT1mZ3Y4w3u1j5DhRQ%40mail.gmail.com.

[Eduard Mol]

Yes, in fact I excluded the strongest peak. It’s in the window marked by the green dots. I’d have to go back and adjust my code to exclude more than one peak, but I’d expect it would only marginally improve the fit.

Op di 18 jun 2024 om 16:53 schreef Nathan Towne <tow...@ownmail.net>

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/d7fc5390-940e-4b80-a516-8836d3805452%40app.fastmail.com.

[Dimitry UA3AVR]

Yes, Eduard ... this is a good alternative to the traditional way of the background correction with integrating off-target spectra. At least, the polynomial fitting do not bring an excessive noise to the spectrum besides other advantages.

вторник, 18 июня 2024 г. в 18:29:00 UTC+3, Eduard Mol:

[Eduard Mol]

Hi Dimitry, I think it depends on the situation and the equipment whether or not polynomial fitting works well. 

So far I have been using just background subtraction using off-target spectra, although this contributes some noise this technique has so far given me the most consistent results with my setup. 

At the Dwingeloo telescope we use a sort of hybrid approach, fitting a polynomial through the off-target spectra and then subtracting that curve from the on- target spectra, instead of subtracting off- target from on-target spectra directly.

I have also experimented with applying a moving average filter (for example 5 point or 7 point moving average) to the off-target spectrum to reduce the noise contribution. So far I’ve only noticed a marginal improvement in SNR.

Best regards,

Eduard

Op do 20 jun 2024 om 10:02 schreef Dimitry UA3AVR <ua3avr...@gmail.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/89e78afb-357d-41a7-9bc9-a19ff636d3a1n%40googlegroups.com.

[fasleitung3]

Am Donnerstag, den 20.06.2024, 16:08 +0200 schrieb Eduard Mol:

At the Dwingeloo telescope we use a sort of hybrid approach, fitting a polynomial through the off-target spectra and then subtracting that curve from the on- target spectra, instead of subtracting off- target from on-target spectra directly.

I have also experimented with applying a moving average filter (for example 5 point or 7 point moving average) to the off-target spectrum to reduce the noise contribution. So far I’ve only noticed a marginal improvement in SNR.

This is the scheme we also use as standard. This avoids adding noise to the spectrum.

Wolfgang

[Dimitry UA3AVR]

I have applied this scheme for recent 6.7 GHz observations. Here the plot with correction by raw off-target background:

... and when the off-target background is fitted by 5th order polynomial:

The integration time for the off-tagret background and the source was the same; so expected difference in the noise floor is about sqrt(2)=1.41... times.

четверг, 20 июня 2024 г. в 19:33:01 UTC+3, fasleitung3:

[b alex pettit jr]

Dimitry,

Can you plot the Correction Curve alone ?

Thanks,

Alex

==================================

On Monday, June 24, 2024 at 04:51:03 AM EDT, Dimitry UA3AVR <ua3avr...@gmail.com> wrote:

I have applied this scheme for recent 6.7 GHz observations. Here the plot with correction by raw off-target background:

... and when the off-target background is fitted by 5th order polynomial:

The integration time for the off-tagret background and the source was the same; so expected difference in the noise floor is about sqrt(2)=1.41... times.

[Dimitry UA3AVR]

Sure, Alex, thanks ...

The form of frequency response of my receiver is significantly uneven in a full SDR bandwidth and bell shaped:

The scale is too large to see differences in raw and smoothed background, but with some zoom of the central part the difference can be seen:

Center part of the bell (with anomaly fall due to DC suppressing in SDR) and edge slopes are excluded from the fit.

понедельник, 24 июня 2024 г. в 13:13:03 UTC+3, b alex pettit jr:

[andrew....@googlemail.com]

Hi Folks,

 

My map so far….

 

https://astronomy.me.uk/successful-processing-all-viable-data-to-22-6-2024

 

Andy

[b alex pettit jr]

Andy,

What are these plots showing ?

They do not look like H-Line spectra

Alex

On Wednesday, June 26, 2024 at 04:57:04 PM EDT, andrew.thornett via Society of Amateur Radio Astronomers <sara...@googlegroups.com> wrote:

Hi Folks,

My map so far….

 Andy

--
--

[andrew....@googlemail.com]

I don’t really know – from ezRA suite – Ted, do you know?

Andy

--

--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/483417779.5851366.1719436568548%40mail.yahoo.com.

[tedcl...@gmail.com]

Wow, this is way off the original post topic.

"What are these plots showing ?"

To study the Galactic arms, one must collect spectrum data from many Galactic plane Longitudes.

One must try to "collect them all".

And better if many samples of each.

On the left is the ezRA software ezGal519 plot showing the number of Galactic plane samples collected, by their 1-degree Galactic Longitude.

Some Galactic Longitudes will be hard to collect from a northern hemisphere telescope location.

Something of a scorecard of current accomplishments.

The "ezGal511velGLonCount.txt" file is a text version of this scorecard, and includes text for an MIT Haystack SRT command file to automate the collecting of the missing Galactic Longitudes with a motored antenna.

I think the left side of this plot says you have collected 200 of the 361 possible 1-degree Galactic Longitudes.

On the right is the ezGal516 plot showing the average velocity collected, by their 1-degree Galactic Longitude.

Not shown is an ezGal517 plot showing the maximum velocity collected, by their 1-degree Galactic Longitude.

More interesting than the ezGal516 plot ?

Helpful ?

---
Ted Cline   N0RQV
tedclinegit at gmail.com

I posted updated ezRA files on June-16-2024.

For folks starting to explore radio astronomy,
      ezRA - Easy Radio Astronomy
      Free 1420 MHz Galactic hydrogen data collection and analysis
      Open source
      Windows and Linux
      https://github.com/tedcline/ezRA

On Wednesday, June 26, 2024 at 3:16:12 PM UTC-6 b alex pettit jr wrote:

Andy,

What are these plots showing ?

They do not look like H-Line spectra

Alex

On Wednesday, June 26, 2024 at 04:57:04 PM EDT, andrew.thornett via Society of Amateur Radio Astronomers wrote:

 

[fasleitung3]

Ted, Andy,

My understanding of Andy's recording is that he did drift scans. This would mean that for each scan he is covering 24 hrs of right ascension at one specific declination depending on the elevation of his telescope (and, of course, his location).

Expressing this in galactic corrdinates means that it covers a certain part of galactic longitudes and galactic latitudes. So this is not a scan of the galactic plane per se.

The plots of the individual spectra do not show the galacitc latitude (or am I missing something here?) of the location. It seens that it is assumed that the galactic latitude is 0, which in case it is not.

So I am a bit confused, maybe you can clarify.

Regards,

Wolfgang

--

--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/0791a8b8-e33c-4728-ae98-772c763a0cf1n%40googlegroups.com.

[tedcl...@gmail.com]

Hi Wolfgang,

Once the data is collected, ezCon and ezGal can selectively focus on the data subset from different Galactic Latitudes with the   ezConGalCrossingGLatCenter   parameters, but the default is Galactic Latitude zero, the plane of the Milky Way.

I see "P00.0" in the filename at the top left of those plots, which says the mentioned  ezGal516, ezGal517, and ezGal519 plots are from that default focus on Galactic Latitude zero.  The default   ezConGalCrossingGLatNear   value is a wide 5 degrees.  So I expect the focused data subset is the samples from Galactic Latitude  -5 to +5 degrees.

Much of the ezRA analysis is to help study and improve the data set.

---
Ted Cline   N0RQV
tedclinegit at gmail.com

[fasleitung3]

Thanks for the clarification. So it picks the spectra which are from positions close to the galactic plane from the total dataset.

Cheers,

Wolfgang