Question on RMS-based integration time planning for HI observations

[Tiago Baroni]

Dear all,

I am at a very early stage in amateur HI radio astronomy and have been trying to move away from the usual heuristics I often see (fixed numbers of integrations, “rule-of-thumb” observing times, etc.), which I find hard to justify quantitatively when planning observations with small instruments.

As an exercise, I have been working through a simple, iterative planning approach based on the measured RMS of pilot observations and the radiometer equation. In essence, the idea is to (i) measure the RMS after baseline removal in line-free regions, (ii) compare the observed RMS with the expected thermal RMS to check whether the system is still in a thermally dominated regime, and (iii) only scale integration time according to the 1/τ​ law while that condition holds. When the RMS stops decreasing efficiently and appears to asymptotically approach a constant value, this is treated as an instrumental/systematic floor and used as a practical stopping criterion, rather than continuing to integrate blindly.

I am not claiming any novelty here; this is mainly an attempt to formalize, in a transparent way, decisions that are often made implicitly. I would very much appreciate feedback from the community on whether this conceptual approach makes sense in practice, what assumptions might be too optimistic for small-dish / SDR-based setups, and whether there are known statistical or instrumental pitfalls (e.g. correlated noise, baseline effects, RFI handling) that tend to invalidate this kind of RMS-based planning if one is not careful.

Comments on how applicable (or not) this framework is across different receivers, backends, or observing strategies would also be extremely helpful. I am very open to corrections and criticism, as this is primarily a learning exercise.

Best regards,
Tiago Baroni

[Alex P]

Hello Tiago,

In principal, your idea is valid ....

BUT, since you are using an SDR , why integrate a Doppler Velocity Shifted Spectrum into a single RMS value ?

I am sure that will stabilize in a very short (5- 10 sec ? ) time period, but capturing data at a rate faster than say 1/10th to 1/20th the beam width of your antenna 

may yield huge numbers of samples per day, but with little useful value, and if observed as a spectrum : quite noisy .

This data is from a 1m class dish and 300 seconds  ( 1.25 degrees of RA drift ) integration per sample 

Regards,

Alex Pettit

https://github.com/AP-HLine-3D/HLine3D

[fasleitung3]

Hi Triago,

Your approach is perfectly fine. In fact, this is exactly what we are asking the master students to do which come to us for a lab course:

As a first step, they measure the system temperature by determining the rms noise of the baseline in comparison to a calibration source like S7. Then they are supposed to observe the hydrogen spectrum of another galaxy. For this, they need to determine what integration time is needed for a specific desired SNR. Taking the brightness temperature of the galaxy from literature, they can do this by applying the radiometer equation. So indeed, what you are proposing is done in practice.

Of course, there are certain limits and caveats which you have also pointed out: RFI will contribute to the baseline rms noise and typically will not decrease with integration time. So in reality the system sensitivity will be degraded by this and a desired SNR may not be achieved even with long integration times. Another factor is the system Allen time. This, however, is more of a theoretical nature as modern digital systems typically have a very long Allen time.

Best regards,

Wolfgang

--

[b alex pettit jr]

Hello Wolfgang,

What sized system are you using ? 

 "We" tried this with my 1m a few years ago but with ambiguous results as the S7 calibration region is quite small 

Thanks,

Alex

================

[Tiago Baroni]

Hello Alex, hello Wolfgang,

Thanks for the helpful comments. Let me briefly clarify one point.

The RMS in my approach is not meant to represent a Doppler-shifted spectrum or to extract scientific information from individual sub-integrations. It is used purely as an operational sensitivity metric, computed after combining sub-integrations, baseline removal, and restricted to line-free regions, with the sole purpose of integration-time planning while the system remains approximately thermal.

Alex’s point about oversampling relative to the beamwidth is absolutely valid for sky mapping and independent spatial sampling. My use of short sub-integrations is instead for quality control (RFI rejection, gain stability), with RMS evaluated only after their combination.

Wolfgang’s remarks align well with this view: baseline RMS and the radiometer equation are useful planning tools, but in practice residual RFI, gain drifts, and other systematics quickly set an operational floor where further integration becomes inefficient.

As a beginner, my goal is mainly to learn how to identify this transition, rather than assuming that longer integration always helps. I appreciate the feedback and am happy to hear further suggestions, especially regarding common pitfalls in small SDR-based systems.

Best regards,

Tiago Baroni

[b alex pettit jr]

Hello Tiago,

FYI, these are plots of the same data 

Power Sum of each sample

Doppler Spectrum of each sample  ( FFT = 512 freqs )

Regards,

Alex Pettit

[fasleitung3]

We are using this for all our telescopes. It is a bit more complex for smaller dishes as the S7 region does not extend over the sky area covered by smaller dishes. In this case we use the simulation tool provided at https://www.astro.uni-bonn.de/hisurvey/euhou/LABprofile/ to generate the refernce spectrum for the appropiate beam size. The agreement between this method and a hot/cold calibration is typically +/- 15%.

Best regards,

Wolfgang

--
--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sara-list/632979107.337547.1768471526282%40mail.yahoo.com.

[b alex pettit jr]

Hello Wolfgang,

Excellent. I will review my data  ... 

From my ( small dish ) experiments, most any region within the Dec +30-+60 Dg & RA11-13 hr time period has low H Line energy .

Thanks,

Alex P

[Tiago Baroni]

Hello Alex, hello Wolfgang,

Thanks again for the comments and figures. They help a lot in connecting “on-paper” planning with real observing practice, especially for small systems.

Based on your experience, I would like to ask a few technical questions to better understand how RMS-based planning behaves in real small-dish / SDR setups:

(1) For small dishes and SDRs, when the 1/τ​ behavior breaks down, do you more often see baseline effects (standing waves, ripple) or temporal gain instabilities (Allan time / drift) dominating the practical noise floor?

(2) Regarding the use of “line-poor” sky regions (e.g. RA ~11–13 h, Dec ~+30–+60° as mentioned by Alex), do you typically treat them only as qualitative reference fields, or do you also use them quantitatively to estimate RMS/Tsys for planning? Are there specific caveats when extrapolating that RMS to other sky regions?

(3) On calibration: when regions like S7 are not well suited for small dishes, do you consider beam-matched reference spectra from simulations sufficient for sensitivity planning (even with ~10–20% uncertainty), or do you still prefer experimental methods (hot/cold, loads) whenever feasible?

(4) Finally, for integration-time planning, do you find a purely spectral metric (line-free RMS after baseline) more robust, or can integrated scalar metrics (e.g. power sum) be useful to detect diminishing returns earlier? Or does this depend strongly on backend and system stability?

The goal behind these questions is to better understand where RMS-based planning works well in practice and where it needs to be complemented by additional observational or instrumental criteria.

Many thanks in advance for any comments or practical examples you may be willing to share.

Best regards,

Tiago Baroni.

[b alex pettit jr]

Hello Tiago,

I defer to Wolfgang for most of these answers.

I have not been concerned with absolute measurements but optimizing the performance of small dish  systems

using a spectral analysis and comparing off HLine Cold_Sky vs Peak HLine amplitudes

I ran this analysis centered on the Dec +45   RA 12:00 region using the webpage Wolfgang referenced and

obtained highly consistent data from several synthesized beam-widths

This suggests it would be a useful and consistent  "Cold Sky"  Total Power Reference  for small systems of wider than S7 beam widths.

=====================================================================

HOWEVER, although Total Power may theoretically seem an elegant way to begin radio astronomy.

You will potentially be dealing with background shifts whose levels greatly exceed even the highest H Line Signals . 

These can be handled with a Spectral Analysis,  not so with Total Power   ( Sun transit near end of drift scan also corrected )

Regards,

Alex Pettit

===============================================================================

[b alex pettit jr]

Hello Tiago,

=====================================================================

HOWEVER, although Total Power may theoretically seem an elegant way to begin radio astronomy.

You will potentially be dealing with background shifts whose levels greatly exceed even the highest H Line Signals . 

There IS another way, but it consumes data acquisition time for capturing corrections vs post test corrections

===================================================

===============================================================================

Alex Pettit

[Tiago Baroni]

Hello Alex,

Thanks for the figures and the detailed explanation, they make the background-drift issue in total-power measurements very clear. I fully agree that, in small systems, these variations can easily exceed the HI line amplitude and make total-power-based metrics unreliable for planning or diagnostics.

Just to be completely clear about my intent: the approach I am exploring does not rely on total-power metrics as a primary criterion. The RMS I refer to is strictly spectral, computed in line-free regions after sub-integration combination and baseline removal, precisely to separate background drift and other systematics from the spectral signal of interest.

In that sense, I see spectral analysis not merely as a processing step, but as the key mechanism for identifying when the system departs from an approximately thermal regime and becomes dominated by drift, ripple, or other instabilities, at which point further integration ceases to be efficient.

Your demonstration that regions such as Dec +45°, RA ~12 h remain stable on average across different beam widths is particularly useful, as it reinforces the idea of a practical “cold sky” reference for planning in small systems, without any claim of absolute calibration.

Thanks again for sharing these practical examples, they help a lot in aligning the conceptual model with real instrumental limitations.

Best regards,
Tiago Baroni

[fasleitung3]

(1) The main effect is baseline effects, i.e. the imperfect removal thereof. Gain instabilities do not play a role for spectral observations.

(2 and 3) I find no specific use for low intensity regions for calibration purposes. I always refer to the S7 region and in case of small dishes I use the synthezised spectrum matching the beam. Tsys, however, will vary with elevation depending on spillover and possibly some other contributions. So I can either measure Tsys using S7 as S7 crosses the sky over the day. Using S7 is so fast and convienient compared to hot/cold that this is the preferred method.

(4) Once I have Tsys as explained above I can just go ahead and plan integration times using the radiometer equation. In practice, we alwas split up the total integration time into several shorter recordings. This is to avoid that a whole long integration is spoiled by a short period of RFI.

Best regards,

Wolfgang

[Tiago Baroni]

Hello Alex, Wolfgang, and all,

Thank you again for the thoughtful discussion and for the practical examples shared in this thread. The points raised about total power drift, spectral vs scalar diagnostics, cold-sky references, and instrumental floor behavior were particularly helpful.

Based on that exchange, I took some time to formalize the approach into a small Python diagnostic tool focused strictly on spectral (line-free) RMS analysis and regime classification (thermal vs instrumental floor). It implements:
- Spectral RMS vs total power separation
- σ² = A/τ + σ_floor² modeling
- Drift detection
- Allan variance analysis
- Optional cold-sky reference comparison

The goal is not calibration, but sensitivity diagnostics and integration-time planning consistency checks.

For anyone interested, the repository is here:  https://github.com/tiagobaroni/hi-sensitivity-diagnostics 

It is still a beta tool and I would welcome any comments, corrections, or suggestions.

Thanks again for the discussion. It significantly improved the conceptual clarity of the implementation.

Best regards,
Tiago Baroni