How AI models (LLMs) handle RA data analysis - a simple experiment
Eduard Mol
Stephen Arbogast
Alfred Aburto
Yes, I asked Google A.I. what was the brightest crater on the Moon and it said Tycho, which was wrong. Asking later it came back with the right answer: Aristarchus ...
But, to be fair, Google warns that you won't always get the right answer, so, yes, be wary --- double check ...
Al
--
--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sara-list/22f9534c-c341-4ec8-9e27-72ee8581fa4an%40googlegroups.com.
Don Latham
From: 'Stephen Arbogast' via Society of Amateur Radio Astronomers <sara...@googlegroups.com>
To: Society of Amateur Radio Astronomers <sara...@googlegroups.com>
Date: Friday, 11 September 2026 8:16 PM MDT
Subject: [SARA] Re: How AI models (LLMs) handle RA data analysis - a simple experiment
--
Eduard Mol
To view this discussion visit https://groups.google.com/d/msgid/sara-list/924996881.198823992.1789247377622.JavaMail.zimbra%40blackfoot.net.
Stephen Arbogast
Pablo Lewin
Hi Ed,
Thanks for posting this. I think your experiment raises exactly the right question: not simply “can an LLM get the right answer?” but what actually happened to the data, can we audit it, and can the result be independently reproduced? Your four possibilities are a useful way of framing the issue, and your Google test appears to have shown a code-generation-and-execution workflow in that particular case.
One important update since our earlier discussion is that the American Astronomical Society (AAS) has now issued formal guidelines for AI/LLM use in AAS journals. The AAS does not impose a blanket prohibition on AI in scientific research or data processing. In fact, its new author guidelines explicitly say that they concern manuscript preparation rather than data analysis and therefore make no recommendation against the use of AI for data analysis or writing code. The guidelines allow supervised AI assistance for code, data presentation, plots and visualizations, provided the scientist verifies the results and discloses the use appropriately. What they explicitly prohibit is presenting AI-generated material as though it were real observational data, inventing data, manipulating values to change results, or using AI-generated scientific figures without checking them against the source data.
That distinction is actually very close to the way I have been conducting these experiments. I am not arguing that an LLM is an infallible astronomer or that it never hallucinates. My question has been whether AI can be used as a transparent reduction assistant when we constrain it to the actual FITS pixels, require intermediate outputs, independently check its conclusions, and allow it to reject data rather than forcing a desired answer.
A few results from my experiments are particularly relevant.
With Kepler-426 b, ChatGPT reduced the calibrated FITS pixels and obtained a transit depth of 15.10 ± 4.23 ppt. I then took the same calibrated pixels into HOPS, using the same principal comparison stars but an independent photometry and fitting system. HOPS obtained 19.32 −5.87/+5.96 ppt. The difference between the two central values is only about 0.58 combined sigma. The purpose was specifically cross-software validation, not proving either pipeline infallible.
There was another experiment that, to me, is even more interesting from the hallucination standpoint. For TIC 285592400.01, the run underwent a meridian flip. Frame 0037 was the last exposure whose astrometric solution verified the correct target field. After the flip, frames 0038 through 0097 formed a consistent field, but extensive registration and star-pattern tests could not demonstrate that it was the target field. The AI therefore rejected all of those frames instead of assuming where the target was and constructing an apparent egress. The resulting report explicitly says that those images cannot be used to claim an egress or post-transit baseline unless the field is independently re-solved.
That result mattered to me because if the objective were merely to “please Pablo” with a nice-looking transit, those post-flip frames were exactly where a system might have been tempted to manufacture one. It did the opposite.
My latest experiment, WASP-80 b / TIC 243921117.01, started with the raw observing directory rather than a pre-made light curve. The reduction isolated 50 R-band science frames, 10 biases, 10 matched 180-second darks and 10 R flats. It independently verified the target, performed ensemble differential photometry, tested aperture sizes and comparison stars, and produced the light curve. The measured depth near transit was about 3.75%, while the AIJ-style fitted/detrended result was 3.649% or 40.36 mmag. The published R-band depth is about 40.26 mmag. The observing run stopped before egress, and the report therefore specifically refuses to claim a precision full-duration or TTV result.
The AIJ-style graph was another deliberate test. I instructed the AI that the plotted observational points had to come from the FITS measurements and that the smooth green curve could only be a mathematical visualization of those measured points. It was not allowed to invent points beyond the end of the run or manufacture the missing egress. That is exactly the sort of distinction the new AAS guidance makes between legitimate AI-assisted visualization of an author's data and generating fictitious scientific data.
There have also been cases where AI did not give me the answer I might have wanted. With TOI 3514.01 / TIC 1870990135.01, there was a very close neighboring star and significant aperture dependence. The analysis found a real fading in the blended light but concluded that the dataset could not establish whether the fading belonged to the planet host, the nearby star, or another blended-source scenario. In other words, it returned “inconclusive” rather than “planet detected.”
I also agree with your later point that insufficiently specified prompts can cause an AI system to infer what the user intends and begin doing things that were never explicitly requested. That is why my prompts have evolved to include instructions such as: do not generate synthetic observational points; do not alter measurements to match an expected transit; keep measured data separate from model/reference data; reject frames whose field identity cannot be verified; and report an inconclusive result when the data do not support a conclusion.
I would be very interested in understanding your Google experiment in more detail, because I think it would help us compare apples with apples. Which AI did you actually use—Gemini, and if so which model/version? Was it the normal Gemini interface, Google AI Studio, Colab, or another environment? Did it invoke a separate coding/data-analysis agent or Python execution environment? What was your exact prompt?
Most importantly, when you say it made “quite a few errors,” what exactly were the errors? Were they Python/runtime errors, use of obsolete libraries, incorrect frequency units, incorrect hydrogen rest frequency, wrong Doppler sign convention or formula, incorrect peak identification, baseline removal, axis construction, or incorrect scientific interpretation of otherwise correctly processed numbers? Did you correct the errors and rerun the same analysis, and if so, what result did it ultimately produce?
I think that distinction is critical. A generated Python syntax error, a bad physical assumption, and a fabricated observational result are three very different failures.
My present view is not that AI replaces HOPS, AstroImageJ, PyTransit or any other established package. It is that AI can become another scientific tool when we force it to show its work, preserve the pixels, expose the code and intermediate products, independently cross-check the answer, and give it permission to say “these data are not usable.”
That is the experiment I am really interested in.
Pablo Lewin
AAVSO LPAC
The Maury Lewin Astronomical Observatory
Marcus D. Leech
Hi Ed,
Thanks for posting this. I think your experiment raises exactly the right question: not simply “can an LLM get the right answer?” but what actually happened to the data, can we audit it, and can the result be independently reproduced? Your four possibilities are a useful way of framing the issue, and your Google test appears to have shown a code-generation-and-execution workflow in that particular case.
One important update since our earlier discussion is that the American Astronomical Society (AAS) has now issued formal guidelines for AI/LLM use in AAS journals. The AAS does not impose a blanket prohibition on AI in scientific research or data processing. In fact, its new author guidelines explicitly say that they concern manuscript preparation rather than data analysis and therefore make no recommendation against the use of AI for data analysis or writing code. The guidelines allow supervised AI assistance for code, data presentation, plots and visualizations, provided the scientist verifies the results and discloses the use appropriately. What they explicitly prohibit is presenting AI-generated material as though it were real observational data, inventing data, manipulating values to change results, or using AI-generated scientific figures without checking them against the source data.
That distinction is actually very close to the way I have been conducting these experiments. I am not arguing that an LLM is an infallible astronomer or that it never hallucinates. My question has been whether AI can be used as a transparent reduction assistant when we constrain it to the actual FITS pixels, require intermediate outputs, independently check its conclusions, and allow it to reject data rather than forcing a desired answer.
A few results from my experiments are particularly relevant.
With Kepler-426 b, ChatGPT reduced the calibrated FITS pixels and obtained a transit depth of 15.10 ± 4.23 ppt. I then took the same calibrated pixels into HOPS, using the same principal comparison stars but an independent photometry and fitting system. HOPS obtained 19.32 −5.87/+5.96 ppt. The difference between the two central values is only about 0.58 combined sigma. The purpose was specifically cross-software validation, not proving either pipeline infallible.
There was another experiment that, to me, is even more interesting from the hallucination standpoint. For TIC 285592400.01, the run underwent a meridian flip. Frame 0037 was the last exposure whose astrometric solution verified the correct target field. After the flip, frames 0038 through 0097 formed a consistent field, but extensive registration and star-pattern tests could not demonstrate that it was the target field. The AI therefore rejected all of those frames instead of assuming where the target was and constructing an apparent egress. The resulting report explicitly says that those images cannot be used to claim an egress or post-transit baseline unless the field is independently re-solved.
That result mattered to me because if the objective were merely to “please Pablo” with a nice-looking transit, those post-flip frames were exactly where a system might have been tempted to manufacture one. It did the opposite.
My latest experiment, WASP-80 b / TIC 243921117.01, started with the raw observing directory rather than a pre-made light curve. The reduction isolated 50 R-band science frames, 10 biases, 10 matched 180-second darks and 10 R flats. It independently verified the target, performed ensemble differential photometry, tested aperture sizes and comparison stars, and produced the light curve. The measured depth near transit was about 3.75%, while the AIJ-style fitted/detrended result was 3.649% or 40.36 mmag. The published R-band depth is about 40.26 mmag. The observing run stopped before egress, and the report therefore specifically refuses to claim a precision full-duration or TTV result.
The AIJ-style graph was another deliberate test. I instructed the AI that the plotted observational points had to come from the FITS measurements and that the smooth green curve could only be a mathematical visualization of those measured points. It was not allowed to invent points beyond the end of the run or manufacture the missing egress. That is exactly the sort of distinction the new AAS guidance makes between legitimate AI-assisted visualization of an author's data and generating fictitious scientific data.
There have also been cases where AI did not give me the answer I might have wanted. With TOI 3514.01 / TIC 1870990135.01, there was a very close neighboring star and significant aperture dependence. The analysis found a real fading in the blended light but concluded that the dataset could not establish whether the fading belonged to the planet host, the nearby star, or another blended-source scenario. In other words, it returned “inconclusive” rather than “planet detected.”
I also agree with your later point that insufficiently specified prompts can cause an AI system to infer what the user intends and begin doing things that were never explicitly requested. That is why my prompts have evolved to include instructions such as: do not generate synthetic observational points; do not alter measurements to match an expected transit; keep measured data separate from model/reference data; reject frames whose field identity cannot be verified; and report an inconclusive result when the data do not support a conclusion.
I would be very interested in understanding your Google experiment in more detail, because I think it would help us compare apples with apples. Which AI did you actually use—Gemini, and if so which model/version? Was it the normal Gemini interface, Google AI Studio, Colab, or another environment? Did it invoke a separate coding/data-analysis agent or Python execution environment? What was your exact prompt?
Most importantly, when you say it made “quite a few errors,” what exactly were the errors? Were they Python/runtime errors, use of obsolete libraries, incorrect frequency units, incorrect hydrogen rest frequency, wrong Doppler sign convention or formula, incorrect peak identification, baseline removal, axis construction, or incorrect scientific interpretation of otherwise correctly processed numbers? Did you correct the errors and rerun the same analysis, and if so, what result did it ultimately produce?
I think that distinction is critical. A generated Python syntax error, a bad physical assumption, and a fabricated observational result are three very different failures.
My present view is not that AI replaces HOPS, AstroImageJ, PyTransit or any other established package. It is that AI can become another scientific tool when we force it to show its work, preserve the pixels, expose the code and intermediate products, independently cross-check the answer, and give it permission to say “these data are not usable.”
That is the experiment I am really interested in.
Pablo Lewin
AAVSO LPAC
The Maury Lewin Astronomical Observatory
The attached noisy image was captured with a 300msec exposure, through a F1.3 lens, with an equivalent focal length of about 480mm, with the "gain" (ISO equivalent)
turned up about half-way on the camera (and SV305C).
Last night, I decided that I'd ask ChatGPT to "improve" the image, using the original data, remove/reduce the sensor noise, remove the overall magenta cast, and improve the contrast on the dimmer stars.
It produced a result that was visually very impressive, and somewhat to my surprise, it had reasonable correspondence to the original image.
But then I asked it what technique it used. It revealed that it basically produced a synthetic image, using my pixels as a kind of "strict" template. A bit like asking
a realist painter from the 1960s to produce a painting from a kind of rough photo.
But when you look at tools like BlurXterminator that astro-photo people use, the delta between those tools and full "AI" is razor thin. BlurXterminator uses
neural networks and ML to try and "make astro images look like they're supposed to".
There will be ongoing philosophical debates about which of these techniques is "legit" and which are "faking it".
Pablo Lewin
One additional note for Ed and anyone else following this discussion who is a member of the AAVSO Exoplanet Section: we are going to have a dedicated discussion on this very subject at the next Exoplanet Section SIG meeting on Tuesday, September 22 at 7:00 PM Eastern / 2300 UTC.
Dennis Conti has organized several of us to give short presentations showing different ways astronomers are experimenting with AI in exoplanet work. I will be discussing my experience using ChatGPT directly on exoplanet FITS data, including the successful reductions, the HOPS cross-validation, the post-meridian-flip rejection case, and the latest WASP-80 b reduction.
George Iwaszek will discuss using AI to compare independent observations to evaluate consistency and investigate possible transit-timing variations. Bob Massey will talk about his own experience working with AI. Dennis Conti will present his work using Claude to develop a Python processing pipeline that interfaces with AstroImageJ.
Each of us is planning roughly a 15-minute presentation, followed by questions, and Dennis also intends to have an open discussion specifically about the hazards and limitations of using AI in astronomical analysis.
I think this is particularly valuable because it is not going to be four people simply saying, “AI is wonderful.” We are approaching it from different directions, using different AI systems, different workflows, and different levels of automation. We will be talking about what worked, what failed, what required human intervention, how the results were verified, and where the scientific risks are.
Ed, I would especially welcome you—and anyone else here who belongs to the AAVSO Exoplanet Section—to attend. Your hydrogen-line spectrum experiment is exactly the kind of example that would add something useful to the discussion. I would very much like to compare what you observed in Google’s system with what we have been seeing in ChatGPT, Claude, HOPS, and AstroImageJ-based workflows.
I am also going to ask Dennis whether he would be comfortable with me inviting a few interested non-AAVSO members to audit the session. I do not know whether that is appropriate under the Exoplanet Section’s meeting rules, so I am not going to assume that it is.
If Dennis says it is okay, I will post the Zoom link here for anyone who would like to listen in. If he prefers that the meeting remain limited to AAVSO members, I will respect that and will not post the link or pursue it further here.
Either way, I think the September 22 discussion should be worthwhile because it will include both the opportunities and the risks of AI-assisted exoplanet analysis, with several astronomers showing different approaches rather than presenting a single point of view.
Pablo Lewin
Pablo Lewin
Marcus,
That’s exactly the distinction I think we need to keep clear. We’re doing science-based research using the scientific method; we’re not asking AI to make astronomical images prettier or to make them look the way we think they ought to look.
For scientific work, the question isn’t whether an image is visually impressive. The question is whether the result is traceable to the original measurements, reproducible, quantitatively testable, and supported by evidence. If an AI system creates synthetic pixels or reconstructs features based on what it expects an astronomical image to contain, that may be perfectly useful for visualization, but it cannot automatically be treated as observational data.
The same principle applies to tools such as BlurXTerminator. Neural networks and machine learning are not inherently unscientific, but if they are used in research, their effects have to be characterized and validated against known data. The original data must remain available so the result can be independently checked.
So I don’t see this primarily as a philosophical debate about whether something is “fake.” In science, we have a much more practical test: Can the result be demonstrated, reproduced, independently verified, and shown not to introduce signals that weren’t present in the observations?
Science requires evidence. Conclusions based simply on appearance, preference, or emotion don’t meet that standard.
Pablo
Marcus D. Leech
Some AI now in use in Astronomy admittedly not the over the counter kind but AI nonetheless..that's why it is important to scientifically prove or disprove the us of AI dispassionately using the scientific method only
Pablo Lewin
But I think there's going to be some philosophical discussions as well. What is the nature of measured reality, etc, etc.
--
--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sara-list/5a28b955-100a-4bc4-90a9-70096e719784n%40googlegroups.com.
Marcus D. Leech
Marcus,
That’s exactly the distinction I think we need to keep clear. We’re doing science-based research using the scientific method; we’re not asking AI to make astronomical images prettier or to make them look the way we think they ought to look.
For scientific work, the question isn’t whether an image is visually impressive. The question is whether the result is traceable to the original measurements, reproducible, quantitatively testable, and supported by evidence. If an AI system creates synthetic pixels or reconstructs features based on what it expects an astronomical image to contain, that may be perfectly useful for visualization, but it cannot automatically be treated as observational data.
The same principle applies to tools such as BlurXTerminator. Neural networks and machine learning are not inherently unscientific, but if they are used in research, their effects have to be characterized and validated against known data. The original data must remain available so the result can be independently checked.
So I don’t see this primarily as a philosophical debate about whether something is “fake.” In science, we have a much more practical test: Can the result be demonstrated, reproduced, independently verified, and shown not to introduce signals that weren’t present in the observations?
Science requires evidence. Conclusions based simply on appearance, preference, or emotion don’t meet that standard.
Pablo
and bridges using AI tools that may be hallucinating results. That's the existential hazard that I worry about.
I did my little test to see what it would do with the original data. I haven't done an object-by-object comparison, but it could certainly fool people into thinking it was an original image.
--
--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sara-list/7e735992-d8fb-414a-bfa2-bf1abb10b24fn%40googlegroups.com.
Pablo Lewin
I reviewed the AI-generated analysis of Edward Mol's 21-cm neutral-hydrogen spectrum obtained with his 3-meter radio telescope in the Netherlands.
There are some useful elements in the analysis, but I believe there are also important scientific limitations that should be recognized before interpreting the reported spectral features as Galactic hydrogen velocities.
This is not meant as criticism of the observation itself. On the contrary, the data may be very interesting. My concern is with the reduction and interpretation methodology used by the AI.
------------------------------------------------------------
1. WHAT THE AI DID CORRECTLY
------------------------------------------------------------
The analysis correctly used the neutral-hydrogen 21-cm rest frequency of approximately:
1420.405751 MHz
It also used the standard first-order radio Doppler relationship:
v = c (f_rest - f_observed) / f_rest
where:
v = radial velocity
c = speed of light
f_rest = hydrogen rest frequency
f_observed = measured frequency
Using this convention:
A frequency above 1420.405751 MHz produces a negative velocity, conventionally corresponding to an approaching/blueshifted component.
A frequency below 1420.405751 MHz produces a positive velocity, corresponding to a receding/redshifted component.
Therefore the mathematical conversion between a measured frequency and a nominal Doppler velocity is straightforward.
However, obtaining a number in km/s is only the beginning of the astronomical analysis.
------------------------------------------------------------
2. THE MOST IMPORTANT PROBLEM
------------------------------------------------------------
The AI analysis did not appear to use:
The telescope's exact geographic location.
The exact UTC date and time of the observation.
The antenna pointing direction.
Azimuth and elevation.
Right ascension and declination.
Galactic longitude and latitude.
These quantities are essential if the objective is to derive a physically meaningful Galactic H I velocity.
Without them, the calculated velocity is essentially a raw telescope-centered, or topocentric, frequency-derived velocity.
It is NOT yet a properly corrected Galactic velocity.
------------------------------------------------------------
3. THE REFERENCE FRAME IS MISSING
------------------------------------------------------------
For Galactic neutral-hydrogen astronomy, velocity normally needs to be expressed in a clearly defined reference frame, commonly the Local Standard of Rest, or LSR/LSRK.
The telescope itself is moving.
Its motion includes:
Earth's rotation.
Earth's orbital motion around the Sun.
The Sun's motion relative to the Local Standard of Rest.
All of these motions have components projected along the particular direction in which the antenna is pointing.
Therefore:
Observed frequency
does NOT automatically equal
Galactic radial velocity.
To calculate the correction properly we need:
Exact observer location.
Exact UTC observing time.
Exact source/antenna pointing.
Only then can we calculate the topocentric-to-barycentric and/or LSR correction.
------------------------------------------------------------
4. A MAJOR INCONSISTENCY IN THE AI RESULTS
------------------------------------------------------------
The first AI analysis identified:
1422.753 MHz
as the "Primary Peak / Main Galactic Signal"
and calculated approximately:
-495.5 km/s
It also identified:
1418.130 MHz
corresponding to approximately:
+480.4 km/s
However, the later baseline-subtracted analysis identifies the actual H I peak as:
1420.5444 MHz
This corresponds to only approximately:
-29 km/s
These are completely different velocity regimes.
The difference is several hundred km/s.
The report does not explain why the supposed principal Galactic component changed from approximately -496 km/s to approximately -29 km/s.
That discrepancy needs to be resolved before any physical interpretation is made.
------------------------------------------------------------
5. THE +/- 500 KM/S FEATURES REQUIRE VERY CAREFUL VERIFICATION
------------------------------------------------------------
Features corresponding to approximately +480 and -495 km/s should not automatically be labeled Galactic H I simply because their frequencies can be mathematically converted into those velocities.
Possible alternative causes include:
Radio-frequency interference (RFI).
Receiver bandpass structure.
Local oscillator artifacts.
Image responses.
Internal electronic spurs.
Band-edge effects.
Incorrect frequency calibration.
Baseline-fitting artifacts.
Real astronomical high-velocity hydrogen can certainly exist, but an extreme velocity claim requires considerably more evidence than locating a peak in a spectrum.
At this stage I would describe these as:
"Candidate spectral features corresponding to nominal topocentric velocities near +/-500 km/s whose astronomical origin has not yet been established."
That is much safer scientifically than calling one of them the "main Galactic signal."
------------------------------------------------------------
6. THE BASELINE SUBTRACTION NEEDS IMPROVEMENT
------------------------------------------------------------
The supplied Python code excludes the frequency interval:
1420.45 to 1420.65 MHz
before fitting a polynomial baseline.
That means the algorithm assumes in advance that the H I emission is located only within that frequency range.
This is risky.
Galactic hydrogen can occupy a much broader velocity interval, depending on where the telescope is pointed.
There may also be multiple H I velocity components.
If real hydrogen emission exists outside the predefined mask, the polynomial baseline may fit through actual astronomical signal and subtract some of it.
A better procedure would be:
First identify RFI.
Determine truly line-free regions.
Fit the instrumental baseline using those regions.
Try more than one reasonable baseline model.
Check whether the inferred H I parameters change significantly when the baseline method changes.
If the answer changes strongly with the baseline model, that uncertainty must be reported.
------------------------------------------------------------
7. RFI ANALYSIS IS MISSING
------------------------------------------------------------
No convincing radio-frequency-interference analysis was shown.
This is essential.
An AI should ask:
Does the signal remain at exactly the same receiver frequency over time?
Is it extremely narrow?
Does it appear intermittently?
Does it remain when the antenna points elsewhere?
Does it repeat when the same astronomical position returns at the same sidereal time?
Is it associated with a known receiver spur?
Does it appear in an off-source spectrum?
A terrestrial signal often remains fixed in receiver frequency.
A genuine Galactic H I feature may change depending on the direction in the sky being observed.
Without an RFI investigation, a frequency peak cannot automatically be classified as neutral hydrogen.
------------------------------------------------------------
8. FREQUENCY CALIBRATION IS MISSING
------------------------------------------------------------
For Doppler spectroscopy we must know how accurate the receiver's frequency scale is.
Important questions include:
What SDR or spectrometer was used?
What frequency reference was used?
Standard crystal oscillator?
TCXO?
GPS-disciplined oscillator?
Rubidium standard?
Was the receiver frequency calibrated?
How much frequency drift occurs during an observation?
A frequency error becomes a velocity error.
The AI cannot report a highly precise astronomical velocity unless the accuracy of the frequency axis is known.
------------------------------------------------------------
9. SPECTRAL RESOLUTION IS MISSING
------------------------------------------------------------
The report should specify:
Total observing bandwidth.
Number of channels.
Channel spacing.
FFT size.
Window function.
Effective spectral resolution.
Integration time.
Number of averages.
For example, if one frequency channel corresponds to several km/s, reporting a velocity to 0.1 km/s would imply a level of precision that the instrument may not actually provide.
Computer-generated decimal places are not the same as measurement accuracy.
------------------------------------------------------------
10. NO ERROR BARS OR UNCERTAINTIES ARE GIVEN
------------------------------------------------------------
A scientific result should not simply say:
v = -29.3 km/s
It should ideally say something similar to:
v = -29.3 +/- X km/s statistical +/- Y km/s systematic
Possible uncertainty sources include:
Channel spacing.
Signal-to-noise ratio.
Frequency calibration.
Baseline fitting.
RFI contamination.
Peak-fitting procedure.
Reference-frame correction.
All of these need to be considered.
------------------------------------------------------------
11. PEAK FREQUENCY ALONE IS NOT ENOUGH
------------------------------------------------------------
Selecting the single highest spectral channel is not a complete H I analysis.
For each credible hydrogen component I would calculate at least:
Peak velocity.
Centroid velocity.
Full Width at Half Maximum (FWHM).
Integrated line intensity.
Velocity dispersion.
Noise RMS.
Peak signal-to-noise ratio.
Integrated signal-to-noise ratio.
Uncertainty in each measured quantity.
The velocity centroid can be particularly useful because a broad or asymmetric line may have a highest point that does not represent the overall velocity of the gas.
------------------------------------------------------------
12. MULTIPLE H I COMPONENTS MAY BE PRESENT
------------------------------------------------------------
A Galactic H I spectrum frequently contains more than one velocity component.
Different hydrogen clouds and different portions of the Galactic disk may lie along the same line of sight.
The spectrum should therefore be investigated for multiple components.
Gaussian decomposition can be useful, but it should be performed carefully.
An AI should not automatically claim that every fitted Gaussian represents one individual physical cloud.
The decomposition is a mathematical description first; physical interpretation comes afterward.
------------------------------------------------------------
13. "RELATIVE ANTENNA TEMPERATURE" MAY BE AN INCORRECT LABEL
------------------------------------------------------------
The supplied plot uses wording similar to:
"Relative Antenna Temp / Amplitude"
However, the demonstrated processing does not show a calibration that converts arbitrary receiver amplitude into antenna temperature.
Unless a suitable calibration was performed, the vertical axis should be described as something such as:
Relative power.
Arbitrary units.
Baseline-subtracted spectral amplitude.
Calling the result antenna temperature implies a physical calibration that has not been demonstrated.
------------------------------------------------------------
14. PROFESSIONAL H I SURVEY COMPARISON IS NEEDED
------------------------------------------------------------
One of the strongest tests would be to compare Edward's spectrum with a professional H I survey covering the same Galactic coordinates.
For example:
HI4PI.
EBHIS.
LAB H I survey.
Edward's 3-meter telescope has a much larger beam than the professional surveys, so ideally the professional data should be averaged or smoothed to approximately the same angular resolution.
Then compare:
Peak velocities.
Line shapes.
Broad velocity components.
Whether the proposed +/-500 km/s signals exist at that sky position.
This would provide an independent reality check on the amateur observation.
------------------------------------------------------------
15. WHAT AN AI AGENT NEEDS TO PRODUCE A PROPER RESULT
------------------------------------------------------------
If an AI agent is going to perform the complete scientific analysis, I believe it should receive at minimum:
1. Original unmodified spectrum file.
2. Definition and units of every column.
3. Exact observatory latitude.
4. Exact observatory longitude.
5. Observatory altitude.
6. Exact observation date.
7. Exact UTC time or per-spectrum timestamps.
8. Antenna azimuth and elevation, OR RA and Dec.
9. Observing mode:
tracking, drift scan, fixed az/el, etc.
10. Telescope diameter.
11. Estimated or measured antenna beamwidth.
12. Feed design and polarization.
13. Receiver / LNA / SDR model.
14. Center frequency.
15. Total bandwidth.
16. Number of spectral channels.
17. Channel spacing.
18. Effective spectral resolution.
19. Integration time.
20. Number of integrations/averages.
21. FFT/window function.
22. Frequency-reference type.
23. Frequency-reference accuracy.
24. Receiver gain settings.
25. Whether AGC was enabled or disabled.
26. Filters used before the receiver.
27. Calibration information.
28. Off-source/reference observations, if available.
29. Known local RFI.
30. Whether amplitudes are linear power, logarithmic power, ADC counts, or calibrated temperature.
------------------------------------------------------------
16. WHAT THE AI SHOULD THEN DO
------------------------------------------------------------
A scientifically rigorous AI analysis should proceed in approximately this order:
STEP 1 - Preserve and verify the raw data.
Check for:
Missing rows.
Duplicated frequency channels.
NaN values.
Clipping.
Receiver saturation.
Corrupted data.
STEP 2 - Verify the frequency scale.
Determine:
Center frequency.
Bandwidth.
Channel spacing.
Frequency-reference accuracy.
STEP 3 - Determine the observing geometry.
Using:
Observatory coordinates.
UTC time.
Antenna pointing.
Calculate:
RA/Dec.
Galactic longitude and latitude.
Local sidereal time.
Projected observer velocity.
STEP 4 - Detect and mask RFI.
Identify:
Narrow spikes.
Persistent fixed-frequency signals.
Intermittent signals.
Receiver artifacts.
Bad channels.
STEP 5 - Determine the instrumental baseline.
Use genuinely line-free channels.
Do not assume in advance that the H I line exists only within one small frequency interval.
STEP 6 - Calculate the noise.
Measure RMS in line-free portions of the spectrum.
STEP 7 - Convert frequency to topocentric radio velocity.
Use the accepted neutral-hydrogen rest frequency.
Clearly label the result as:
TOPOCENTRIC RADIO VELOCITY
at this stage.
STEP 8 - Correct the Earth's motion.
Use:
Exact location.
Exact UTC time.
Exact pointing.
Calculate barycentric/heliocentric corrections.
STEP 9 - Convert to the Local Standard of Rest.
For Galactic H I analysis, provide an explicitly defined LSR/LSRK velocity.
STEP 10 - Measure the hydrogen profile.
For every statistically significant feature calculate:
Peak velocity.
Centroid.
FWHM.
Velocity dispersion.
Integrated intensity.
Noise RMS.
S/N.
Uncertainties.
STEP 11 - Test for multiple components.
Where appropriate, fit multiple components and objectively compare whether one, two, or more components are justified.
STEP 12 - Compare against professional H I surveys.
Retrieve an H I spectrum from the same Galactic coordinates and compare it with Edward's observation.
STEP 13 - Interpret the result.
Only after the previous steps should the AI begin discussing:
Galactic rotation.
Local hydrogen.
Spiral-arm structure.
High-velocity clouds.
Extragalactic gas.
Or other astrophysical interpretations.
------------------------------------------------------------
17. AN AI AGENT ALSO NEEDS "STOP CONDITIONS"
------------------------------------------------------------
This is extremely important.
The AI should not be forced to give an astrophysical answer when essential information is missing.
For example:
If pointing is unknown:
Do not identify the Galactic structure.
If observation time is unknown:
Do not claim an accurate barycentric or LSR correction.
If frequency calibration is unknown:
State that the absolute velocities are provisional.
If a feature occurs near a band edge:
Do not automatically classify it as astronomical.
If a feature is extremely narrow:
Investigate RFI first.
If professional survey data do not show the feature:
Treat it as suspicious until independently reproduced.
If different baseline models produce substantially different velocities:
Report the result as baseline-dependent.
If the signal-to-noise ratio is too low:
Report an upper limit rather than a detection.
This is essential if we want to use AI as a scientific analysis tool rather than simply as a system that generates a plausible-looking explanation.
------------------------------------------------------------
18. WHAT THE FINAL RESULT SHOULD LOOK LIKE
------------------------------------------------------------
A finished scientific report should include a table something like:
Telescope:
3-meter radio telescope
Observatory:
Exact Netherlands location and coordinates
Observation:
Exact date and UTC
Pointing:
Az/El and/or RA/Dec
Galactic coordinates:
l = ___ degrees
b = ___ degrees
Hydrogen rest frequency:
1420.405751 MHz
Channel spacing:
___ kHz
Velocity resolution:
___ km/s
Frequency-reference uncertainty:
___ Hz/kHz
Baseline method:
Clearly documented
RFI rejection:
Clearly documented
Peak topocentric velocity:
___ +/- ___ km/s
Peak LSRK velocity:
___ +/- ___ km/s
Centroid LSRK velocity:
___ +/- ___ km/s
FWHM:
___ +/- ___ km/s
Integrated intensity:
___ +/- ___
Peak S/N:
___
Professional survey comparison:
HI4PI / EBHIS / LAB result
Interpretation:
Only after all of the above measurements have been established.
------------------------------------------------------------
CONCLUSION
------------------------------------------------------------
My conclusion is that Edward's observation may contain perfectly valid and potentially interesting Galactic neutral-hydrogen emission.
The problem is not necessarily the observation.
The problem is that the AI interpretation went substantially farther than the demonstrated analysis supports.
The calculation:
frequency -> Doppler velocity
is easy.
The difficult scientific question is:
"What caused that spectral feature, what reference frame is the velocity measured in, where in the sky was the telescope looking, and does independent evidence confirm that the signal is actually Galactic H I?"
The later reported feature near 1420.5444 MHz, corresponding to approximately -29 km/s in the raw topocentric radio convention, appears much more reasonable as a conventional Galactic H I candidate than simply declaring the +/-500 km/s peaks to be Galactic.
However, even that -29 km/s value still requires the actual observing coordinates, UTC time, pointing, receiver characteristics, calibration information, RFI analysis, and LSR correction before it should be presented as the final astronomical velocity.
The next step should therefore not be another AI interpretation of a screenshot.
The next step should be to give the AI the ORIGINAL DATA together with the COMPLETE OBSERVING METADATA and require it to produce an auditable reduction from raw spectrum to final LSR-corrected H I measurements.
That would allow us to distinguish:
Measured quantities,
Calculated quantities,
Instrumental effects,
RFI,
and astrophysical interpretation.
That is the standard I believe we should aim for if AI is going to be used seriously in amateur/pro-am radio astronomy.
73,
Pablo Lewin
The Maury Lewin Astronomical Observatory
Minor Planet Center Observatory Code: U76