Diffusion models for assimilative ionospheric maps

[Andrew Rodland]

Hi,

I wrote a little bit about this in last year's "AI" thread and I've been meaning to follow up since then, but the thing has been constantly changing and it seems like every time I sit down to write a post, half of the details have changed before I get done writing. So, this time I'm going to try to keep it a bit briefer and see if it catches anyone's interest. If you have any questions please ask and I'll be happy to elaborate.

For years, the model behind prop.kc2g.com has been based on IRI + Gaussian process regression. IRI is the background / climate model, the differences between ionosonde observation and IRI prediction are filtered and extrapolated by the GP model, and those GP-filtered residuals are added back to IRI to make a final forecast. It's worked surprisingly well, but it isn't easily adaptable to other sorts of observations, so I've been looking for alternatives.

For about the past year I've been looking at diffusion models. They're the doodads behind the "AI art" generators you see nowadays, but they're actually quite usable for scientific purposes. Originally they were discovered as a funny special case of what happens if you take a smart, content-aware "noise removal" network and progressively apply it to an input containing nothing but noise: if you do it right, it generates things that look a lot like the images you trained it on. More modern theory recognizes them as a class of algorithms for learning a differential equation (a "flow") that transforms one distribution into another, allowing for efficient sampling from high-dimensional, implicitly-defined distributions.

The model I've been working on is a 150M-parameter latent diffusion transformer operating on 361x181x3 or 361x181x4 images: 1°x1° gridded foF2+hmF2+MUF(+TEC), and conditioned on observations as well as the date, time, and approximate SSN. It's all built from scratch using the Huggingface diffusers library, with some help from Claude code. The backbone is essentially DiT-B/2 from the paper, and the VAE shares its architecture with TAESD (it originally *was* TAESD).

I went through several different iterations including a UNet-based version and several different attempts at using inpainting or image-space gradient descent to conform to observations, but when I switched from UNet to DiT I realized just how crazy powerful transformers and "attention" are, and that I could condition directly on observations.

Since there's no such thing as a coherent global ionospheric map to use for training data I've been using synthetic training data generated by IRI2020 and then augmented/"perturbed" in various ways to allow the model to generalize. In particular once I extended the model to forecasting rather than nowcasting, I had to somehow introduce some dynamics to the training set for the model to learn, which has been a huge challenge. I've been able to cobble something together by sampling from the very same GP kernels my previous model uses — it's imperfect, but it works. I'd be thrilled to talk to anyone who has ideas about improving it.

I had a version of the model running in February/March that compared favorably to IRTAM, and was comparable to or better than my GP model, in validation against GNSS-RO, which I think is pretty cool. 

For the past month or so I've been working on adding TEC as one of the model channels, and adding assimilation of NOAA's GloTEC maps, to repair the gap left by the loss of so many North American ionosondes. I got it up and running and so far the results have been a little disappointing — the TEC-aware version of the model is underperforming, both with and without GloTEC inputs. But I've got a few tweaks that I'm hoping will smooth the issues out there and get me better results in the near future. The current training run is version 42, so with any luck it will contain the answer to life, the universe, and everything.

And if I get that working... conditioning on WSPR or PSKReporter spots is a definite possibility in the future (and one that has a lot of HamSCI flavor to it). That's the amazing thing about transformers: a new kind of observation is just a new token for the patches to attend to and hopefully extract some information from. It brings a whole ton of new challenges to the training process, but I think they're manageable ones, with enough effort, and then we could have a truly crowdsourced global ionospheric picture.

73,

KC2G

[Jon Abel]

That's interesting.  Pictures attached. 

 I'm attempting to design & build an electrostatic device to sense charge waves & Alfven waves in the ionosphere & Van Allen belts - more of a ES sensor than a EM receiver.   The patent I am using is 512,340 for an interleaved conical coil, and then I will be putting a resonant frequency across it to create an electric field.   The idea is to try to collect perturbations within the electric field and measure them with a RSP Duo.   Since freqeuencies are so low in the Van Allen belts (.0017 Hertz), integration times will be 10 minutes per cycle (600 seconds).   The RSP Duo only goes down to 1 Khz, but if it is modulating the resonant frequency of the coil, it might work.

The RSPDuo also has a High-Z input, since the coil is not 50 ohms.  The interleaved coil (when wires are attached in series) is 300 ohms total.   

Analyze patent 512,340 - it might give you some ideas.     

I'm also in the process of purchasing a Rigol DG-1032Z signal generator (micro-hertz resolution) & RSP Duo since they both accept a external clock (10 Mhz & 24 Mhz).  Since my dual external clock is 10 Mhz, I have to purchase probably a SI570 board to get it up to 24Mhz ($60).   I will also need a low noise MOSFET driver for the coil, which I am currently researching.   

  

--
Please follow the HamSCI Community Participation Guidelines at http://hamsci.org/hamsci-community-participation-guidelines.
---
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hamsci/8939db43-ef79-41e7-82e2-87dbcbcaa751n%40googlegroups.com.

[Architect of the Risk Flux Framework]

Hi Andrew,

The attached MP4 visualizes the quaternion norm as a function of time and normalized elevation, serving as a diagnostic tool for ionospheric conditions affecting the GPS-LEO pairs. The ionospheric risk is inferred from the magnitude and variability of this norm, with higher and more erratic values indicating greater potential for signal degradation. For a comprehensive risk assessment,  I need to correlate these peaks with external data (e.g., geomagnetic indices like Kp) and refine the model to predict risk levels quantitatively.

This plot is part of an analysis of ionospheric effects on GPS signals, using data from a specific LEO satellite and a GPS satellite. The quaternion approach used to model the combined effects of Total Electron Content (TEC), scintillation (S4), Radio Frequency Interference (RFI), and elevation angle, providing a compact representation of signal integrity. 

The 3D surface shows how the quaternion norm varies as a function of time and normalized elevation. The height and shape of the surface indicate the magnitude of the quaternion norm, with peaks and valleys reflecting changes in the ionospheric conditions or signal properties. 

I define Ionospheric Risk as:   Ionospheric risk refers to the potential impact of ionospheric disturbances on satellite-based navigation and communication systems, such as GPS. 

The ionosphere, a layer of Earth's atmosphere (approximately 50-1000 km altitude), contains charged particles that can affect radio signals. Here's how it relates to the plot and the data: 

• Indicators of Ionospheric Risk: 
• Total Electron Content (TEC): High TEC values (e.g., up to 991.50 in the dataset) indicate a dense ionosphere, which can cause signal delays and errors in GPS positioning. 
• Scintillation (S4): The S4 index (up to 0.98284) measures amplitude scintillation, where signal strength fluctuates due to ionospheric irregularities. Higher values suggest increased risk of signal fading. 
• Radio Frequency Interference (RFI): Extreme RFI values (up to 4.9278717) could indicate man-made or natural interference, adding to the risk. 
• Elevation Angle: Low elevation angles (e.g., -22.8555738°) increase the path length through the ionosphere, amplifying these effects. 

Axes and Variables: 

• X-axis (Time, ( t )): This axis represents time, labeled with dates in the format "YYYY-MM-DD" (e.g., "2024-04-07" based on the data). It shows the temporal evolution of the data over a specific period (in this case, April 7, 2024, from 00:15:00 to 20:55:00).  
• Y-axis (Normalized Elevation): This axis represents the normalized elevation angle of the GPS-LEO (Low Earth Orbit) satellite pair. Normalization scales the elevation (originally in degrees or radians) to a range between 0 and 1, based on the minimum and maximum values in the dataset. It indicates the angle of the signal path relative to the horizon. 
• Z-axis (Quaternion Norm): This axis represents the magnitude (norm) of a quaternion. In this context, the quaternion is derived from normalized ionospheric measurements (TEC, S4, RFI, and elevation) from the GPS-LEO pair. 

Risk Assessment: We are trying to identify vulnerable(high ionospheric risk)  time periods and elevation angles, aiding in mitigating risks by adjusting satellite operations or improving error correction algorithms. 

I am currently doing a write-up.  If anyone has any questions regarding data and stuff, let me know  if you'd like to dive deeper into any aspect

All feedback is welcome

Warm regard

Lawrence Habahbeh

Chair, Black Swan Insurance Working Party | Member, IFoA Risk Management Board 

Architect of the Risk Flux Framework

--

[Steve Kaeppler]

Dear Andrew-

What you described sounds very interesting! When do you plan to
publish this? I am actually not kidding when I say this, it sounds
like the makings of a nice JGR article.

Second, have you considered getting TEC data from the Madrigal
database? Maybe you already are?

73,
Steve
AD0AE

> --
> Please follow the HamSCI Community Participation Guidelines at http://hamsci.org/hamsci-community-participation-guidelines.
> ---
> You received this message because you are subscribed to the Google Groups "HamSCI" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/hamsci/8939db43-ef79-41e7-82e2-87dbcbcaa751n%40googlegroups.com.

--

--------------------------------------------------------------------
Steve Kaeppler
steve.k...@gmail.com
cell: 262-707-0932
Amateur Radio Call: AD0AE/6

[Andrew Rodland]

Thanks, Steve.

As for publication: it might happen, if I can get some solid results. There's also a possibility I'll be bringing this to CEDAR as a poster.

As for Madrigal data: I spoke to Phil W1PJE a bit about that and he said that what I got from there would likely not be real-time enough to be much use for forecasts, which is why I've been working with GloTEC. They're hooked into some of the GNSS networks, they have their own assimilative model (informed by GNSS TEC where it's available, and returning to IRI where there are no observations), and they publish runs at 15-30 minutes behind realtime, which is pretty good for my purposes.

Of course, if I can't make it work with that then I may be looking at other options :)

73,

Andrew

You received this message because you are subscribed to a topic in the Google Groups "HamSCI" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hamsci/uMBgJuS8_B0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hamsci+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hamsci/CANK6cPr71KGPvcxbzM0LXvNv2yGqnfHbd0GYhRkqwF-3pB7HMg%40mail.gmail.com.

[Architect of the Risk Flux Framework]

The attached MP4 visualizes the quaternion norm as a function of time and normalized elevation, serving as a diagnostic tool for ionospheric conditions affecting the GPS-LEO pairs. The ionospheric risk is inferred from the magnitude and variability of this norm, with higher and more erratic values indicating greater potential for signal degradation. For a comprehensive risk assessment,  I need to correlate these peaks with external data (e.g., geomagnetic indices like Kp) and refine the model to predict risk levels quantitatively.

This plot is part of an analysis of ionospheric effects on GPS signals, using data from a specific LEO satellite and a GPS satellite. The quaternion approach used to model the combined effects of Total Electron Content (TEC), scintillation (S4), Radio Frequency Interference (RFI), and elevation angle, providing a compact representation of signal integrity. 

The 3D surface shows how the quaternion norm varies as a function of time and normalized elevation. The height and shape of the surface indicate the magnitude of the quaternion norm, with peaks and valleys reflecting changes in the ionospheric conditions or signal properties. 

I define Ionospheric Risk as:   Ionospheric risk refers to the potential impact of ionospheric disturbances on satellite-based navigation and communication systems, such as GPS. 

The ionosphere, a layer of Earth's atmosphere (approximately 50-1000 km altitude), contains charged particles that can affect radio signals. Here's how it relates to the plot and the data: 

• Indicators of Ionospheric Risk: 
• Total Electron Content (TEC): High TEC values (e.g., up to 991.50 in the dataset) indicate a dense ionosphere, which can cause signal delays and errors in GPS positioning. 
• Scintillation (S4): The S4 index (up to 0.98284) measures amplitude scintillation, where signal strength fluctuates due to ionospheric irregularities. Higher values suggest increased risk of signal fading. 
• Radio Frequency Interference (RFI): Extreme RFI values (up to 4.9278717) could indicate man-made or natural interference, adding to the risk. 
• Elevation Angle: Low elevation angles (e.g., -22.8555738°) increase the path length through the ionosphere, amplifying these effects. 

Axes and Variables: 

• X-axis (Time, ( t )): This axis represents time, labeled with dates in the format "YYYY-MM-DD" (e.g., "2024-04-07" based on the data). It shows the temporal evolution of the data over a specific period (in this case, April 7, 2024, from 00:15:00 to 20:55:00).  
• Y-axis (Normalized Elevation): This axis represents the normalized elevation angle of the GPS-LEO (Low Earth Orbit) satellite pair. Normalization scales the elevation (originally in degrees or radians) to a range between 0 and 1, based on the minimum and maximum values in the dataset. It indicates the angle of the signal path relative to the horizon. 
• Z-axis (Quaternion Norm): This axis represents the magnitude (norm) of a quaternion. In this context, the quaternion is derived from normalized ionospheric measurements (TEC, S4, RFI, and elevation) from the GPS-LEO pair. 

Risk Assessment: We are trying to identify vulnerable(high ionospheric risk)  time periods and elevation angles, aiding in mitigating risks by adjusting satellite operations or improving error correction algorithms. 

I am currently doing a write-up.  If anyone has any questions regarding data and stuff, let me know  if you'd like to dive deeper into any aspect

All feedback is welcome

Warm regards,

Lawrence Habahbeh

Chair, Black Swan Insurance Working Party | Member, IFoA Risk Management Board 

Architect of the Risk Flux Framework

To view this discussion visit https://groups.google.com/d/msgid/hamsci/CABFQKmO1nzYZQ7SJjnmFWXHFpSwvqbL%2BxGKnyMEMv75gRp3-wA%40mail.gmail.com.

[Jon Abel]

This is AI, he just sent this email twice.

To view this discussion visit https://groups.google.com/d/msgid/hamsci/ZFtsJvV36yJo2EuKhX4xxjXd5Xm9yt8TO4a-tenVG1aRA3iubCp5Bhx521gFOl3sW-9sCdtCWopNO0W_USCUuCzC0jgnC-LjjPcg__YTlnU%3D%40pm.me.

[Architect of the Risk Flux Framework]

This is a sweeping accustaion.

 

is that you ?Volunteer Jon Abel banned from NDSU for Stanley Meyers lab research

Warm regards,

Lawrence Habahbeh

Chair, Black Swan Insurance Working Party | Member, IFoA Risk Management Board 

Architect of the Risk Flux Framework

To view this discussion visit https://groups.google.com/d/msgid/hamsci/CAGut4k0f37ZxgCNRe%3DmkSD6UzAVO79uPJ%2B2SMaK2LLngg3nRJg%40mail.gmail.com.

[Jon Abel]

Lots of Jon Abel's in the world.

To view this discussion visit https://groups.google.com/d/msgid/hamsci/Npb-LkvxC8xNX5AkBNhLkoLblu8RWM92vtx_iNP3rFVnCXiegWghM-_VUmw6_sgyxwnfpNQADs-F3X0vajJtUl_ppLLmmU1JC__GagtaFFs%3D%40pm.me.

[Jon Abel]

Only an AI would make that type of assumption.