Comparison of McIntosh vs NASA/NOAA/ISED Cycle 25 forecasts

[Frank Howell]

Scott McIntosh and I have completed our study of how accurate are his team's forecasts of SSN & SFI for Cycle 25 in comparison to the official ones released by the NASA/NOAA/ISED team. I described the solar physics behind the McIntosh team's forecasts in a previous Zoom presentation to the HamSCI group.

You can read the results here:

https://k4fmh.com/2025/08/26/progress-in-the-revolution-sunspot-cycle-forecast-accuracy-at-cycle-25-peak/

To acknowledge the start of American football season: the McIntosh team is in a statistically significant lead at half-time with the crowd of scientists fully engaged in the contest!

73,

Frank

K4FMH

[David Themens]

Hi,

I've mentioned it to Scott a few times, but it's a bit disingenuous to compare his forecast that was revised in 2023 to the consensus forecast from 2019, especially when he had a forecast that was generated at a similar point in time. If you instead compare the NASA 13-month smoothed SSN forecast from the time he did his 2023 paper's forecast, then you find that the NASA forecast is actually quite good (see below, I've only provided the median value, but they provide 75% and 95% confidence intervals as well). 

The fact is, once the terminator is known, even conventional forecasts do quite well. For example, here are all the NASA forecasts (in red, now of F10.7 since I happened to have that plot handy, but they're related linearly in their forecast approach), the terrible consensus forecast (in blue, which was never in agreement with NASA's operational forecasts), and the true values (black) up until when I made this figure back in 2023 (the last time I had this discussion).

As you'll note, the forecasts are generally quite good during the rising and lowering phase of the cycle, but tend to struggle in predicting exactly when the cycle transitions happen (how long solar min and solar max will be). Once you've entered the corresponding phase of the cycle things are pretty good.

It's great to see people validating these things and taking an interest, but comparisons should be done apples to apples. Either compare the earlier (2019) MacIntosh forecast to the consensus one or compare the 2023 MacIntosh forecast to the corresponding NASA operational forecast generated at that time, when both forecasts had the same information to work with.

Cheers,

David

--
Please follow the HamSCI Community Participation Guidelines at http://hamsci.org/hamsci-community-participation-guidelines.
---
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hamsci/56829968-ca92-4ec3-bb1c-a47b9ad1bb2bn%40googlegroups.com.

[Frank Howell]

Hi David,

Thank you for taking the time to comment. I see from your Univ of Birmingham page that you are trained in Physics and I in Statistics. This may explain the differences in your approach to comparing forecasts to ours. Scott did not mention your earlier communications with him as we prepared this paper so it must not have made much of an impact on his thinking. 

Here are my reactions to your thoughts and analyses. I am being detailed so that I am as clear as possible in my response. My intention is to facilitate a scholarly exchange rather than the conflictual debates that so often occur on lists. HamSCI has fortunately not had that culture!

a. We used well-accepted methods in statistics and econometrics for comparing the relative accuracy of two time series for a given horizon. This design fits the desired comparison of the first half of Cycle 25 rather than picking-and-choosing features of each as you have done to create your graphs.

b. I do not think our approach is disingenuous at all as you claim. Disingenuous means "not candid or sincere, typically by pretending that one knows less about something than one really does." Instead, it is likely because of your training in Physics and not statistics, you think we are intentionally "ignorant" about the topic. Instead, it may be because you are not aware of the accepted methods in statistics and econometrics. In other words, you suffer from being wholly in your own discipline's paradigm on the topic and cannot see the merits of methods and approaches from others ("boundary maintenance"). When I consulted with physicists and engineers over the years, I ran into this frequently. I characterize it as the burden of FFTs: everything looks like that problem! Not really, but you get my point. There can be multiple approaches to the same problem without either being legitimately judged as incorrect as they are presented.

c. The conventional methods for comparing the forecast accuracy of two time series goes back to the 1960s with Henri Theil's classic text, cited in our paper. What we were doing is a comparison for the approximate first-half of the Cycle 25 period of SSNs (and SFI). Thus, we chose those starting and ending points as shown in Figure 2 of our paper. We put them on the same "apples to apples" time horizon to compare their respective forecast errors. This favors neither team's forecast. 

You have ginned-up a different look at it and that is fine but it does not demonstrate that our conventional statistical approach is incorrect. You show an entire Cycle whereas we are focusing only on the first-half of Cycle 25. We do plan another study once Scott and I are satisfied (him more than me) that C25 is complete and the "second half" competition is complete for the full Cycle.

d. I taught data viz for a number of years and was awarded grants for computational hardware to pioneer some approaches for the US Dept of Agriculture during the 1990s. As I taught engineering (and a few physics) students who came to my classes back then (no one in their departments taught this subject), I emphasized that data viz is an inductive method (see John Tukey's pioneering work). It can allow one to legitimately see different things in a given visualization. Your graph, which you say is "generally quite good," is an inductive assessment, not one that formally compares the time series per se. This is what we have done here which, frankly, is a more rigorous comparison than an ad hoc graphic, subject to interpretation.

As Theil showed as far back as the mid-1960s (cited in our paper), a statistical comparison of the forecast errors of the two series should accompany any data viz of them. Even Tukey would say that. We did such by using the MAE and MAPE for the first half of Cycle 25. That is the stated scope of our paper. We stand by that approach. It is not disingenuous as you state.

e. We gave NASA/NOAA/ISED's published forecasts ("official" ones) the benefit of their "false start" and used their revised forecast. They published this without comment as they did for their original forecast. This is something highly counter to good science, not revealing methods and such, which we note in our RadCom papers. McIntosh's team has done this in accordance with sound scientific norms. His team also has a developing theory of why the rise and fall occurs whereas the NNI offers nothing, except a notation they compared some 60+ models sent to them (in our RadCom papers). So our comparisons are for a fixed term of the approximate first-half of Cycle 25 for SSN and SFI with an advantage given to the NNI team through their "false start" adjustment.

f. NNI had no knowledge of the McIntosh team's concept of the Terminator and the timing of the Terminator as an underlying force in shaping their forecast...at either their first edition or the revised six-month starting point version. So there is no basis for your suggestion to make both forecasts on some handicapped time horizon. This would render a "rigged comparison" that honors neither forecast. Our approximation of the first half is independent of how either team designed their forecast in terms of the year and month. You and others may dicker with our specific dates but they are the same "apples to apples" time horizon for both forecasts.

I want to thank you again for taking the time to comment on our paper. I openly encourage you to conduct your own comparison of the two forecasts using your thoughts expressed in your post to this list and publish them with sufficient documentation for all to see. This is what we have done. 

We are in agreement about the value of others validating these things. We just use an approach long accepted in statistics and econometrics as well as the newer citation analysis procedures ("bibliometrics") used to track change in science. I've used these methods to advise both NIH and NSF to identify promising areas to create grant announcements. You choose to use more ad hoc methods that you favor. Let's see more of them!

Best regards,

Frank

K4FMH

To view this discussion visit https://groups.google.com/d/msgid/hamsci/CACyAW55LCFiLEo1cV6GFmRBkirgK8rPROtOGkWSqyPx1F0g74g%40mail.gmail.com.

[David Themens]

Hi,

Please note that I didn't address your statistical approach or methods at all. I don't disagree with those. My point is that you are comparing a forecast that was generated in 2022/2023 to a forecast that was generated in 2019. My concern regarding whether it is disingenuous is because there is a McIntosh 2019 forecast that you could have compared to, just as there is a 2022/2023 forecast from NASA that you could have compared to. By not comparing forecasts generated at similar times in the solar cycle, the comparison cannot provide a controlled assessment of the performance of either approach since it is comparing forecasts that had very different amounts of information available to them. Conducting a model comparison under controlled, equivalent conditions is not a "rigged comparison"; it is the fundamental basis of the scientific method. 

I apologize if my wording implied nefarious intent or was more confrontational than it should have been, that wasn't my intention. Nor was my intention to undermine the statistical approach that you employ, as your statistical metrics are appropriate. My comment merely concerned the conditions under which the comparisons were completed. My comments here are the same comments I've made to Scott publicly in the past. I've made it clear in my past comments that the approach that Scott developed is excellent and I have a great deal of confidence in its performance after the terminator is observed, but the approach requires a prediction of the terminator for any forecasts prior to that point. A solar cycle prediction from his approach before the terminator is thereby under very different limitations from the one that would be generated after the terminator. Similar limitations underpin other approaches to solar cycle prediction, so a comparison of solar activity forecasts is best done with forecasts that are conducted at the same point in time. 

The point of my second plot was merely to show the entire prediction history from NASA, which is openly available, to show how these predictions refine over time and highlight how the performance of individual forecasts changes significantly depending on when the forecast was conducted. I could conduct a thorough statistical analysis, but my plot was only for illustration and was not to undermine your assessment approach. In fact, it would be very interesting to see your analysis applied to all of the NASA operational SSN forecasts that cover the time range that you looked at, if only to see how their performance changes as more information becomes available over a cycle or to compare to Scott's forecasts. Such a comparison would be very relevant and useful to many operational centers that need SSN forecasts or currently use the operational NASA forecasts. The NASA forecasts can be found at the following link if you'd like to add them to your analysis: https://www.nasa.gov/solar-cycle-progression-and-forecast/archived-forecast/ 

Cheers,

David

To view this discussion visit https://groups.google.com/d/msgid/hamsci/CAFz2rFjWU74ukeBj3YrN8R1qA_%2B40-26S5k-T3PH8q1bW6N91Q%40mail.gmail.com.

[Frank Howell]

Hi David,

No worries...just wanted to give a thorough response to your critical comments. Sure, you have a different scope in mind from the conventional statistical design to compare time series that we use. We look forward to seeing you implement your preferred design to compare the NNI to the McIntosh team's forecasts in a published format. It would help push the cause of better understanding the sunspot cycle that we hope to have contributed to in this paper.

Best regards,

Frank

K4FMH

To view this discussion visit https://groups.google.com/d/msgid/hamsci/CACyAW56m8DCJABy9T9eDWzFM_VcXEgRp7qzKRGa8jfMGKCWR%2Bg%40mail.gmail.com.

[john Campbell]

this has been one of the most thoroughly enjoyable discussions.  intellectually.    I have filed it.  not sure I understand it all…but I aspire.  

thank you to all ! - W3MHZ

<1742136470390.jpg>

The fact is, once the terminator is known, even conventional forecasts do quite well. For example, here are all the NASA forecasts (in red, now of F10.7 since I happened to have that plot handy, but they're related linearly in their forecast approach), the terrible consensus forecast (in blue, which was never in agreement with NASA's operational forecasts), and the true values (black) up until when I made this figure back in 2023 (the last time I had this discussion).

To view this discussion visit https://groups.google.com/d/msgid/hamsci/CAFz2rFhCaSLud_3VOtDKRSL6Y2fgXDzT-V%3DtAWttL3mFka5f%2BA%40mail.gmail.com.

[Frank Howell]

Hi John,

Thank you...as HamSCI is all about that, right? 

73,

Frank

K4FMH

To view this discussion visit https://groups.google.com/d/msgid/hamsci/D2D96BD8-4E10-4E0F-8FD9-EDD8076A77DE%40gmail.com.