Non-metric NOAA file?

[Paul Dunphy]

I am tinkering with a program that analyzes the NOAA data looking for trends in climate. All I have to work with are my NOAA files, which are metric. The units are specified in the header lines of these files.

Would one of the developers in the US be kind enough to post one of your monthly NOAA files here or send it to me by email? The files of names like "NOAA-2022-04.txt" are the ones I'm using. Any file will do as long as it is not metric. I do not care which month.

Thanks, Paul VE1DX

[Tom Keffer]

I've got 15 years worth. Here's this month's: https://www.threefools.org/weewx/NOAA/NOAA-2022-08.txt

Just substitute year and month for others.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/ff115426-41ff-4f0b-a36f-e6df8cfc4c45n%40googlegroups.com.

[Paul Dunphy]

Thanks, Tom (and Vince.)  That's all I need. If I get this thing working, I'll post the GitHub URL here. It would be interesting to see it with 15 years of data.  I only have 7, and that's not enough to rule out inter-annual variability. I don't think it's worth turning loose on the general user base, but we'll see.

- Paul VE1DX

[Paul Dunphy]

(Sorry if this is a duplicate post.  Our Internet is iffy due to Hurricane Fiona)

About a month ago, I asked for an example of the weewx US NOAA data format. First, both Tom and Vince responded, and then Vince was kind enough to send me his entire set of NOAA files.

Since then, I have been developing a Python program to look for trends in the weather. Unfortunately, I only have 7 years of my own data, but Vince had data back to November 2006 sans precipitation until 2009. Combined, this was enough to develop wee_trend. It's completed, and here is the URL:

https://github.com/ve1dx/wee_trend

My goal was to learn Python and how to use git and GitHub. Thanks to much coaching from my son, I believe wee_trend is ready to release to you developers. However, it's not likely appropriate for the general user base because it's only coded for use with Python 3.5 and above.  

Also, The results may not be statistically significant. The rule of thumb is 20 years of data to look for climate signals. Otherwise, we have just sampled some inter-annual variability. Climatology experts say we need 30 or 50 years to be sure, with 20 years considered a bare minimum. Still, weewx users may wish to look for trends within their data.

I tested it on Ubuntu 22.04, MAC OS X (macOS Monterey 12.6), and a Raspberry Pi 4 running the Arm64 port of Debian GNU/Linux 11. For interest sake, here are the relative performance metrics:

Ubuntu : (Intel i7-4790K CPU @ 4.00GHz) = 9 seconds.
Mac: (3.6 GHz 10-Core Intel Core i9) = 10 seconds.
Pi 4: (Cortex-A72 1.5GHz) = 43 seconds.

You guys might want to try it on your own NOAA data. I opted not to read the weewx.conf file because I could obtain the unit information from the NOAA files. This has the added advantage of allowing it to be run on a non-weewx system as long as it contains NOAA files.

I'd appreciate feedback, either as posts here or issues on GitHub. Go easy on me, as this is my first "toe in the water" with modern programming after a career in 3GL!

[Vince Skahan]

Lets see some example plots !!!!

(and good luck with Fiona)

[Paul Dunphy]

I believe I accidentally replied to Vince only.  Here's what it generates.  Well, 120 of them if you run it in batch mode.  Many are irrelevant because of the geographical location, and most are just "noise" because there isn't enough data. However, they are "interesting."  I'd be more interested in feedback if the download/install works on other systems in addition to the ones I tried.

- Paul VE1DX

[rcst...@gmail.com]

I had to install liblapack-dev (and liblapack3) and libatlas3-base-dev on my Pi to get this to work. (just sudo apt-get install liblapack-dev libatlas3-base-dev)

Took my Pi4 46 seconds to process everything, though for some reason it dropped 38 of the months claiming missing days... also tried running months that don't exist yet (Working on month 2022-12; Tolerance for missing days is 0; Data for 2022-12 incomplete, dropping it). =)

Any idea why it thinks some days are missing in the reports?

[Paul Dunphy]

Thanks for testing and the feedback. It can't process 2022-12 because we have no data for that month. The tolerance set to 0 will drop any month with even 1 day missing. 0 is the default. If you re-run it with -t 10 or the like, it will process incomplete months, and the 38 dropped months should be less.

The output of interest is the plots. If you see plots with a data point after the current month, that's an impossible situation, which means there is a defect in my program.

The wording of the verbose messages may be confusing. Maybe it should say "Inspecting" instead of "Processing," and I will look into that.

The processing time of 46 seconds you see on a Pi 4 is consistent with mine (43 seconds.) It will always generate 120 plots, but the number of months it has to work on will depend on how many years of data you have.

[Paul Dunphy]

I reworded the diagnostic messages.  Suggest you uninstall wee_trend (pip uninstall wee_trend), download the new version, and re-install it. Run it with 'wee_trend -V 1' to give a verbose set of diagnostics.  The reason for the dropped months will then become more obvious. You can adjust the tolerance with the -t parameter to make it process incomplete months with up to 25 days of missing data, but this quality control issue requires the user's judgment. Interpreting the results is beyond the scope of wee_trend.

[rcst...@gmail.com]

Thanks Paul. Just did that, and indeed, message is more verbose, but might still need some clarity

For example: 
Processing month December
Inspecting month 2016-12

Tolerance for missing days is 0

2016-12 has 27 days of complete data
Data for 2016-12 incomplete, dropping it

But looking at my NOAA report: https://www.staze.org/weewx/tabular.html?report=NOAA/NOAA-2016-12.txt

It's missing some wind data, I assume because my anemometer must have been encased in ice/snow. Is the script dropping that whole month from ALL plots, or just the Wind plot (in this case)? If the former, then why? If the latter, then maybe adding some language about which data is incomplete and indicating it's just being dropping from that plot. Basically, every December is being flagged for that exact reason. 

Thanks, this is pretty cool. 

[Paul Dunphy]

Thanks for the feedback and testing the program with your data. It works as intended, but that doesn't mean it can't be better. Consider it between alpha and beta testing now.

I am dropping entire months if any column has more missing data points than allowed by the tolerance parameter. In your case, you could get around that by specifying a higher -t value. However, I think a better option would be for me to refactor the code to drop the bad columns with missing data.

There's nothing wrong with your temperature, precipitation, etc., data for that month. I'll re-work it to do as described above. Meanwhile, try it with a -t 4 or -t 5 and see if that keeps the month. It may take me a few days. I'll let you know when I have it ready.

[rcst...@gmail.com]

Awesome, thanks Paul! And please don’t consider anything as criticism. Just testing. =)

 

Thanks for your work on this, it’s pretty cool! I’m curious what we’ll see when Tom runs it with all his data. =P

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/d9f99aad-030d-4769-9be6-9b830f7be97dn%40googlegroups.com.

[Paul Dunphy]

It has been more than a "few days," but the issues identified have been addressed. If you are "upgrading" to this version (wee_trend 1.1.0,) you

have to do an uninstall and re-install as described in README.md.

There are a couple of items I want to mention. First, this program is not part of the weewx project and is not a supported wee_weeex utility. I'll try to fix any errors or omissions as best I can, but Tom et al. are not involved with wee_trend.

Second, most problems encountered are quality control, of which only the user is aware. wee_trend plots are "interesting," but few to no weewx users have 20+ years of NOAA data. Also, as amateur weather observers, we typically do not maintain strict, consistent industry standards with our hardware. We upgrade or change weather stations, move their locations occasionally, etc. For example, my weather station was lowered from the meteorological standard of 10 meters to 8.5 meters above ground in 2019. This created a bias toward lower wind speeds. wee_trend can't know this. It also can't know if a bird built a nest in your rain gauge for several months. These things may make the plots appear to show trends that are not statistically significant.

To paraphrase Tom, I wrote wee_trend during the summer of 2022 to learn a bit about Python/git/GitHub, and because it was hot and muggy in Nova Scotia. As before, it's available here:

https://github.com/ve1dx/wee_trend

- Paul VE1DX

[Chuck Rhode]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 28 Sep 2022 05:48:34 -0700 (PDT)
Paul Dunphy <foc...@gmail.com> wrote:

> The rule of thumb is 20 years of data to look for climate signals.

Yes, well....

20 samples is a rule of thumb I was taught in statistics class: You
can't talk about confidence in any summary figures such as the mean
(average) unless you have at least 20 samples. It's surprising how
often percentages are reported in the Media with two places of
accuracy on studies done with fewer than 20 samples.

The opinion that follows *is not* WeeWX related. I have hesitated
about posting it more than once though because I have no credentials
that entitle me to such an opinion, but I have decided *not to*
refrain further in this space at this time and go ahead and blurt out
my feelings about seeing climate trends.

I applaud the programming effort that examines trend in WeeWX data for
evidence of climate change. I, too, have looked at local temperature
variability. My work is with regard to cumulative growing-degree-day
models of insect development (yet another topic on which I have no
credentials to speak). Here is my attempt to forestall a
climate-change interpretation of my figures.

I am posting because I feel that readers of this thread — whether they
are moved or not — need to hear some political skepticism that goes a
little beyond statistical skepticism. In particular, the original
poster's concerns over station siting are my concerns, too.

o https://lacusveris.com/Phenology/models.shtml#Fig.%20H

- --
.. Be Seeing You,
.. Chuck Rhode, Sheboygan, WI, USA
.. Weather: http://LacusVeris.com/WX
.. 60° — Wind WNW 8 mph — Sky overcast. Light rain; mist.

-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQT+MY/5I/LMPSswTbVg2/xipKOWUgUCYz8NrQAKCRBg2/xipKOW
UjnZAJ9dszRC42dBNOOQKjozpqv29BNWLQCfZZdfFQd9uhkMuwHIQwv33WfHIRw=
=vaFk
-----END PGP SIGNATURE-----