Min/max value for a given interval

[michael.k...@gmx.at]

When I write an extension in python, can I get the minimum/maximum value for an obs_type and a given timespan and their related timestamp?

Something like:

maxValue, datetime = getMax("outTemp",  1726092000, 1726178400)

The min/max has to be obtained from the archive_day table, the archive table might not contain the exact max/min value, and also not the exact timestamp the value arrived.

[Tom Keffer]

The xtypes system can do this. See the section, XTypes API.

Your example would be:

results = weewx.xtypes.get_aggregate("outTemp", TimeSpan(1726092000, 1726178400), "max", dbmanager)

where "dbmanager" is an open database manager.  The results will be a ValueTuple.

Here's a more complete example:

import weecfg

import weewx.manager

import weewx.xtypes

from weeutil.weeutil import TimeSpan

config_path, config_dict = weecfg.read_config('/home/weewx/weewx.conf')

with weewx.manager.open_manager_with_config(config_dict, 'wx_binding') as dbmanager:

    results = weewx.xtypes.get_aggregate("outTemp", TimeSpan(1726092000, 1726178400), "max", dbmanager)

print(results)

(68.4, 'degree_F', 'group_temperature')

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/6d13cde1-43c4-4ca3-a4a4-d220fc5e2040n%40googlegroups.com.

[michael.k...@gmx.at]

Thank you Tom, this works for the max/min value as I need it. But the dateTime for that value isn't in the result and I don't have a clue, how to get it for that particular value.

[Tom Keffer]

Just substitute "mintime" / "maxtime" for "min" / "max".

See the table of Aggregation Types.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/2aaf12b7-25ad-4013-8185-b379168b4bc5n%40googlegroups.com.

[michael.k...@gmx.at]

Thanks! Although obviously there, I didn't get it :)

[michael.k...@gmx.at]

Is it possible, that this function returns different values, if the timespan is across more than a single calendar day?

results = weewx.xtypes.get_aggregate("outTemp", TimeSpan(1726264800, 1726335600), "min", dbmanager)

    print(results)

    results = weewx.xtypes.get_aggregate("outTemp", TimeSpan(1726264800, 1726335600), "mintime", dbmanager)

    print(results)

yields

(4.5, 'degree_C', 'group_temperature')
(1726303300, 'unix_epoch', 'group_time')

Where 1726264800 is the start of today in my timezone. 1726303300 is the min outtemp of toady as in archive_day_outtemp.

results = weewx.xtypes.get_aggregate("outTemp", TimeSpan(1726238400, 1726335600), "min", dbmanager)

    print(results)

    results = weewx.xtypes.get_aggregate("outTemp", TimeSpan(1726238400, 1726335600), "mintime", dbmanager)

    print(results)

yields

(4.555172413793103, 'degree_C', 'group_temperature')
(1726303500, 'unix_epoch', 'group_time')

Where  1726238400  is some random time yesterday in my timezone. 1726303500 is the first timestamp of the record in archive, where outtemp has the minimum value.

Obviously, when doing the second "query", I don't get the exact minimum and it's timestamp. Can I only achieve my desired result, if I limit the timespan to not overlap a calendar day change?

[Tom Keffer]

I'm not following your question --- too many numbers flying around --- but let me try to explain the calculation in the hopes that it helps.

When calculating an aggregate, there's always a well-defined answer. In this case, it's the minimum temperature over the aggregate interval, so you scan all the relevant records and look for the minimum and the timestamp of the record that contains it.

However, if the interval starts and ends on day boundaries (that is, at midnight), then a shortcut can be taken. The algorithm can use the daily summaries to speed the calculation. Otherwise, the main archive table is used.

I didn't look too closely, but I don't think either of your examples satisfies this condition. The aggregation interval is something like 20 hours or 27 hours. So, the main archive table should have been used.

Either way, you should get the same answer, although the daily summaries can be slightly more precise for the timestamp. The reason is that while the archive table has the minimum value seen in the interval, it doesn't have the precise time, only the timestamp of the record with the lowest value. By contrast, the daily summaries have the timestamp of the LOOP packet that provided the minimum value, so the precision can be down to a few seconds.

Otherwise, there should be no difference.

Does that help?

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/c183248b-1a04-47bd-8359-51ec5f215741n%40googlegroups.com.

[michael.k...@gmx.at]

You are right, none of those examples meets the condition (starting and ending on day boundaries), but the first example starts at the start of today, ending with the latest timestamp in the database. Which is still today, but not a day's boundary. So my best guess is, that the shortcuts using the daily summaries, is also taken in that particular case.

And yes, your explanation helps.

[Tom Keffer]

Yes, it is. In the interests of clarity, I did not give the complete set of conditions for the daily summaries to be used. The start time must either be the first record in the database, or a midnight boundary. The stop time must either be the last record in the database, or a midnight boundary. Otherwise, the archive database is used.

So, if the ending timestamp of your first example is the last record in the database, then the daily summary would have been used.

I'm glad you're showing an interest in this! It's deep in the guts of WeeWX, but important.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/8138a283-82d2-418b-8a06-d919daee47b0n%40googlegroups.com.

[michael.k...@gmx.at]

It's connected to this issue I am to resolve currently: https://github.com/brewster76/fuzzy-archer/issues/156

The plot is showing a number lower/higher than the real max/min from the stats/NOAA data, which obtain their min/max values from the daily summaries, while the plot is only showing the average of the values for every archive_interval. My approach will be the following:

- determine every day boundary for the given timespan

- get the min/max for every calendar day in that timespan

- if the datetime of a single min/max is within the timespan, insert it to the values of the plot (most likely leading to an extra data point between two existing data point, but that's desired behavior)

-  if the datetime of a single daily min/max is not within the timespan, ignore

This way we have the real min/max values, with their exact datetime of arrival, for every day, in the plot, if they are within the given timespan. If they are not in the timespan, we simply don't have a more precise information about the min/max value for an archive_interval in the plot. If they are within the timespan, but not the min/max value of the plot, we know it is a min/max value of that certain calendar day that value lies within. The user probably won't recognize these values, if not zooming in on the chart, but that's perfectly OK.

[Tom Keffer]

I don't know what aggregation type, if any, was used for the red line. Most likely no aggregation type was called for, in which case the curve will represent "outTemp" for each archive record, which by default (it depends on the accumulator extractor type) will be the average value of all LOOP packets seen in the archive interval. 

By contrast, the max value in the table is the max value seen in any LOOP packet. This can, and most likely will be, higher than these averages.

Not sure what the user expected. WeeWX does not save every LOOP packet, so the curve can only represent the values in the archive records. 

In any case, for a plot of the day's temperatures, the daily archives don't come into play at all. They are only used if the aggregation interval is a multiple of a whole day. 

Hope that helps.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/1c0b6e38-6d6f-4ce9-ae27-107f430cc310n%40googlegroups.com.

[michael.k...@gmx.at]

The red line is just like you expect: the "outTemp" for each archive record, no aggregation. The chart library I use to draw the plot, has a feature to show things like the max and min value of the plot. The user (in that case: myself) wants to have the min/max of each day shown in the plot. If one doesn't know, that the archive doesn't contain the daily min/max values, but the daily summaries do, he might be confused that the plot shows a different maximum, than the NOAA data show, even if it is just a little off. There is a perfectly "legal" technical explanation, why this is the case. Anyway, if the max of the plot matches the max of a particular day, no questions ever will be asked in that regard.

What I will do, is augment the archive data with the max/min values and their timestamps, and combine archive with archive_day, if the min/max values from the daily summary is within the timestamp.

import weecfg

import weewx.manager

import weewx.xtypes

from weeutil.weeutil import TimeSpan

from datetime import datetime, time, timedelta

config_path, config_dict = weecfg.read_config('/home/michi/weewx-data/weewx.conf')

with weewx.manager.open_manager_with_config(config_dict, 'wx_binding') as dbmanager:

    start_time = 1726347600

    end_time = datetime.now()

    start_of_day = datetime.combine(datetime.fromtimestamp(start_time), time.min)

    min_list = []

    while start_of_day < end_time:

        min_tupel = []

        start_of_next_day = start_of_day + timedelta(days=1)

        min_value = weewx.xtypes.get_aggregate("outTemp", TimeSpan(start_of_day.timestamp(), start_of_next_day.timestamp()), "min", dbmanager)[0]

        min_time = weewx.xtypes.get_aggregate("outTemp", TimeSpan(start_of_day.timestamp(), start_of_next_day.timestamp()), "mintime", dbmanager)[0]

        min_tupel.append(min_time * 1000)

        min_tupel.append(min_value)

        if min_time >= start_time and min_time <= end_time.timestamp():

            min_list.append(min_tupel)

        start_of_day = start_of_next_day

print(min_list)

The result (min_list) will be a list of timestamp/values, that will be inserted in the data series for the plot.

[Tom Keffer]

If the plot is for a long time period and a daily aggregation is used, then one can plot "max" values. For example,

        [[[yearhilow]]]
            [[[[hi]]]]
                data_type = outTemp
                aggregate_type = max
                label = High
            [[[[low]]]]
                data_type = outTemp
                aggregate_type = min
                label = Low Temperature

        

With this example, the plot should exactly measure the NOAA data: they are using the same data, that is, the daily summaries.

I didn't quite follow your augmentation plan. I can see getting the regular series of outTemp from the archive table, then squeezing in a false extra point with timestamp and value of the max seen temperature for the day. That should look pretty good. If this is what you are trying to do, I would implement it as a new xtype "get_series()" function. Use the regular xtype function get_series() to get the series, then augment.

See the XType documentation and examples.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/66060704-cfb5-4a5a-8aaa-1c3db3589ba0n%40googlegroups.com.

[michael.k...@gmx.at]

Thank you for the suggestion, I'll take a look into it. I might have to refactor a lot more than I planned. Some parts of "fuzzy archer" didn't age very well, as WeeWX in the mean time, provides functionality that was implemented there in a similar way. But the less code in the extension, the better. 

[Tom Keffer]

The focus of the last couple of development cycles has been on making it easy for extension authors. Glad it shows!

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/3eb8ecfc-043a-4ff3-ba48-be9c4d02d961n%40googlegroups.com.

[michael.k...@gmx.at]

With your hints I've manged to get at least, what I want. I din't (yet) implement a net xType, since I need the high/lows to definde the mark points in the plot. Anyway, here is the result. A plot that hast the daily high/low values in between the archive records, and marked as such:

[Tom Keffer]

Nice! 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/a1fbd799-95cc-4e04-9073-4d24860a8d40n%40googlegroups.com.

[michael.k...@gmx.at]

Thanks, and by the way, I messed up the comments in the picture above, the outTemp/dewpoint plot does show yesterday's low. The humidity and pressure plots don't show yesterday's highs, which are outside the plot's time windows.