Ramblings from an XType developer

[bell...@gmail.com]

Here are some of my findings. I gathered them here to hopefully help others. I hope they are correct, but I can’t guarantee it. These are based on using the XType as a tag, I haven’t yet delved into direct calls from other services/processing. This is based on 5.0.2 plus commits up to 3625c02.

If one raises a CannotCalculate in the XType get_scalar, for the time periods ‘$current’ and ‘$latest’, it will bubble up to the ‘tag’ layer where it will become a ‘None’ value.

For the get_aggregation, the CannotCalculate is not caught and will bubble all the way, causing report generation to halt. Based on the above behavior, I am catching any CannotCalculate from a get_scalar call and returning a ‘None’ value from get_aggregation.

Based on the above two observations, if get_scalar raises a CannotCalculate exception due to missing data that is required, a debug message is logged. If it is for some other reason, an error debug message is logged. Errors might be due to bad configuration. I’m currently looking at if it is possible to detect all of these at start up.

The only way to break out of the XType get_aggregation is to return a value. In my case the ‘sum’ aggregate does not make much sense. But if I don’t return a value the XTypeTable will perform it (assuming I have prepended my XType into the system). Since I can easily calculate the ‘sum’, I do… (See below, I don’t think there is any benefit to my implementation. But its my XType and I want control… ha ha. Well it does stop the XType system for looking for an implementation)

Some of the aggregation types supported by XTypeTable can be ‘expensive’. One may/should prepend the Xtype to ‘override’ these. This is because the ‘default’ has to loop through and calculate all the values in the interval. In my case, only calculating the ‘sum’ and ‘avg’ require this. Others could be computed after a single sql query.
 

I’ll post any other findings/ramblings/corrections I find. Hope this helps someone.
 rich
 

[Karen K]

See issue #939 for some details of the performance of XTypeTable.

[bell...@gmail.com]

Its hard to come up with general implementations of the aggregation types that will perform. I bet most, if not all, XTypes will need to ‘override’ the ones in XTypeTable. Take ‘min’ for example. The general implementation has to calculate all values in the interval. My specific implementation can query for the minimum value of the inputs and just compute from that result. Granted the computation is simple, but if it was complex enough; I’d persist the value.
We’ll see what magic Tom can perform

[Karen K]

I thought about observation types that are calculated within an XType, but then saved to the database. After that summaries could be retrieved from the database instead of calculated again. So I guess neither inserting the XType at the beginning nor appending it at the end is the best solution. For the weewx-GTS extension I now decided to insert it before XTypeTable, but after DailySummaries and ArchiveTable like this:

        # Register the class
        archive_seen = False
        summaries_seen = False
        for idx,xtype in enumerate(weewx.xtypes.xtypes):
            if (isinstance(xtype,weewx.xtypes.XTypeTable) or
                                            (archive_seen and summaries_seen)):
                weewx.xtypes.xtypes.insert(idx,self.GTSextension)
                break
            if isinstance(xtype,weewx.xtypes.ArchiveTable):
                archive_seen = True
            elif isinstance(xtype,weewx.xtypes.DailySummaries):
                summaries_seen = True
        else:
            weewx.xtypes.xtypes.append(self.GTSextension)

[bell...@gmail.com]

Hmmm, another vote for the XTypeTable always being the last XType in the list; serving as a ‘backstop’ if there are no more specific implementations.
I hadn’t thought about the ArchiveTable XType. If person A persists my XType (adds it to the db), I need to make sure my XType is after ArchiveTable but before XTypeTable.
I suppose there is a use case for overriding ArchiveTableXType (prepending it, but at this time I don’t see it….
My head hurts! Thanks for the code snippet.

[Tom Keffer]

In practice, this is not a problem.

The tag $current.myxtype checks the database first before using the xtypes system.

For an aggregate, for your xtype to be used, it must appear before XTypeTable and it must provide get_aggregate() and it must provide the requested type of aggregation. So, for a tag such as $day.myxtype.min, if you prepend your xtype, it will only be used if you provide get_aggregate() and provide an implementation for aggregate type 'min'. 

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/2ffcddf2-5695-4d24-bb49-c04de6b87467n%40googlegroups.com.

[Karen K]

Tom Keffer schrieb am Samstag, 2. März 2024 um 02:00:54 UTC+1:

For an aggregate, for your xtype to be used, it must appear before XTypeTable ...

In the Wiki about XTypes all the examples append the XType to the list. Not a single example does it otherwise. And there are a lot of examples. I must confess I did not realize that one sentence about prepending the XType there. By now I thought you wanted users to always append their XTypes. And there is no word that it may be necessary to insert the XType between others. May be that confused developers.

I wonder why WeeWX 4.X worked fine without get_aggregate() in XTypeTable.

[Tom Keffer]

1. In most cases, an xtype offers a new type or a new aggregation, in which case it doesn't matter if you prepend or append. This is the case of the examples in the xtype document. However, if an optimization is offered, that is, a specialized version of get_aggregate() then, yes, it should be prepended. I will make the wiki change.

2. I am not aware of a case where it's necessary to insert an xtype between the others. Let me know if I missed something.

3. As I have said repeatedly, V4.x does less. It does not "work fine" --- it doesn't work at all. For example, you can't do an aggregate of a synthetic type in V4.x.

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/1c5b42cc-07c8-4eb8-b83d-af6b077a4b64n%40googlegroups.com.

[Karen K]

Tom Keffer schrieb am Samstag, 2. März 2024 um 13:02:03 UTC+1:

2. I am not aware of a case where it's necessary to insert an xtype between the others. Let me know if I missed something.

I see 2 cases. 

("you" means the XType developer in the following sentences. I am not sure with my English.)

First is the ArchiveTable class. It calls the general get_aggregate function and thus also your XType. If you want ArchiveTable handle get_series aggregations for your XType, you have to insert your XType after ArchiveTable. But if you implemented aggregations and do not want XTypeTable handle those aggregations, then you need to insert your XTypes before XTypeTable. That is, between ArchiveTable and XTypeTable.

If your users want to speed up something and include your XTypes into the database, they surely want the DailySummaries class handle aggregations. But that does not mean that you want XTypeTable handle the aggegations of your XType in the other case. You may have provided your own implementation of the aggregations. That means you have to insert your XType after DailySummaries but before XTypeTable.

So I guess if you want your XType to behave in an intuitive way and you have implemented the aggregations, then you will have to insert it after ArchiveTable and DailySummaries, but before XTypeTable.

[Tom Keffer]

On Sat, Mar 2, 2024 at 9:38 AM Karen K <kk44...@gmail.com> wrote:

First is the ArchiveTable class. It calls the general get_aggregate function and thus also your XType. If you want ArchiveTable handle get_series aggregations for your XType, you have to insert your XType after ArchiveTable. But if you implemented aggregations and do not want XTypeTable handle those aggregations, then you need to insert your XTypes before XTypeTable. That is, between ArchiveTable and XTypeTable.

I guess I'm not seeing that. If ArchiveTable.get_series() calls xtypes.get_aggregate(), it will run down the list of xtypes. It won't use the one in ArchiveTable because it doesn't know about your new type. It keeps going until it gets to your xtype extension. 

 

If your users want to speed up something and include your XTypes into the database, they surely want the DailySummaries class handle aggregations. But that does not mean that you want XTypeTable handle the aggegations of your XType in the other case. You may have provided your own implementation of the aggregations. That means you have to insert your XType after DailySummaries but before XTypeTable.

If the XType is in the database, it will be handled like any other type. There is no longer anything special about it. There is no reason for XTypeTable to come into play. I also don't know why you'd want your own implementation of get_aggregate() if the type is in the database.

So I guess if you want your XType to behave in an intuitive way and you have implemented the aggregations, then you will have to insert it after ArchiveTable and DailySummaries, but before XTypeTable.

Respectively disagree. New XTypes can be appended or prepended to the list --- the only time it matters is a straight optimization of get_aggregate(). Then it must appear at the beginning of the list so it appears before XTypeTable.get_aggregate().

See the new wiki article on xtypes performance. It explains some of this.

-tk

[John Kline]

It looks like I specialized get_aggregate() for my weewx-purple extension in July of 2020.  AQI isn’t in the database, but pm2_5 is, which makes things efficient.

I don’t know if get_aggregate was actually called in v4 (since I update my skin pages on every loop record, the values initially generated in reports aren’t interesting).

I do know that reporting got slower in v5.

And this simple change dramatically speeded things up:

- weewx.xtypes.xtypes.append(AQI())

+ weewx.xtypes.xtypes.insert(0, AQI())

I’m glad I finally paid attention to this subject.  Thanks, Tom and others.

On Mar 2, 2024, at 5:16 PM, Tom Keffer <tke...@gmail.com> wrote:



--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEA8h_DYLP6%2BrTB4VeEo46%2B9ySXsfasExohbptVuk8peSw%40mail.gmail.com.

[Tom Keffer]

I've added an optimized version of get_aggregate(). This version recognizes the monotonic relationship between temperature and vapor pressure, so it can use the daily summaries to find the min, max, mintime, and maxtime of vapor pressure. It would not work for the average value.

See https://github.com/weewx/weewx/wiki/XTypes-performance#optimized-version-of-get_aggregate

[bell...@gmail.com]

Tom,

Thanks for the example. I did the same thing, just not as elegant. When I have time, I will try this out.

I also did it for ‘not_null’ since it is called when ‘has_data’ is called. For most people, probably not necessary, but I had a sensor offline recently, so the default get_aggregate of ‘not_null’ had to iterate until a time when the sensor was online. 

Again, thanks!
rich

[Tom Keffer]

Rich,

Going back to your post in this thread, one thing I forgot to mention: You and I talked about how the XTypes system is not very rigorous on when a function should or should not raise weewx.CannotCalculate.  HEAD has tightened this up and now has a few changes that prevent get_aggregate() and get_series() from raising weewx.CannotCalculate. Instead, they return None. 

However, get_scalar() can still raise CannotCalculate.

The XTypes wiki has been revised to clarify this 

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/c1b54377-63b9-4261-89f6-af80ce29779cn%40googlegroups.com.

[bell...@gmail.com]

Tom,
I run almost everything WeeWX out of github. Either checking out tags or running at HEAD (or close to it), so my development environment is currently at head. My specialized get_aggregate and get_series catch any CannotCalculate and return None. I had one CannotCalculate that is now handled at startup by checking the configuration. All makes sense now and is working well. I guess now I don’t actually have to catch it.
I think my AQI XType is now working well. It was a fairly simple one because it only depends on particulate matter. Only the ‘avg’ aggregate requires calculating all values in the series and averaging. ‘min’, ‘max;, ‘first’, ‘last’ can just retrieve the single particulate matter and calculate the AQI.
I’m now experimenting with the Nowcast algorithm. One could argue why… Because? It uses up to the last 12 hour readings, so I think any aggregation period less than a day makes no sense. So I will probably need to handle that somehow.  We’ll see what I learn from this.
- rich