use non standard aggregations

[enu...@gmail.com]

I want to make a graph using the xaggs extension which has aggregations like historical_avg and is used like this: outtemp.historical_avg.
To make the graph use
[[[dayhistorical]]]
             [[[[outTemp]]]]
                 aggregate_type = historical_temp
but nothing happens. Neither error nor graphic.

Can I do that kind of graphics?

[enu...@gmail.com]

Where I write historical_temp is historical_avg

[enu...@gmail.com]

if i put aggregate_interval = day i get a error

Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine: Caught unrecoverable exception in generator 'weewx.imagegenerator.ImageGenerator'
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****  outTemp
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 196, in run
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****      obj.start()
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 281, in start
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****      self.run()
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 41, in run
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****      self.genImages(self.gen_ts)
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 181, in genImages
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****      aggregate_interval=aggregate_interval)
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 99, in get_series
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****      raise weewx.UnknownType(obs_type)
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****  weewx.UnknownType: outTemp
Dec  9 15:45:34 raspberrypi weewx[17702] ERROR weewx.reportengine:         ****  Generator terminated

[Karen K]

The error message says, outTemp is missing. 

Please set debug to 1 in weewx.conf restart WeeWX und wait one report cycle to pass. Then post the log here. Be sure to obfuscate all personal data in what you post. As well post skin.conf.

What driver do you use?

[enu...@gmail.com]

With debug=1 don't change nothing. Only the error comes out as before. My driver is sdr and in my skin.conf i only change xtype_services = user.xaggs.XAggsService,. I can use the aggregation to put info in the web page with $day.outTemp.historical_avg but i can't use it with graphs like [[[historical]]]

                                                                                                                 [[[[outTemp]]]]

                                                                                                                       aggregation_type=historical_avg

If only i write this i get the message "Aggregate interval required for aggregate type historical_avg but if I add aggregation_interval = day i get the error that i put before

Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine: Caught unrecoverable exception in generator 'weewx.imagegenerator.ImageGenerator'
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****  outTemp
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 196, in run
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****      obj.start()
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 281, in start
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****      self.run()
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 41, in run
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****      self.genImages(self.gen_ts)
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 181, in genImages
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****      aggregate_type=aggregate_type,
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 99, in get_series
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****      raise weewx.UnknownType(obs_type, timespan, aggregate_type, aggregate_interval)
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****  weewx.UnknownType: outTemp
Dec  9 17:40:48 raspberrypi weewx[22716] ERROR weewx.reportengine:         ****  Generator terminated

[Karen K]

Look, we would like to see the whole log from starting WeeWX until up to the first reporting cycle. Not only the short snippet you posted. 

What I guess from that short log is that there are no values for outTemp in your database or the driver does not supply an observation type of that name.

Certainly it has nothing to do with aggregation.

[Tom Keffer]

There's a small bug in the installer for weewx-xaggs. It neglects to add the startup service for the xaggs xtypes. 

To fix, go into weewx.conf, and add 'user.xaggs.XAggsService' to the end xtype_services. When you're done, the [Engine] section will look something like this:

[Engine]

    [[Services]]

        ...

        xtype_services = weewx.wxxtypes.StdWXXTypes, weewx.wxxtypes.StdPressureCooker, weewx.wxxtypes.StdRainRater, weewx.wxxtypes.StdDelta, user.xaggs.XAggsService

        ...

 

Then restart weewxd.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a772675a-4b2c-40cb-8fc2-6e0e81e8f4bfn%40googlegroups.com.

[Tom Keffer]

My apologies: xaggs cannot do what you want. It does not contain a function to calculate series, only scalars.

I don't think it would be hard to extend it. Let me think about this for a while.

-tk

[Tom Keffer]

I was able to get this to work. The key is to specify an aggregation interval of one day. For example,

        [[[monthhistorical]]]
             [[[[min]]]]
                 label = min
                 aggregate_type = historical_min
                 aggregate_interval = day
                 data_type = outTemp
             [[[[outTemp]]]]
             [[[[max]]]]
                label = max
                aggregate_type = historical_max
                aggregate_interval = day
                data_type = outTemp

Gave the plot:

[enu...@gmail.com]

I had already tried aggregation_interval = day but had not tried data_type = outTemp. I have tried the example you have sent and the result is the same.

Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine: Caught unrecoverable exception in generator 'weewx.imagegenerator.ImageGenerator'
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****  outTemp
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 196, in run
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****      obj.start()
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 281, in start
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****      self.run()
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 41, in run
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****      self.genImages(self.gen_ts)
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 181, in genImages
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****      aggregate_interval=aggregate_interval)
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 99, in get_series
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****      raise weewx.UnknownType(obs_type) #, timespan, db_manager, aggregate_type, aggregate_interval)
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****  weewx.UnknownType: outTemp
Dec  9 22:40:49 raspberrypi weewx[32025] ERROR weewx.reportengine:         ****  Generator terminated

The strange thing is that the type who it says doesn't know is outTemp. I don't understand that it works for you and it doesn't work for me.

[enu...@gmail.com]

The problem was I had put that section on the day charts. If i put in week or month there no errors but it don't plot anything.

[Tom Keffer]

Did you add 'user.xaggs.XAggsService' to the end of xtype_services, as mentioned in an earlier email in this thread?

On Thu, Dec 9, 2021 at 2:18 PM enu...@gmail.com <enu...@gmail.com> wrote:

The problem was I had put that section on the day charts. If i put in week or month there no errors but it don't plot anything.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/96003050-8169-41b4-91cc-9dadf59aed18n%40googlegroups.com.

[enu...@gmail.com]

Yes. I have user.xaggs.XAggService in xtype_services in weewx.conf

[enu...@gmail.com]

I haven't troubles with the scalars. I can use it in the web page with the tag $day.outTemp.historical_min and it gives results.

[Tom Keffer]

Perhaps you don't have enough data? How many years of data in your database?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/5e2ac174-f98b-47ea-a4a1-cce3ceae565cn%40googlegroups.com.

[Tom Keffer]

How many years of data?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/45caefaf-a03b-4fc2-8a6c-17a108b9a4f1n%40googlegroups.com.

[enu...@gmail.com]

More than ten. I insist that if I use  $day.outTemp.historical_min   it works and give me a number

[Tom Keffer]

Hmmm. It certainly works for me.

Can you post your copy of skin.conf, and the [Engine] section of weewx.conf? I'll use them on my end and see if I can reproduce your result.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a5b1bd4a-dc42-44e0-a7a7-256215106348n%40googlegroups.com.

[enu...@gmail.com]

[Tom Keffer]

I would like to see skin.conf as well, but, no matter, your xtype_services section does not include  'user.xaggs.XAggsService'. See an earlier email in this thread. It should read

        xtype_services = user.xaggs.XAggsService, weewx.wxxtypes.StdWXXTypes, weewx.wxxtypes.StdPressureCooker, weewx.wxxtypes.StdRainRater, weewx.wxxtypes.StdDelta, user.xaggs.XAggsService

(all on one line).

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/ba71942e-d16c-4c3a-a0cd-352d1fef25f5n%40googlegroups.com.

[enu...@gmail.com]

But it does. It is the first of the line xtypes_services

xtype_services = user.xaggs.XAggsService , weewx.wxxtypes.StdWXXTypes, weewx.wxxtypes.StdPressureCooker, weewx.wxxtypes.StdRainRater, weewx.wxxtypes.StdDelta

Also I can use its historical_max, historical_min ... tags on the web page with $day.outTemp.historical_min
And skin.conf is stock unless I change to make this chart

[enu...@gmail.com]

I have tested the chart you posted and it is very strange. The only place it works is putting it in monthly charts like the one you sent.
In the weekly chart, the chart that I sent before appears where neither the min nor the historical max appear only outTemp
And if I put it on the daily chart and putting aggregation_period = hour it produces the same error that I showed before with the exception
  **** weewx.UnknownType: outTemp

[enu...@gmail.com]

I think at least one problem is that aggs tags have no aggregation. They simply give one data per day. The problem is that imagegenerator.py doesn't support aggregation_type without aggregation_interval.
I have tried with None and it is the same as if I don't put aggregation_interval.
Give a message:
Aggregate interval required for aggregate type historical_min

What I don't understand is its different behavior in the weekly charts, it doesn't paint anything and in the monthly charts it does.

Are you the author of xagss?

[Tom Keffer]

Sorry, I was looking for user.xaggs.XAggsService at the end of xtype_services.

Yes, an aggregation type and an aggregation interval must always be specified together. Otherwise, they wouldn't mean anything. 

The aggregation interval must always be one day. I'll amend the documentation to make that clear.

If you look closely at your weekly chart, you'll see that it is indeed showing dots for the historical data --- it's just that they are not connected. See the section "Dots in the plots" in the User's Guide. 

Yes, I am the author of XAggs.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/08a26d42-1f89-4bf5-991c-225d7a39bc74n%40googlegroups.com.

[enu...@gmail.com]

I had not seen the points on the weekly chart.
Then the only thing that doesn't work is the daily chart because when adding aggregate_interval = day or hour it gives an error and this is the one that interests me the most.
I wanted to make a graph with the historical temperature of the current moment and the current temperature. For this I had created an sql in xaggs:
'historical_temp': "select avg (outtemp) from archive where strftime ('% m-% d% H', datetime, 'unixepoch') = strftime ('% m-% d% H', 'now')",
Can you think of a way to do it?
I understand that in the case of xaggs labels, aggregation should not be necessary since it only gives one data per day. Unless we group by weeks or months.

[Tom Keffer]

I don't see why your approach would not work, although you have a syntax error in your calls to strftime(). There cannot be a space between the '%' and the time modifier. It should look something like this:

select avg (outtemp) from archive where strftime ('%m %d %H', datetime, 'unixepoch') = strftime ('%m %d %H', 'now')";

I'm not sure what you mean by "xaggs labels", but if you mean the $day tag, then, yes, it is indeed aggregating over one day. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/023a50f2-aa2e-4c75-b298-53fb0b207558n%40googlegroups.com.

[enu...@gmail.com]

The query is ok. I can use it with $day.outTemp.historical_temp.The space I think was a bad copy.  I refer to xaggs labels to the aggregations of xaggs like historical_avg.

What I can't do is get it to work for me on the daily chart. Before it produced the error that I put before. 

Now for some reason it just doesn't plot the graph but it doesn't give an error. I'm looking at where the difference can be from before but I can't find it. I tried with interval_aggregation= hour and day but nothing.

[Tom Keffer]

I was able to get this to work, but not without considerable modifications and limitations. 

1. You need to define new aggregation types. You can't use 'historical_min', because that SQL statement is designed for using the daily summaries. To calculate historical hourly data, you'll have to use the main archive table:

'historical_hour_min': "SELECT min({obs_type}) FROM {table} "
                       "WHERE STRFTIME('%m-%d-%H', datetime, 'unixepoch', 'localtime') "
                       "= '{month:02d}-{day:02d}-{hour:02d}'",
'historical_hour_max': "SELECT max({obs_type}) FROM {table} "
                       "WHERE STRFTIME('%m-%d-%H', datetime, 'unixepoch', 'localtime') "
                       "= '{month:02d}-{day:02d}-{hour:02d}'",

2. The Python code in xaggs.py is designed for using the daily summaries, so it must be modified as well. Here's the total diff:

*** xaggs.py 2021-12-09 07:00:04.000000000 -0800

--- /Users/Shared/weewx/bin/user/xaggs.py 2021-12-12 04:12:45.000000000 -0800

***************

*** 51,56 ****

--- 51,62 ----

                                    "WHERE STRFTIME('%m-%d', dateTime,'unixepoch','localtime') = '{month:02d}-{day:02d}';",

              'historical_avg': "SELECT SUM(`wsum`), SUM(`sumtime`) FROM {table}_day_{obs_type} "

                                "WHERE STRFTIME('%m-%d', dateTime,'unixepoch','localtime') = '{month:02d}-{day:02d}';",

+             'historical_hour_min': "SELECT min({obs_type}) FROM {table} "

+                                    "WHERE STRFTIME('%m-%d-%H', datetime, 'unixepoch', 'localtime') "

+                                    "= '{month:02d}-{day:02d}-{hour:02d}'",

+             'historical_hour_max': "SELECT max({obs_type}) FROM {table} "

+                                    "WHERE STRFTIME('%m-%d-%H', datetime, 'unixepoch', 'localtime') "

+                                    "= '{month:02d}-{day:02d}-{hour:02d}'",

          },

          'mysql': {

              'historical_min': "SELECT MIN(`min`) FROM {table}_day_{obs_type} "

***************

*** 81,106 ****

          if aggregate_type not in XAggsHistorical.sql_stmts[dbtype]:

              raise weewx.UnknownAggregation(aggregate_type)

  

          # The time span must lie on midnight-to-midnight boundaries

!         if db_manager.first_timestamp is None or db_manager.last_timestamp is None:

!             raise weewx.UnknownAggregation(aggregate_type)

!         if not (isStartOfDay(timespan.start) or timespan.start == db_manager.first_timestamp) \

!                 or not (isStartOfDay(timespan.stop) or timespan.stop == db_manager.last_timestamp):

!             raise weewx.UnknownAggregation("%s of %s" % (aggregate_type, timespan))

! 

!         start_day = datetime.date.fromtimestamp(timespan.start)

!         stop_day = datetime.date.fromtimestamp(timespan.stop)

! 

!         # The timespan must cover just one day

!         delta = stop_day - start_day

!         if delta.days != 1:

!             raise weewx.UnknownAggregation("%s of %s" % (aggregate_type, timespan))

  

          interp_dict = {

              'table': db_manager.table_name,

              'obs_type': obs_type,

              'month': start_day.month,

!             'day': start_day.day

          }

  

          # Get the correct sql statement, and format it with the interpolation dictionary.

--- 87,113 ----

          if aggregate_type not in XAggsHistorical.sql_stmts[dbtype]:

              raise weewx.UnknownAggregation(aggregate_type)

  

+         start_day = datetime.datetime.fromtimestamp(timespan.start)

+         stop_day = datetime.datetime.fromtimestamp(timespan.stop)

+ 

          # The time span must lie on midnight-to-midnight boundaries

!         if aggregate_type not in ['historical_hour_min', 'historical_hour_max']:

!             if db_manager.first_timestamp is None or db_manager.last_timestamp is None:

!                 raise weewx.UnknownAggregation(aggregate_type)

!             if not (isStartOfDay(timespan.start) or timespan.start == db_manager.first_timestamp) \

!                     or not (isStartOfDay(timespan.stop) or timespan.stop == db_manager.last_timestamp):

!                 raise weewx.UnknownAggregation("%s of %s" % (aggregate_type, timespan))

!             # The timespan must cover just one day

!             delta = stop_day - start_day

!             if delta.days != 1:

!                 raise weewx.UnknownAggregation("%s of %s" % (aggregate_type, timespan))

  

          interp_dict = {

              'table': db_manager.table_name,

              'obs_type': obs_type,

              'month': start_day.month,

!             'day': start_day.day,

!             'hour' : start_day.hour,

          }

  

          # Get the correct sql statement, and format it with the interpolation dictionary.

3. The skin.conf image specification looks like:

[[day_images]]
    x_label_format = %H:%M
    bottom_label_format = %A %d %B %Y  %H h %M min
    time_length = 97200 # 27 hours
    
    [[[dayhistorical]]]
         time_length = 86400
         [[[[min]]]]
             label = min
             aggregate_type = historical_hour_min
             aggregate_interval = hour

             data_type = outTemp
         [[[[outTemp]]]]
         [[[[max]]]]
            label = max

            aggregate_type = historical_hour_max
            aggregate_interval = hour
            data_type = outTemp

4. Finally, it is slow. The reason is that for the daily historical statistics, it can use the daily summaries, which are quite small. To find historical hourly data, the query engine must scan the entire database table. On my Mac M1, it took 27 seconds to do one plot!

Here's the results:

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b69d3c69-9144-4643-bda1-69b54df0c5cbn%40googlegroups.com.

[enu...@gmail.com]

Thank you very much for your solution. Yes, it is very slow, on my raspberry 4 it takes almost 2 minutes. Does this have any drawbacks? I have the archive interval every 5 min.
There is something that seems strange to me in the graph because according to it today we reach both the historical max and the minimum. It seems odd.

Where I can't make it work is with the belchertown skin, both the historical_ * and the historical_hour_ *

I have in graph.conf
[[chart10]]
         title = historical
         aggregate_type = historical_max
         aggregate_interval = 86400
         observation_type = outTemp