Adding archive data to daily report

[Thomas Carlin]

First I would like to thank TK and everyone else who has contributed to this project.  It is a fantastic project, and great to see an active community.  

I have been playing with modifying the reports to fit my wants, and one thing that I would like to do is be able to display historical records on the reports.  On the daily, High's and lows this day last year, the monthly, maybe averages from this month last year, etc.  I have spent some time searching the Group and the docs, but have not come up with much on the topic.  

Could someone point me in the right direction?

For those who are interested, a little back story on my project:

I have a Vantage Pro2, that I have cobbled together a wireless serial relay using an ESP8266 chip (Look Ma, No wires!).  I didn't want my station to be tied to a computer, and those Raspberry Pi's are so clunky!

Largely, I used the information that was compiled by DeKay, using his USB method, I simply adapted it to run over IP instead.  The project is still in its infancy, but I hope to put some lipstick on this pig, and post my progress somewhere that others may benefit from it.  

It has only been running for a week or so at this point, but my station can be seen here http://carlincomputing.duckdns.org/weewx/index.html

[gjr80]

Hi,

What you really need to do is to create some new tags that you can use in your report template. This can be done in a number of ways, for simple cases, such as 'what is the current air density' you could do this with some in-line python code in the template. For more advanced cases, for example those where you need to query the database, this is usually best done through a Search List Extension (SLE). SLEs are the mechanism available for user's to extend the search list that is used by weeWX and the Cheetah template engine to produce reports. You can read about some of the basics of SLEs in the defining new tags section of the Customization Guide.

Assuming you are keen to go down this path, the example SLE in the above link should give you most of what you need to know/do, where you will depart from the example is the calculation of the timespan on which to base your stats for your day last year. To develop an SLE that will make accessible statistics for this day last year one approach might be to:

1. Calculate timestamps for the start and end of the day one year ago. In an SLE timespan.stop will hold the timestamp of the report so that effectively gives you 'now', so you just need to go back a year and get a start of day and end of day timestamps. Hint, have a look at archiveDaySpan() in weeutil.py. Remember you will need to cater for February 29 in your calculations.
2. Once you have start and end of day timestamps you can create a TimeSpan object from them (refer to weeutil.py again).
3. Then you can get a TimeSpanBinder object based on your newly created TimeSpan object. If you are keen, look through tags.py to learn how a TimeSpanBinder works.
4. Then just 'return' your TimeSpanBinder object in a list of dictionaries (look at #8 and #9). If you use a contruct like:
   

today_last_year = TimeSpanBinder(.....)

search_list_extension = {todaylastyear: 'today_last_year'}

then in your template you will be able to use

$todaylastyear.outTemp.max

to display the maximum temperature on this day last year.

That is the hard bit done, the rest is just making sure you new tags are available to your template and then using them in your template. Everything you need for this you will find in the defining new tags section of the Customization Guide.

Have a good read of the defining new tags section of the Customization Guide, in fact having a look at the Customizing templates section would probably help too. Also have a look through the pieces of code/files I mentioned and give it a go. Don't be afraid of coming back if you run into issues, trust me, the first one is the hardest and they get easier after that!

Gary

[Thomas Carlin]

That was the missing link, and should get me going in the right direction!  Hopefully I'll have a chance to play with it this weekend  a bit.

Thanks Gary!

[Thomas Carlin]

I'm making pretty good progress on this, but it is slow going to troubleshoot my extension script since I have to wait for the Cheetah Generator to run.  Is there any way to run that process manually?

Many thanks!

[Thomas Keffer]

http://weewx.com/docs/customizing.htm#wee_reports

-tk

On Sat, Feb 4, 2017 at 12:03 PM, Thomas Carlin <thomas...@gmail.com> wrote:

I'm making pretty good progress on this, but it is slow going to troubleshoot my extension script since I have to wait for the Cheetah Generator to run.  Is there any way to run that process manually?

Many thanks!

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Carlin]

Thanks Tom, I knew it had to be there somewhere!

On Saturday, February 4, 2017 at 1:38:49 PM UTC-7, Tom Keffer wrote:

http://weewx.com/docs/customizing.htm#wee_reports

-tk

[Thomas Carlin]

Hopefully my last question for this project.  If you are interested in the progress, the link in my first post includes the added history sections.

I have defined several new tags in examples/ArchiveSearch.py

today_last_year
week_last_year
mont_last_year
last_year
last_year_to_date

and those all work as expected.  I can use them in my templates, and they give me the data that I expect to see (More or less, today_... is off one day, the wrong direction to compensate for leap year last year, but that is beside the point) but when it comes to the date fields, mintime, maxtime, etc. they are displaying using the 'current'  format defined in [[TimeFormats]]  I tried adding each of my tags to the [[TimeFormats]] list, but had no success.  

Is there something that I missed in the docs that explains this?

[Thomas Keffer]

The initializer for the TimespanBinder object takes an argument 'context', which is what formatting to use for time. The default is "current". Other options are "hour", "day", "week", "month", "year", and "rainyear". Choose one of those, depending on whether you are displaying an aggregate over a day, week, etc.

-tk

--

[Thomas Carlin]

That did the trick.  Thanks again Tom

[Chris Alemany]

Hi Thomas! 

This is a great addition, love how you presented it on your website too.

Do you have the extensions you made available to download anywhere?

Thanks!

Chris

[Thomas Carlin]

Hi Chris!

Thank you for the kind words.  I did not make this available anywhere, but I would be happy to post it here if you are interested.  Keep in mind that I am not a programer by any stretch, but I haven't had any issues with it.  The really cool extension on the history tab I didn't write, but I extensively modified it to auto-define the color scheme, and color the fonts accurately so they don't get lost on the background.  The original unmodified version is available here:  https://github.com/brewster76/fuzzy-archer/blob/master/bin/user/historygenerator.py, and I am less confident in the modifications I made to it than the historical summary cards, but I have debated submitting it to the author for review anyway.  

Let me know if you would like the History Cards, and I'll post some code.

[Chris Alemany]

The history cards are totally cool! Would love to see how you made them. Thanks for this!

[Thomas Carlin]

Hi Chris,

Sorry it took so long, I started a new job a few weeks ago, and hobbies have taken a back seat!  Let me start by saying that all the information, and better descriptions of what is happening can be found in the Weewx Customization Guide that TK and the guys have put together.  http://weewx.com/docs/customizing.htm

All of the back end is contained in one file, On my system \usr\share\weewx\user\ArchiveSearch.py  Look at Defining New Tags in the docs, the similarities should be immediately apparent.  This then get's included in the search list extensions in skin.conf.  You can then include your new tags in your *.tmpl files, to be generated each time the generator runs. 

First the ArchiveSearch.py:  I removed most of the comments that are in the documentation for the sake of brevity.  It is worth noting that you may need to install python relativedelta on your system for this to work.

import datetime

import time

from dateutil.relativedelta import relativedelta

from weewx.cheetahgenerator import SearchList

from weewx.tags import TimespanBinder

from weeutil.weeutil import TimeSpan

from weeutil.weeutil import archiveDaySpan

from weeutil.weeutil import archiveWeekSpan

from weeutil.weeutil import archiveMonthSpan

from weeutil.weeutil import archiveYearSpan

class ArchiveSearch(SearchList):                                                 # 1

    """My search list extension"""

    def __init__(self, generator):                                           # 2

        SearchList.__init__(self, generator)

    

    def get_extension_list(self, timespan, db_lookup):                       # 3

        """Returns a search list extension with two additions.

        

        Parameters:

          timespan: An instance of weeutil.weeutil.TimeSpan. This will

                    hold the start and stop times of the domain of 

                    valid times.

          db_lookup: This is a function that, given a data binding

                     as its only parameter, will return a database manager

                     object.

        """

        # First, create TimespanBinder object for all time. This one is easy

        # because the object timespan already holds all valid times to be

        # used in the report.

        all_time = TimespanBinder(timespan, 

                                   db_lookup,

                                   formatter=self.generator.formatter,

                                   converter=self.generator.converter,

                                  context="month")       # 4

        

        # Now get a TimespanBinder object for the last seven days. This one we

        # will have to calculate. First, calculate the time at midnight, seven

        # days ago. The variable week_dt will be an instance of datetime.date.

        today_last_year_dt = datetime.date.fromtimestamp(timespan.stop) - relativedelta(years=1)

        # Convert it to unix epoch time:

        today_last_year_ts = time.mktime(today_last_year_dt.timetuple())       # 6

        today_last_year_time_span = archiveDaySpan(today_last_year_ts)

        # Form a TimespanBinder object, using the time span we just

        # calculated:

        today_last_year= TimespanBinder(today_last_year_time_span,

                                         db_lookup,

                                         formatter=self.generator.formatter,

                                         converter=self.generator.converter,

                                         context="month") # 7

#Assembles this week last year

        week_last_year_time_span = archiveWeekSpan(today_last_year_ts)

        # Form a TimespanBinder object, using the time span we just

        # calculated:

        week_last_year= TimespanBinder(week_last_year_time_span,

                                         db_lookup,

                                         formatter=self.generator.formatter,

                                         converter=self.generator.converter,

                                         context="month") # 7

#Assembles this month last year.

        month_last_year_time_span = archiveMonthSpan(today_last_year_ts)

        # Form a TimespanBinder object, using the time span we just

        # calculated:

        month_last_year= TimespanBinder(month_last_year_time_span,

                                         db_lookup,

                                         formatter=self.generator.formatter,

                                         converter=self.generator.converter,

                                         context="month") # 7

#Assembles last year.

        last_year_time_span = archiveYearSpan(today_last_year_ts)

        # Form a TimespanBinder object, using the time span we just

        # calculated:

        last_year= TimespanBinder(last_year_time_span,

                                         db_lookup,

                                         formatter=self.generator.formatter,

                                         converter=self.generator.converter,

                                         context="month") # 7

        #Assembles last year to date.

        

        last_year_start_dt = datetime.date(today_last_year_dt.year,1,1)

        last_year_start_ts = time.mktime(last_year_start_dt.timetuple())

        # Form a TimespanBinder object, using the time span we just

        # calculated:

        last_year_to_date= TimespanBinder(TimeSpan(last_year_start_ts, today_last_year_ts),

                                         db_lookup,

                                         formatter=self.generator.formatter,

                                         converter=self.generator.converter,

                                         context="month") # 7

        

        # Now create a small dictionary with keys 'alltime' and 'seven_day':

        search_list_extension = {'today_last_year'   : today_last_year,

                                 'week_last_year' : week_last_year,

                                 'month_last_year': month_last_year,

                                 'last_year': last_year,

                                 'last_year_to_date' : last_year_to_date,

                                 'all_time' : all_time}              # 8

        

        # Finally, return our extension as a list:

        return [search_list_extension]                                       # 9

Next, we add it to skin.conf, I'm using Sofaskin, written by Blaues Ledersofa:

[CheetahGenerator]

    # This section is used by the generator CheetahGenerator, and specifies

    # which files are to be generated from which template.

    # Possible encodings are 'html_entities', 'utf8', or 'strict_ascii'

    encoding = html_entities

    search_list_extensions = user.ArchiveSearch.ArchiveSearch

Finally, we use our new tags in our *.tmpl files.  Here is a snippet from my Year tab.  You will see how the new tags are being used, again i removed some bits for brevity.

      <div class="card" >

                    <h1><i class="fa fa-history m-rot" ></i> History: Last Year to Date</h1>

     <div>

    <table class="tablespacer">

      <tr><td colspan=3><h3>Outdoor Conditions</h3></td></tr>

                      <tr><td>High Temperature</td><td>$last_year_to_date.outTemp.max</td><td>$last_year_to_date.outTemp.maxtime</td></tr>

                      <tr><td>Low Temperature</td><td>$last_year_to_date.outTemp.min</td><td>$last_year_to_date.outTemp.mintime</td></tr>

      <tr><td>Average Temperature</td><td>$last_year_to_date.outTemp.avg</td><td></tr></tr>

                        <tr><td>High Heat Index</td><td>$last_year_to_date.heatindex.max</td><td>$last_year_to_date.heatindex.maxtime</td></tr>

                        <tr><td>Low Wind Chill</td><td>$last_year_to_date.windchill.min</td><td>$last_year_to_date.windchill.mintime</td></tr>

                      <tr><td>High Dewpoint</td><td>$last_year_to_date.dewpoint.max</td><td>$last_year_to_date.dewpoint.maxtime</td></tr>

                        <tr><td>Low Dewpoint</td><td>$last_year_to_date.dewpoint.min</td><td>$last_year_to_date.dewpoint.mintime</td></tr>

                        <tr><td>High Humidity</td><td>$last_year_to_date.outHumidity.max</td><td>$last_year_to_date.outHumidity.maxtime</td></tr>

                        <tr><td>Low Humidity</td><td>$year.outHumidity.min</td><td>$last_year_to_date.outHumidity.mintime</td></tr>

                        <tr><td>High Barometer</td><td>$last_year_to_date.barometer.max</td><td>$last_year_to_date.barometer.maxtime</td></tr>

                        <tr><td>Low Barometer</td><td>$last_year_to_date.barometer.min</td><td>$last_year_to_date.barometer.mintime</td></tr>

      <tr><td colspan=3><h3>Wind and Rain</h3></td></tr>

                      <tr><td>High Wind Speed</td><td>$last_year_to_date.wind.max $last_year_to_date.wind.gustdir</td><td>$last_year_to_date.wind.maxtime</td></tr>

                        <tr><td>Average Wind</td><td>$last_year_to_date.wind.avg</td><td></td></tr>

                        <tr><td>RMS Wind</td><td>$last_year_to_date.wind.rms</td><td></td></tr>

                        <tr><td>Vector Average Speed</td><td>$last_year_to_date.wind.vecavg</td><td></td></tr>

                        <tr><td>Vector Average Direction</td><td>$last_year_to_date.wind.vecdir</td><td></td></tr>

      <tr><td colspan=3><h3>Indoor Conditions</h3></td></tr>

                      <tr><td>High Inside Temperature</td><td>$last_year_to_date.inTemp.max</td><td>$last_year_to_date.inTemp.maxtime</td></tr>

                        <tr><td>Low Inside Temperature</td><td>$last_year_to_date.inTemp.min</td><td>$last_year_to_date.inTemp.mintime</td></tr>                       

                    </table>

     </div>

      </div>

Before you do any of this to your system, I would take an hour and read through the entire customization guide.  You may not understand everything as you read it, but it will help you.  I asked many stupid questions that were covered in the guide when I was getting started.  

Good Luck!

[RothMa]

Hello Thomas,

I want to create my own skin for Weewx. My search on how to display historical data on the websites led me to https://github.com/brewster76/fuzzy-archer and https://github.com/weatherstorm/Sofaskin-CW9009. I also found this thread. Thanks for the detailed explanations.
Now I took some code from both Github projects and was able to integrate the historical tables in my site using the "historygenerator.py". I'm able to change HTML and CSS configurations, but I'm not good at programming Python.
On your wonderful page you have a monthly summary on the right side of the tables and an annual overview below each table. Can you please show me how you've extended the historygenerator.py to create the Monthly Summary column and the Annual Summary row?

thank you very much
Best regards
Martin


[Thomas Carlin]

Hey Martin,

Thank you for your interest!  I don't know if I mentioned it here, but even cooler than that IMHO, the color scale on all the charts is dynamic, and will adjust as the range does. The font colors also change dynamically for better visibility.

For the record, I am not a python programmer by any means, everything I know is self inflicted, and I am nearly positive that there is a better way to do everything that I have done here, which is why I haven't submitted these changes to GitHub for peer analysis, and integration.  Working with someone else's code and modifying it to work for your needs is difficult, and not always effective.  When I have time in the next few days, I will try to pull the code off my machine and post it, with some basic direction, but the support that I provide will be limited.  If anyone from FuzzyArcher sees this, I would be happy to submit it to GitHub.  I would hesitate to say that this is a 'finished' product, but it works for my needs.  Thoughts and recommendations are welcome!

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/FHuFLPrg_DE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thomas Carlin
970-401-3805