adding extension which applies to reporting on one file only

[Rich Altmaier]

I'm creating a Report extension which is fairly expensive to calculate.  When I add my extension to skin.conf, with the usual  search_list_extensions under CheetahGenerator, I find my extension is invoked about 14 times, which costs many seconds.   Is it possible to make a Report extension apply to a selected file generation, such as in generating index.html only?

Thank you!

Rich

[gjr80]

Hi Rich,

Unfortunately the SLEs are applied to each report in a skin. Can you pull out the template concerned into its own skin and use the SLE in that skin only?

Gary

[Thomas Keffer]

Hello, Rich.

I'm afraid I'm not quite understanding your situation.

There are 3 different occasions when your SLE may come into play:

1. When the Cheetah report generator starts up. This is when your SLE gets instantiated, that is, its __init__ function, if any, gets invoked. This happens only once per report generation.
   
2. When a template gets generated. This is when get_extension_list() is invoked, which happens once per template. For the standard search list that comes with weeWX, this is when the top of the tag chain (class TimeBinder), described in How tags work, gets created, however, the chain itself does not actually get chased. 
3. When looking for a tag. The Cheetah template engine will probe all the search list extensions looking for a specific tag. It does this by calling __getattr__ repeatedly with the tag name as an argument --- a cheap call. If it finds a hit, then this is typically when the expensive stuff starts happening. Note that this probing will happen for every tag, whether your tag or not. But, the rest of the tag chain gets invoked only if Cheetah finds a hit. For the search list that comes with weeWX nothing expensive happens until the very end of the tag chain. That's when the database lookups start happening. 

You say your expensive calculation is happening 14 times. If you have 14 of your tags sprinkled throughout the templates, then this is unavoidable: 14 requests for your tag, 14 calculations.  However, if you expect all 14 tags to produce the same results (an example would be $station.altitude) then you should be doing your calculation farther up the stack described above. Perhaps you should be doing the calculation in get_extension_list() or even __init__.

I hope this make sense. If not, give us some more specifics.

Cheers,

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gjr80]

Tom,

Perhaps I am not understanding you correctly, but I have run into this issue a number of times with a processing intensive SLE. Whether or not any of the tags created by the SLE are used, the SLE is run in its entirety for each template listed in the skin. For example, I have a SLE that returns some rain related tags and it takes about 1 second for get_extension_list() to execute. I included this SLE in the search_list_extensions list in the Standard skin.conf. None of the tags returned by the SLE are used in any template in the Standard skin, yet during the report cycle the SLE is invoked 15 times in total taking a little over 15 seconds in total.

I can understand that the TimeBinder 'chain' is more efficient in that time consuming tasks such as db queries only occur if the tag is 'hit', but not every SLE uses the TimeBinder chain approach. I think that is the issue Rich is seeing (and I have encountered in the past)

Gary

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[gjr80]

We first came across this issue with weeWX-WD back in 2013. We shaved a lot of time off our overall report generation time by judiciously dividing our reports between a number of skins and only including those SLE in each skin that were actually used by all reports in the skin.

Gary

[Thomas Keffer]

I think we're all on the same page here.

The Standard report typically involves 14 file generations, so the function get_extension_list() will be invoked 14 times. If that's taking a long time, I would suggest trying to restructuring the SLE so the get_extension_list() invocation is not so expensive. For example, perhaps the parameters for the invocation could be stored in the object, but not actually do anything unless the tag is used (lazy evaluation). Or, if all 14 of those expensive invocations are identical, put it in __init__(), instead of get_extension_list(), then do it just once and store the results.

Of course, if the expensive calculation needs to be done for every template and they are all different, then there's no way around it. But, I find it hard to imagine such a case.

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[gjr80]

Agree with the last para.

There is probably a lot that can be done in terms of restructuring some of my old SLEs given that some are 5 years old (I regularly cringe at some of the code I used back then). Needs a clearer head than I have just now in the wee hours.

Getting back to Rich's issue, perhaps if you want to show us your SLE code we might be able to give you a few pointers as to how it can be refactored to operate more efficiently.

Gary

[Thomas Keffer]

Agreed. The SLE architecture is layered for a reason: it gives you 3 different choices of where to perform a calculation: once per report, once per template, or once per (hit) tag. I'm pretty confident we can find the right place for Rich's expensive calculation.

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Rich Altmaier]

Wow, you guys are giving a lot of ideas there, thanks so much!    First to supply info:

  -I have only one tag usage presently, in the index.html.tmpl  file.

  -the cost comes from DB queries, for which I wanted the db_lookup class, which is passed to get_extension_list.  

  -I am creating the TimespanBinder, to cover a rainstorm period.   Thereby allowing various stats over the storm period, obviously $last_rain_storm.rain.sum, but could be other interesting things like $last_rain_storm.outTemp.max, etc.     So I followed the example in setting the calculated TimespanBinder into a variable which is added to the dictionary returned by get_extension_list.  

Some clarifying questions:

-I didn't understand __init__, so that looks useful, but how can I obtain the db_lookup method there?

-mentioning "once per tag" or do code when tag is used, what is the technique to detect or run on tag usage, to calculate the TimespanBinder?

I can also fall back to saving the complex results in a temp DB and reusing without recalculating.

Thanks! Rich