Cheetahgenerator gives "math domain error" while processing Seasons skin

[Rahul Bedi]

Hi,

This morning, weewx running Raspberry Pi starting logging the following errors:

Sep 13 21:25:17 PrideWeatherStation weewx[1608]: manager: Added record 2018-09-13 21:25:00 IST (1536854100) to database 'weewx.sdb'

Sep 13 21:25:17 PrideWeatherStation weewx[1608]: manager: Added record 2018-09-13 21:25:00 IST (1536854100) to daily summary in 'weewx.sdb'

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: cheetahgenerator: Generate failed with exception '<type 'exceptions.ValueError'>'

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Seasons/index.html.tmpl

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: cheetahgenerator: **** Reason: math domain error

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****  Traceback (most recent call last):

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 330, in generate

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****      print >> _file, compiled_template

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1005, in __str__

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****      rc = getattr(self, mainMethName)()

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "_etc_weewx_skins_Seasons_index_html_tmpl.py", line 208, in respond

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1615, in _handleCheetahInclude

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****      self._CHEETAH__cheetahIncludes[_includeID].respond(trans)

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "_etc_weewx_skins_Seasons_hilo_inc.py", line 414, in respond

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/tags.py", line 329, in __getattr__

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****      return self._do_query(aggregate_type)

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/tags.py", line 343, in _do_query

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****      val=val, **self.option_dict)

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/wxmanager.py", line 60, in getAggregate

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****      return weewx.manager.DaySummaryManager.getAggregate(self, timespan, obs_type, aggregateType, **option_dict)

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/manager.py", line 1314, in getAggregate

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****      _result = math.sqrt(_row[0]/_row[1]) if _row[1] else None

Sep 13 21:25:30 PrideWeatherStation weewx[1608]: ****  ValueError: math domain error

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: cheetahgenerator: Generate failed with exception '<type 'exceptions.ValueError'>'

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Seasons/statistics.html.tmpl

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: cheetahgenerator: **** Reason: math domain error

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****  Traceback (most recent call last):

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 330, in generate

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****      print >> _file, compiled_template

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1005, in __str__

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****      rc = getattr(self, mainMethName)()

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "_etc_weewx_skins_Seasons_statistics_html_tmpl.py", line 143, in respond

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1615, in _handleCheetahInclude

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****      self._CHEETAH__cheetahIncludes[_includeID].respond(trans)

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "_etc_weewx_skins_Seasons_statistics_inc.py", line 1043, in respond

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "_etc_weewx_skins_Seasons_statistics_inc.py", line 427, in __errorCatcher52

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "<string>", line 1, in <module>

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/tags.py", line 329, in __getattr__

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****      return self._do_query(aggregate_type)

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/tags.py", line 343, in _do_query

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****      val=val, **self.option_dict)

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/wxmanager.py", line 60, in getAggregate

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****      return weewx.manager.DaySummaryManager.getAggregate(self, timespan, obs_type, aggregateType, **option_dict)

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****    File "/usr/share/weewx/weewx/manager.py", line 1314, in getAggregate

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****      _result = math.sqrt(_row[0]/_row[1]) if _row[1] else None

Sep 13 21:25:32 PrideWeatherStation weewx[1608]: ****  ValueError: math domain error

Sep 13 21:25:40 PrideWeatherStation weewx[1608]: copygenerator: copied 0 files to /var/www/html/weewx

Sep 13 21:25:41 PrideWeatherStation weewx[1608]: imageStackedWindRose: Generated 2 images for StackedWindRose in 1.06 seconds

I assumed that it may be some messed up value in the database, so went back to yesterday's database that had worked fine but the same error keep coming up.

Rebuilt the daily summaries but still the same problem.

Due to this my main page (below) for Seasons skin does not update even though the individual graphs update just fine.

http://bedi.co.in:10000/weather/index.html

Any ideas what could be going wrong ?

Thanks.

- Rahul

[gjr80]

Hi,

I would suspect some dodgy data in your daily summaries. You said you rebuilt them, did you drop them using --drop-daily before you rebuilt them, if not try dropping them first then rebuilding.

Gary

[Thomas Keffer]

Line 1314 is calculating an rms value. Make sure you're doing this against observation type 'wind'. For example $month.outTemp.rms could give you that error.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rahul Bedi]

I did drop and rebuild from scratch but the error did not go away.

I do suspect that there is some data error but unable to figure out which one.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Rahul Bedi]

As I have made no code or config changes since the day before, I believe it doing this for the right data. Let me try tweaking the config and start eliminating some of these fields and see if the error goes away.

Thanks for the help.

- Rahul

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[gjr80]

On Friday, 14 September 2018 11:30:42 UTC+10, Rahul Bedi wrote:

As I have made no code or config changes since the day before, I believe it doing this for the right data.

This sounds to me like you have changed some of the seasons skin files. The error is coming form the use of a .rms tag in one of the seasons .tmpl or .inc files. The seasons skin only uses the .rms tag with the WeeWX wind field. As Tom indicated .rms only works against the wind field, if your changes have used the .rms tag against another field then this could be the cause. I suggest you go through all of your changes to the seasons skin files and see what fields are being used with the .rms tag. Until we know for sure, and given you seem to have made changes to the skin, we will likely be chasing our tails.

Gary

[Rahul Bedi]

Thanks Gary.  

Charges were done about a month ago but have seen no errors till this episode.  I will do a diff with the original skin files and post it here.  

Thanks.

- Rahul

--

[Rahul Bedi]

I found rms used in the statictics.ico and hilo.ico files in the Seasons directory. removed those segments. Diff as follows:

$ diff statistics.14092018.inc statistics.inc

134,140d133

<         <tr>

<           <td class="label">$obs.label.wind_rms</td>

<           <td class="units">$unit.label.wind</td>

<           #for $archive in $archive_data

<           <td class="data new_row">$archive.wind.rms.formatted</td>

<           #end for

<         </tr>

$ diff hilo.inc hilo.14092018.inc

132a133,140

>       <tr>

>         <td class="label">$obs.label.wind_rms</td>

>         #for $archive in $archive_data

>         <td class="data new_row hilo_$archive[0]">

>           $archive[1].wind.rms.formatted</td>

>         #end for

>         <td class="units">$unit.label.wind</td>

>       </tr>

Apart from this any file referring to rms were exactly the same as the original install. Attaching skin.conf with this message.

As expected the problem is gone but rms is not getting reported. Which means the issue still stays.

Could you help me with what fields in the archive are referred to by rms ? I can try to manually check for any value discrepancy.

Thanks.

- Rahul

[Thomas Keffer]

You're on the right track: there must be an bad value in your database. Did you, by any chance, use a visual sql editor? They often put null strings in the database.

In any case, the field 'wsquaresum' and 'sumtime' are used by rms. So, try this:

sqlite3 /var/lib/weewx/weewx.sdb

sqlite> SELECT datetime(dateTime, 'unixepoch','localtime'), wsquaresum, sumtime FROM archive_day_wind WHERE dateTime > 1533081600;

This will show all values from (roughly) 1-August (about 45 values). If the bad value is before that, we'll have to use a different strategy, rather than go looking for a needle in a haystack.

-tk

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Rahul Bedi]

Here is what I get:

2018-08-02 00:00:00|2851473.0|86400

2018-08-03 00:00:00|2139447.0|86400

2018-08-04 00:00:00|2867211.0|86400

2018-08-05 00:00:00|3306195.0|86400

2018-08-06 00:00:00|3167394.0|86400

2018-08-07 00:00:00|2924283.0|86400

2018-08-08 00:00:00|2255208.0|86400

2018-08-09 00:00:00|2797065.0|86400

2018-08-10 00:00:00|2693187.0|86400

2018-08-11 00:00:00|2981601.0|86400

2018-08-12 00:00:00|3869136.0|86400

2018-08-13 00:00:00|3632787.0|86400

2018-08-14 00:00:00|2651829.0|86400

2018-08-15 00:00:00|2104410.0|86400

2018-08-16 00:00:00|3682674.0|86400

2018-08-17 00:00:00|3499992.0|86400

2018-08-18 00:00:00|3855531.0|86400

2018-08-19 00:00:00|3875121.0|86400

2018-08-20 00:00:00|3915861.0|86400

2018-08-21 00:00:00|3523719.0|86400

2018-08-22 00:00:00|1637909.95179334|85620

2018-08-23 00:00:00|506702.215048778|25380

2018-08-24 00:00:00|523285.18735593|101340

2018-08-25 00:00:00|2047231.90482286|92280

2018-08-26 00:00:00|2207680.59885907|85800

2018-08-27 00:00:00|2112653.43729687|78000

2018-08-28 00:00:00|1927695.38113441|85200

2018-08-29 00:00:00|1696867.80614767|76740

2018-08-30 00:00:00|1892572.6564943|85500

2018-08-31 00:00:00|1669207.56530998|86100

2018-09-01 00:00:00|1882876.90336366|85080

2018-09-02 00:00:00|1552040.97205686|86100

2018-09-03 00:00:00|1550916.37957731|86100

2018-09-04 00:00:00|1093704.34743737|85740

2018-09-05 00:00:00|1688380.4274756|86100

2018-09-06 00:00:00|2349485.94738889|86100

2018-09-07 00:00:00|2676307.36370007|85560

2018-09-08 00:00:00|2198034.86178985|86100

2018-09-09 00:00:00|-7107216.29856356|-302280

2018-09-10 00:00:00|288713.672694377|85560

2018-09-11 00:00:00|351094.880509644|85860

2018-09-12 00:00:00|441249.222795585|85500

2018-09-13 00:00:00|60033.1634844758|85620

2018-09-14 00:00:00|21222.2555021576|62820

I am guessing that the negative values above may be the culprit. How are these populated in the archive_day_wind table or should I just edit these values out ?

- Rahul

[Thomas Keffer]

I think the thing to do is to figure out how a negative value got in there, fix that, then rebuild the daily summaries.

One possibility is that a record in your database has a negative value for 'interval' --- the time length of an archive record. Let's check that.

sqlite3 /var/lib/weewx/weewx.sdb

sqlite> SELECT datetime(dateTime, 'unixepoch','localtime'), interval FROM archive WHERE interval<0;

-tk

[Rahul Bedi]

Yup found a record with -ve interval, changed that to 5 as thats what I am using. 

And problem fixed !!

Thanks a lot.

Not sure how this bad data entered into the db.

Will be great if there is utility that finds such errors in the db and flags them.

- Rahul

[Thomas Keffer]

Good.

Looking through the WMR200 driver code (I didn't write it), it appears that a positive value for 'interval' is not guaranteed. I'll ask the author to take a look at it.

-tk

[Chris Manton]

I suppose if the archived packets are presented out of order there could be a negative interval...

I have not seen this but the wmr200 is full of wonderful surprises.

One potential drive-by fix is to drop all packets satisfying this failing condition...here is one untested fix:

$ git diff

diff --git a/bin/weewx/drivers/wmr200.py b/bin/weewx/drivers/wmr200.py

index eaab5a14..62b743e0 100644

--- a/bin/weewx/drivers/wmr200.py

+++ b/bin/weewx/drivers/wmr200.py

@@ -1943,7 +1943,13 @@ class WMR200(weewx.drivers.AbstractDevice):

                 timestamp_packet_interval = timestamp_packet_current \

                         - timestamp_packet_previous

-                if pkt.timestamp_record() > (timestamp_packet_previous

+                if timestamp_packet_interval < 0:

+                    loginf(('genStartup() Discarding received archive'

+                        ' record that presented out-of-order'

+                        ' current_threshold:%s previous_timestamp:%s')

+                        % (weeutil.weeutil.timestamp_to_string(pkt.timestamp_record())

+                            weeutil.weeutil.timestamp_to_string(timestamp_packet_previous)))

+                elif pkt.timestamp_record() > (timestamp_packet_previous

                                              + self._archive_threshold):

                     loginf(('genStartup() Discarding received archive'

                             ' record exceeding archive interval cnt:%d'

[Thomas Keffer]

Thanks, Chris

I worry that this approach doesn't cover the case where the "previous" record has, say, a timestamp far in the future. This patch would cause all of the remaining records to get thrown out, but the "future" record to be kept.

In any case, it's way better than what we have, which allows negative intervals.

-tk

[Thomas Keffer]

Implemented in commit ea9fe13

-tk

[Rahul Bedi]

On the topics of negative numbers, I have also realized that in cases where I shutdown weewx running on Pi, unplug the USB cable running to the WMR200, plug it back and restart the weewx, sometimes I see -ve value for rain getting populated in the database which I have to manually remove. Looks like WMR200 is certainly an interesting WS. Is there a way to guard against such erroneous records ?

Thanks.

- Rahul

[Thomas Keffer]

What's a "-ve" value?

-tk

[Rahul Bedi]

Negative value. 

Thanks.

- Rahul

[Thomas Keffer]

In that case, just use the [StdQC] service. 

-tk