Multiple errors in weewx 3.8.2

[Jonis Maurin Ceará]

Hi.

I have weewx 3.8.1 running on my RPi for almost one year. Last week my archive page stoped working and looking for any problem, I found some problems in database query....but looking for solutions in this group, I found that was related to saving time here in Brazil, which changed to one week later than usual.

After waiting this one week, problem persist. The problem is that 'wind' column does not exist in database (Mysql or sqlite).

After creating this field (as double), I got another error: Invalid aggregation type 'gustdir'

What could be wrong? I'm using MySQL.

What I've done: remove of weewx package (I'm using .deb on raspberry pi with debian 9.4), remove all databases and files (including /var/lib/weewx and /usr/share/weex, /etc/weex) everything.....and reinstall package, leave with default configuration and simulator. Weewx create new databases (tested with sqlite and mysql), but still with same error on both tests! 

This is the first error and the second (after creating wind column).

cheetahgenerator: Generate failed with exception '<class 'weedb.NoColumnError'>'
Nov  5 10:05:19 SkyWeather weewx[30088]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Standard/week.html.tmpl
Nov  5 10:05:19 SkyWeather weewx[30088]: cheetahgenerator: **** Reason: no such column: wind
Nov  5 10:05:19 SkyWeather weewx[30088]: ****  Traceback (most recent call last):
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 330, in generate
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      print >> _file, compiled_template
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1005, in __str__
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      rc = getattr(self, mainMethName)()
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "_etc_weewx_skins_Standard_week_html_tmpl.py", line 949, in respond
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "_etc_weewx_skins_Standard_week_html_tmpl.py", line 245, in __errorCatcher26
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "<string>", line 1, in <module>
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/tags.py", line 329, in __getattr__
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      return self._do_query(aggregate_type)
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/tags.py", line 343, in _do_query
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      val=val, **self.option_dict)
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/wxmanager.py", line 60, in getAggregate
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      return weewx.manager.DaySummaryManager.getAggregate(self, timespan, obs_type, aggregateType, **option_dict)
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/manager.py", line 1249, in getAggregate
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      **option_dict)
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/manager.py", line 458, in getAggregate
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      _row = self.getSql(select_stmt % interpolate_dict)
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/manager.py", line 394, in getSql
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      _cursor.execute(sql, sqlargs)
Nov  5 10:05:19 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weedb/sqlite.py", line 41, in guarded_fn
Nov  5 10:05:19 SkyWeather weewx[30088]: ****      raise weedb.NoColumnError(e)
Nov  5 10:05:19 SkyWeather weewx[30088]: ****  NoColumnError: no such column: wind

Second:

Nov  5 10:25:16 SkyWeather weewx[30611]: manager: Added record 2018-11-05 10:25:00 -02 (1541420700) to database 'weewx'
Nov  5 10:25:16 SkyWeather weewx[30611]: manager: Added record 2018-11-05 10:25:00 -02 (1541420700) to daily summary in 'weewx'
Nov  5 10:25:18 SkyWeather weewx[30611]: cheetahgenerator: Generate failed with exception '<class 'weewx.ViolatedPrecondition'>'
Nov  5 10:25:18 SkyWeather weewx[30611]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Standard/week.html.tmpl
Nov  5 10:25:18 SkyWeather weewx[30611]: cheetahgenerator: **** Reason: Invalid aggregation type 'gustdir'
Nov  5 10:25:18 SkyWeather weewx[30611]: ****  Traceback (most recent call last):
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 330, in generate
Nov  5 10:25:18 SkyWeather weewx[30611]: ****      print >> _file, compiled_template
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1005, in __str__
Nov  5 10:25:18 SkyWeather weewx[30611]: ****      rc = getattr(self, mainMethName)()
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "_etc_weewx_skins_Standard_week_html_tmpl.py", line 952, in respond
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "_etc_weewx_skins_Standard_week_html_tmpl.py", line 252, in __errorCatcher27
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "<string>", line 1, in <module>
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "/usr/share/weewx/weewx/tags.py", line 329, in __getattr__
Nov  5 10:25:18 SkyWeather weewx[30611]: ****      return self._do_query(aggregate_type)
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "/usr/share/weewx/weewx/tags.py", line 343, in _do_query
Nov  5 10:25:18 SkyWeather weewx[30611]: ****      val=val, **self.option_dict)
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "/usr/share/weewx/weewx/wxmanager.py", line 60, in getAggregate
Nov  5 10:25:18 SkyWeather weewx[30611]: ****      return weewx.manager.DaySummaryManager.getAggregate(self, timespan, obs_type, aggregateType, **option_dict)
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "/usr/share/weewx/weewx/manager.py", line 1249, in getAggregate
Nov  5 10:25:18 SkyWeather weewx[30611]: ****      **option_dict)
Nov  5 10:25:18 SkyWeather weewx[30611]: ****    File "/usr/share/weewx/weewx/manager.py", line 449, in getAggregate
Nov  5 10:25:18 SkyWeather weewx[30611]: ****      raise weewx.ViolatedPrecondition("Invalid aggregation type '%s'" % aggregate_type)
Nov  5 10:25:18 SkyWeather weewx[30611]: ****  ViolatedPrecondition: Invalid aggregation type 'gustdir'
Nov  5 10:25:20 SkyWeather weewx[30611]: cheetahgenerator: Generated 13 files for report StandardReport in 3.53 seconds
Nov  5 10:25:21 SkyWeather weewx[30611]: imagegenerator: Generated 12 images for StandardReport in 1.19 seconds
Nov  5 10:25:21 SkyWeather weewx[30611]: copygenerator: copied 0 files to /var/www/html/weewx

[Andrew Milner]

there is no wind column as such in the database

see this thread for a description of wind and how it may be used as part of daily summary

https://groups.google.com/forum/?fromgroups#!searchin/weewx-user/wind$20column%7Csort:date/weewx-user/siTLWVi_fLM/hMwqeKygBwAJ

- so introducing a wind column was probably not the correct thing to do in the first place.  You probably need to go back to the original problem and solve that one

[Thomas Keffer]

Hello, Jonis

You say you are using MySQL, but your first error was actually raised by the sqlite driver. Check weewx.conf and make sure it is using the database you think it is using.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonis Maurin Ceará]

Tks Andrew, I have re-installed again and keept as original database schema. But it doesn't make sense, unless the standard skin is not even working anymore?!

As I said, I'm using default installation with default configuration/skin/simulation driver. Tried with sqlite and Mysql, both smae errors. Standard skin is broken?!

[Jonis Maurin Ceará]

Thomas, I mixed my answers...but is not config mistake. I've made tests with both drivers, sqlite and MySQL, both has the same problem/error (with their corresponding errors/drivers).

[gjr80]

This is unlikely to be a MySQL or SQLIte error and it definitely is not due to a missing wind or gustdir field. The original missing wind field error is possibly because your archive has some future dated records which causes some of the aggregates to misbehave. To rule this out we need to see some info about your database. The easiest way to do this without explaining a number of SQL commands is for you to post the output of wee_debug (http://weewx.com/docs/utilities.htm#wee_debug_utility). Just make sure you check the output before positing to ensure any passwords, logins or other sensitive data has been obfuscated.

Gary

[Jonis Maurin Ceará]

Sure!

Log attached

I tought about that....daylight saving started yesterday (should be started a week ago, when problem started, but was delayed one week by our president).

The only thing that I don't understand is that even with empty database fresh created by weewx, the problem persist! 

In this debug, I'm using my old database with my old data saved (but without any aditional field, like 'wind', just original schema)

Em seg, 5 de nov de 2018 às 12:41, gjr80 <gjrod...@gmail.com> escreveu:

This is unlikely to be a MySQL or SQLIte error and it definitely is not due to a missing wind or gustdir field. The original missing wind field error is possibly because your archive has some future dated records which causes some of the aggregates to misbehave. To rule this out we need to see some info about your database. The easiest way to do this without explaining a number of SQL commands is for you to post the output of wee_debug (http://weewx.com/docs/utilities.htm#wee_debug_utility). Just make sure you check the output before positing to ensure any passwords, logins or other sensitive data has been obfuscated.

Gary

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/llOh8p8emYY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[gjr80]

That report indicates there are no future dated records. Just out of interest what s the content of the file that the fileparse driver is reading, /var/weather.data, it doesn't have a field named dateTime does it?

Gary

[Jonis Maurin Ceará]

Content of /tmp/weather.data:

rainRate=0.0

windSpeed=10.0466685247

windGust=16.3395133004

pressure=28.0709227

outTemp=84.2

outHumidity=71.1999969482

windDir=0.0

UV=1.59

VIS=554

But the problem is that this errors ocours even with simulator on fresh install....

Em seg, 5 de nov de 2018 às 13:28, gjr80 <gjrod...@gmail.com> escreveu:

That report indicates there are no future dated records. Just out of interest what s the content of the file that the fileparse driver is reading, /var/weather.data, it doesn't have a field named dateTime does it?

Gary

[gjr80]

On Tuesday, 6 November 2018 01:57:22 UTC+10, Jonis Maurin Ceará wrote:

Content of /tmp/weather.data:

rainRate=0.0

windSpeed=10.0466685247

windGust=16.3395133004

pressure=28.0709227

outTemp=84.2

outHumidity=71.1999969482

windDir=0.0

UV=1.59

VIS=554

OK, so dateTime is not being overridden by your source data (for fileparse driver the default is for the dateTime field to be set from the system time).

But the problem is that this errors ocours even with simulator on fresh install....

There must be more to this. What do you mean by 'simulator on fresh install'. A fresh install of weeWX using the simulator I presume, but what about your database? You were using MySQL so a new weeWX install does not necessarily mean a clean database, whereas a new install using SQLite does erase the database. My apologies if your situation is very clear to you, but you have used two drivers, two different database engines and two versions of weeWX so it is not very clear to me exactly what your current setup is.

Gary

[Jonis Maurin Ceará]

Ok, let me explain:

1) Remove all packages and files relates to weewx (including all dir and subdirs on /etc/weewx, /usr/share/weewx, /var/lib/weewx or /usr/lib/weewx, don't rememebr now). Everything

2) Install from .deb package and accept default settings (name, location, driver=simulation, etc...)

3) Just start, done!

When you access webpage, loads fine.....but still the same error in logfile, wind column not found (of course, the error is using sqlite, and not mysqllite like I posted).

Second attempt:

1) Change archive database to mysql instead of sqlite.

2) Re-start weewx. Tables are created and archive page is created

3) Still the same problem on log (wind column), but information on log's say 'mysql' driver, and not sqlite.

And then with fileparser driver, with exactly same results....

I found something related to brazilian daylight saving time, but I lost the link of thread and also I don't knwo where to put suggested user code (is modification of core code, I think).

--

[tomn...@frontier.com]

FWIW, this is a recent install (for all the time periods):

cer8000_/home/weewx/weewx-3.8.2/skins/Standard> grep gustdir * | & grep tmpl

index.html.tmpl:                  $day.wind.max from $day.wind.gustdir at $day.wind.maxtime

Chris

[gjr80]

Ok, the problem is that if you are starting from a clean install (including database) running the simulator after a daylight savings change the timezone/daylight savings state of your system is irrelevant, all times/timestamps are drawn from the system clock and if it is not changing (since the daylight saving cutover was in the past) then daylight saving cannot be the cause of the error.

Do you still have the logs on your system from when you did the clean install using simulator and SQLite? If so can you post the log from when weeWX first started after that install through until the error appeared. If you don't have the logs then please do a clean install using the simulator and SQLite and provide the log showing weeWX startup through until the error occurs.

Gary

[Jonis Maurin Ceará]

Yes, logs from fresh install yesterday.

Nov  5 10:02:31 SkyWeather weewx[30084]: engine: Initializing weewx version 3.8.2

Nov  5 10:02:31 SkyWeather weewx[30084]: engine: Using Python 2.7.13 (default, Sep 26 2018, 18:42:22) #012[GCC 6.3.0 20170516]

Nov  5 10:02:31 SkyWeather weewx[30084]: engine: Platform Linux-4.14.71-v7+-armv7l-with-debian-9.4

Nov  5 10:02:31 SkyWeather weewx[30084]: engine: Locale is 'en_GB.UTF-8'

Nov  5 10:02:31 SkyWeather weewx[30084]: engine: pid file is /var/run/weewx.pid

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Using configuration file /etc/weewx/weewx.conf

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Loading station type Simulator (weewx.drivers.simulator)

Nov  5 10:02:31 SkyWeather weewx[30073]: Starting weewx weather system: weewx.

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: StdConvert target unit is 0x1

Nov  5 10:02:31 SkyWeather weewx[30088]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware

Nov  5 10:02:31 SkyWeather weewx[30088]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Archive will use data binding wx_binding

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Record generation will be attempted in 'hardware'

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Using archive interval of 300 seconds (specified in weewx configuration)

Nov  5 10:02:31 SkyWeather weewx[30088]: manager: Created and initialized table 'archive' in database 'weewx.sdb'

Nov  5 10:02:31 SkyWeather weewx[30088]: manager: Created daily summary tables

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Using binding 'wx_binding' to database 'weewx.sdb'

Nov  5 10:02:31 SkyWeather weewx[30088]: manager: Starting backfill of daily summaries

Nov  5 10:02:31 SkyWeather weewx[30088]: restx: StationRegistry: Registration not requested.

Nov  5 10:02:31 SkyWeather weewx[30088]: restx: Wunderground: Posting not enabled.

Nov  5 10:02:31 SkyWeather weewx[30088]: restx: PWSweather: Posting not enabled.

Nov  5 10:02:31 SkyWeather weewx[30088]: restx: CWOP: Posting not enabled.

Nov  5 10:02:31 SkyWeather weewx[30088]: restx: WOW: Posting not enabled.

Nov  5 10:02:31 SkyWeather weewx[30088]: restx: AWEKAS: Posting not enabled.

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Starting up weewx version 3.8.2

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Clock error is -0.30 seconds (positive is fast)

Nov  5 10:02:31 SkyWeather weewx[30088]: engine: Starting main packet loop.

Nov  5 10:05:16 SkyWeather weewx[30088]: manager: Added record 2018-11-05 10:05:00 -02 (1541419500) to database 'weewx.sdb'

Nov  5 10:05:16 SkyWeather weewx[30088]: manager: Added record 2018-11-05 10:05:00 -02 (1541419500) to daily summary in 'weewx.sdb'

Nov  5 10:05:19 SkyWeather weewx[30088]: cheetahgenerator: Generate failed with exception '<class 'weedb.NoColumnError'>'

Nov  5 10:05:23 SkyWeather weewx[30088]: cheetahgenerator: Generated 13 files for report StandardReport in 6.81 seconds

Nov  5 10:05:31 SkyWeather weewx[30088]: imagegenerator: Generated 49 images for StandardReport in 7.38 seconds

Nov  5 10:05:31 SkyWeather weewx[30088]: copygenerator: copied 9 files to /var/www/html/weewx

Nov  5 10:10:16 SkyWeather weewx[30088]: manager: Added record 2018-11-05 10:10:00 -02 (1541419800) to database 'weewx.sdb'

Nov  5 10:10:16 SkyWeather weewx[30088]: manager: Added record 2018-11-05 10:10:00 -02 (1541419800) to daily summary in 'weewx.sdb'

Nov  5 10:10:17 SkyWeather weewx[30088]: cheetahgenerator: Generate failed with exception '<class 'weedb.NoColumnError'>'

Nov  5 10:10:17 SkyWeather weewx[30088]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Standard/week.html.tmpl

Nov  5 10:10:17 SkyWeather weewx[30088]: cheetahgenerator: **** Reason: no such column: wind

Nov  5 10:10:17 SkyWeather weewx[30088]: ****  Traceback (most recent call last):

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 330, in generate

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      print >> _file, compiled_template

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1005, in __str__

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      rc = getattr(self, mainMethName)()

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "_etc_weewx_skins_Standard_week_html_tmpl.py", line 949, in respond

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "_etc_weewx_skins_Standard_week_html_tmpl.py", line 245, in __errorCatcher26

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "<string>", line 1, in <module>

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/tags.py", line 329, in __getattr__

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      return self._do_query(aggregate_type)

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/tags.py", line 343, in _do_query

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      val=val, **self.option_dict)

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/wxmanager.py", line 60, in getAggregate

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      return weewx.manager.DaySummaryManager.getAggregate(self, timespan, obs_type, aggregateType, **option_dict)

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/manager.py", line 1249, in getAggregate

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      **option_dict)

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/manager.py", line 458, in getAggregate

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      _row = self.getSql(select_stmt % interpolate_dict)

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weewx/manager.py", line 394, in getSql

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      _cursor.execute(sql, sqlargs)

Nov  5 10:10:17 SkyWeather weewx[30088]: ****    File "/usr/share/weewx/weedb/sqlite.py", line 41, in guarded_fn

Nov  5 10:10:17 SkyWeather weewx[30088]: ****      raise weedb.NoColumnError(e)

Nov  5 10:10:17 SkyWeather weewx[30088]: ****  NoColumnError: no such column: wind

Nov  5 10:10:18 SkyWeather weewx[30088]: cheetahgenerator: Generated 13 files for report StandardReport in 2.02 seconds

Nov  5 10:10:19 SkyWeather weewx[30088]: imagegenerator: Generated 12 images for StandardReport in 0.98 seconds

Nov  5 10:10:19 SkyWeather weewx[30088]: copygenerator: copied 0 files to /var/www/html/weewx

[gjr80]

I have not forgotten this. Definitely starting with a clean database:

Nov  5 10:02:31 SkyWeather weewx[30088]: manager: Created and initialized table 'archive' in database 'weewx.sdb'
Nov  5 10:02:31 SkyWeather weewx[30088]: manager: Created daily summary tables

Interesting that the only template that fails is week.html.tmpl, all of its friends (index.html.tmpl, month.html.tmpl and year.html.tmpl) also contain wind tags but only week.html.tmpl fails. Suggests the issue is confined to week.wind.xxx.

Can I ask you to run a couple of commands on your weeWX system and post the output. The commands are:

locale
timedatectl

Gary

[Jonis Maurin Ceará]

Sure!

And one more info: this error (wind) happens on all skins.

pi@SkyWeather:~ $ locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=
pi@SkyWeather:~ $ timedatectl
      Local time: Fri 2018-11-09 08:26:49 -02
  Universal time: Fri 2018-11-09 10:26:49 UTC
        RTC time: n/a
       Time zone: America/Sao_Paulo (-02, -0200)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

[gjr80]

I have set up a Stretch VM using the same locale and timezone settings as you running a .deb weeWX install and I cannot reproduce the fault you are experiencing. This really brings me back to the your database. I will instrument one of the weeWX files with some extra logging so we can see exactly what is happening when the offending tag(s) is/are called. In the meantime can you confirm exactly what version of weeWX you are now running on the problem system (3.8.2 I am guessing) so that I can ensure I give you a compatible version of the file.

Gary

[Jonis Maurin Ceará]

Yes, I'm using this file: weewx_3.8.2-1_all.deb

Currently I'm using with MySQL....the problem persist (wind error in logs), but archive is being generated. http://weather.jonis.com.br/

[Andrew Milner]

The site URL you gave is not running the standard weewx skin, but is instead appears to be using the Belchertown skin and using Highcharts to do the graphing.  Do you get the same 'problems' if you just use the Standard skin which comes with weewx????  What else have you loaded on top of the standard weewx installation????

Belchertown problems and issues should really be referred to the Belchertown author via his GitHub site.

[Jonis Maurin Ceará]

Yes Andrew, same problem. Like I said before, I've tried with a fresh-new install, with all default configs and no mods, including skins, just default installation and Standard skin/configuration, including default sqlite database. Problem is alwayd the same.

I'm running new skin now because despiste error in log, page is being updated fine.....but problem (wind on log) still.

1) Remove all packages and files relates to weewx (including all dir and subdirs on /etc/weewx, /usr/share/weewx, /var/lib/weewx or /usr/lib/weewx, don't rememebr now). Everything
2) Install from .deb package and accept default settings (name, location, driver=simulation, etc...)
3) Just start, done!
When you access webpage, loads fine.....but still the same error in logfile, wind column not found (of course, the error is using sqlite, and not mysqllite like I posted).
Second attempt:
1) Change archive database to mysql instead of sqlite.
2) Re-start weewx. Tables are created and archive page is created
3) Still the same problem on log (wind column), but information on log's say 'mysql' driver, and not sqlite.
And then with fileparser driver, with exactly same results....

[Thomas Keffer]

Gary: one thing I'm noticing is that the tag DaySummaryManager.getAggregate() is using its superclass version, Manager.getAggregate(), presumably because the boundaries of the query are not falling on midnight.

A clue, perhaps...?

-tk

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[gjr80]

Tom, yes knew that was going on as soon as I saw the error message. Just having a trouble finding out why. Somewhere a timestamp must be missing a day boundary but I haven't yet worked out which. Particularly confusing when a clean database exhibits the same problem immediately. Also odd that week.html.tmpl throws the error but month.html.tmpl and year.html.tmpl do not which suggests week.wind is the culprit. Really want to see some of the timestamps being used to try and work out what is being used and why.

Gary

[Jonis Maurin Ceará]

I'm not sure who is the mais author/developer, but I can give you access to my station. It's in use right now, but is not imperative to keep online 24/7, so you can access and check/test. The only thing is that I'm using my old mysql database....but. like I said before, I've made tests with clean and new database (sqlite and mysql). If you like, you can test and re-create those databases again......just don't delete my old mysql database :)

Em seg, 12 de nov de 2018 às 20:43, gjr80 <gjrod...@gmail.com> escreveu:

Tom, yes knew that was going on as soon as I saw the error message. Just having a trouble finding out why. Somewhere a timestamp must be missing a day boundary but I haven't yet worked out which. Particularly confusing when a clean database exhibits the same problem immediately. Also odd that week.html.tmpl throws the error but month.html.tmpl and year.html.tmpl do not which suggests week.wind is the culprit. Really want to see some of the timestamps being used to try and work out what is being used and why.

Gary

[gjr80]

Tom is the author/developer, the rest of us just help out where/when we can.

I have  attached a version of tags.py with some extra logging to hopefully give us a clue why things are happening. What I would like you to do on your problem install is:

1. rename /usr/share/weewx/weewx/tags.py as tags_orig.py

$ sudo mv /usr/share/weewx/weewx/tags.py /usr/share/weewx/weewx/tags_orig.py

2. download/copy the attached tags.py to /usr/share/weewx/weewx

3. restart weeWX.

4. Post the log from weeWX startup through until the error occurs.

5. Start a MySQL/Mariadb shell and run the following commands:

> USE weewx;
> SELECT dateTime,max FROM archive_day_wind ORDER BY dateTime DESC LIMIT 32;

6. Post the output from the above commands.

To revert the tags.py changes just delete tags.py and rename tags_orig.py back to tags.py and restart weeWX.

Gary

[Jonis Maurin Ceará]

Gary, I've sent logs direct to you.

[gjr80]

Thanks Jonis,

The following log extract captures the error along with a few extra details from tags.py, in particular the timespan being used by the $week.wind.avg call that caused the error

Nov 13 12:31:52 SkyWeather weewx[17639]: ObservationBinder._do_query: timespan=[2018-11-03 00:00:00 -03 (1541214000) -> 2018-11-04 01:00:00 -02 (1541300400)] aggregate_type=avg val=None
Nov 13 12:31:52 SkyWeather weewx[17639]: ObservationBinder._do_query: data_binding=None
Nov 13 12:31:52 SkyWeather weewx[17639]: ObservationBinder._do_query: first_timestamp=1527810600 last_timestamp=1542119400
Nov 13 12:31:52 SkyWeather weewx[17639]: cheetahgenerator: Generate failed with exception '<class 'weedb.NoColumnError'>'
Nov 13 12:31:52 SkyWeather weewx[17639]: cheetahgenerator: **** Ignoring template /etc/weewx/skins/Belchertown/NOAA/NOAA-YYYY-MM.txt.tmpl
Nov 13 12:31:52 SkyWeather weewx[17639]: cheetahgenerator: **** Reason: (1054, "Unknown column 'wind' in 'field list'")
Nov 13 12:31:52 SkyWeather weewx[17639]: ****  Traceback (most recent call last):
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 330, in generate
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      print >> _file, compiled_template
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/lib/python2.7/dist-packages/Cheetah/Template.py", line 1005, in __str__
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      rc = getattr(self, mainMethName)()
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "_etc_weewx_skins_Belchertown_NOAA_NOAA_YYYY_MM_txt_tmpl.py", line 451, in respond
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "_etc_weewx_skins_Belchertown_NOAA_NOAA_YYYY_MM_txt_tmpl.py", line 224, in __errorCatcher23
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "<string>", line 1, in <module>
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/share/weewx/weewx/tags.py", line 332, in __getattr__
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      return self._do_query(aggregate_type)
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/share/weewx/weewx/tags.py", line 350, in _do_query
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      val=val, **self.option_dict)
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/share/weewx/weewx/wxmanager.py", line 60, in getAggregate
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      return weewx.manager.DaySummaryManager.getAggregate(self, timespan, obs_type, aggregateType, **option_dict)
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/share/weewx/weewx/manager.py", line 1249, in getAggregate
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      **option_dict)
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/share/weewx/weewx/manager.py", line 458, in getAggregate
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      _row = self.getSql(select_stmt % interpolate_dict)
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/share/weewx/weewx/manager.py", line 394, in getSql
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      _cursor.execute(sql, sqlargs)
Nov 13 12:31:52 SkyWeather weewx[17639]: ****    File "/usr/share/weewx/weedb/mysql.py", line 48, in guarded_fn
Nov 13 12:31:52 SkyWeather weewx[17639]: ****      raise klass(e)
Nov 13 12:31:52 SkyWeather weewx[17639]: ****  NoColumnError: (1054, "Unknown column 'wind' in 'field list'")

At first glance the timespan appears wrong 00:00:00 -> 00:01:00, but that agrees with the dateTime filed in archive_day_wind (dateTime fields are midnight values calculated by weeWX):

MariaDB [weewx]> SELECT dateTime,max FROM archive_day_wind ORDER BY dateTime DESC LIMIT 32;
+------------+---------------+
| dateTime   | max           |
+------------+---------------+
| 1542074400 | 27.9085344298 |
| 1541988000 | 25.0258600554 |
| 1541901600 | 24.9870900035 |
| 1541815200 | 27.5289972104 |
| 1541728800 | 29.3639762739 |
| 1541642400 | 25.3571126701 |
| 1541556000 | 25.0108380298 |
| 1541469600 | 26.9323996768 |
| 1541383200 | 26.3870808852 |
| 1541300400 | 24.9563264287 |
| 1541214000 | 29.2825768668 |
| 1541127600 |  65.639768948 |

Unfortunately, when weeWX determines whether to use the daily summaries or archive for an aggregate one of the conditions that is checked (in this case) is whether the start and stop timestamps of that timespan are on midnight boundaries. According to the weeWX isMidnight function the timestamp 1541300400 is not a 'midnight'. Thus weeWX determines that it cannot use the daily summaries for $day.wind.avg for 3 November and tries to hit the archive and consequently we get the field not found error.

So that is the mechanism that is occurring in this case, as for a fix that is a little more problematic. Simply rebuilding the daily summaries will have no effect, as weeWX will continue to use 1541300400 as the midnight timestamp on 4 November and at the same time isMidnight will continue to say that 1541300400 is not a 'midnight'. I suspect the issue is tied to the fact that the Brazilian changeover time for daylight saving is midnight and linux/python gets a little confused with the 12:00/01:00 issue. I had hoped that perhaps a linux update tzdata update may solve the issue but my up-to-date Stretch 9.6 VM exhibits the same error when I run the simulator in generator mode over the 3/4 November period. The problem I can't fathom is why has this not happened before, the error is not one that silently disappears in the hour after changeover, this error will continue as long as weeWX is run with data in a database covering 3/4 November. It also doesn't explain why a fresh install suffered the same issue, unless somehow the old database was still being used, though the log extract does not support this. Still confused.





TOM !?


Gary

[Jonis Maurin Ceará]

Yeah.....I thought the same about brazilian saving time change......that's why I've uninstalled everything and cleared all database files so Weewx create a new one on startup.....but as you know, no success.

Maybe this can help?! https://groups.google.com/forum/?hl=pt-BR#!searchin/weewx-user/brazilian|sort:date/weewx-user/ZXGZ1tlw-7k/E-Xwo7hdBAAJ

Looks like the same problem....I just don't know where to put/try this small mod proposed by Guilherme.

[gjr80]

I think I now understand why a fresh install still encounters the same error, weeWX uses a whole pile of different timespans to work out aggregates over a whole pile of different periods, some into the past. If one of those timespans happens to include midnight when daylight savings cuts in then you will get the error. I suspect that if you do a clean install next month error will not occur.

Thanks for the link, I don't remember that but it sounds relevant. Will read through. The important thing here is that this is just a presentation issue, your underlying data is being recorded correctly and is safe.

Gary

PS. When i pulled up the link there were all these posts from 'eu', was wondering who 'eu' was as he had my email address. Realised then I was looking a Brazilianised version of the thread and 'eu' is in fact 'I' or 'me' in Portuguese. Was a bit worried I had some sort of Brazilian issue on my machine :)

[Jonis Maurin Ceará]

oohhhh my mistake! 😀 The link that I've sent includes Brazilian language set.......and yes, 'eu' means "I" ;)

[gjr80]

Yes that link is exactly the same problem. Seems the issue was identified but never fixed. I know that a gambiarra was posted in that thread but I don't think that is really a viable long term solution. As I said earlier this is a presentation issue and not affecting the data going to database so I think we can take a little longer to work through and fix it properly. So that we don't lose track of it this time I have raised issue #356.

Gary

[Jonis Maurin Ceará]

tks! 

:)

Em qua, 14 de nov de 2018 às 11:19, gjr80 <gjrod...@gmail.com> escreveu:

Yes that link is exactly the same problem. Seems the issue was identified but never fixed. I know that a gambiarra was posted in that thread but I don't think that is really a viable long term solution. As I said earlier this is a presentation issue and not affecting the data going to database so I think we can take a little longer to work through and fix it properly. So that we don't lose track of it this time I have raised issue #356.

Gary

--

[Andrew Milner]

Gary - FYI - there are only 3 countries who change clocks back at midnight - Brazil, Cuba and I think one other - everyone else changes at 0200 or 0100, so on your reasoning would not have any issues.  A specific fudge may therefore not be too unreasonable an approach.