Limiting weewx logging only to error/fail

947 views
Skip to first unread message

Jarmo Seppänen

unread,
Apr 13, 2016, 11:01:32 AM4/13/16
to weewx-user
Hi,

Just wondering would it be possible to limit weewx logging only to errors or failures?
Current log level is 0 which produces some 15 rows which are not too informative when everything is working.

Apr 13 17:50:16 raspberrypi weewx[400]: manager: added record 2016-04-13 17:50:16 EEST (1460559016) to database 'cmon'
Apr 13 17:50:16 raspberrypi weewx[400]: manager: added record 2016-04-13 17:50:16 EEST (1460559016) to daily summary in 'cmon'
Apr 13 17:50:16 raspberrypi weewx[400]: manager: added record 2016-04-13 17:50:00 EEST (1460559000) to database 'weewx'
Apr 13 17:50:16 raspberrypi weewx[400]: manager: added record 2016-04-13 17:50:00 EEST (1460559000) to daily summary in 'weewx'
Apr 13 17:50:16 raspberrypi weewx[400]: engine: garbage collected 2866 objects
Apr 13 17:50:17 raspberrypi weewx[400]: restx: Wunderground-PWS: Published record 2016-04-13 17:50:00 EEST (1460559000)
Apr 13 17:50:18 raspberrypi weewx[400]: restx: OWM: Published record 2016-04-13 17:50:00 EEST (1460559000)
Apr 13 17:50:20 raspberrypi weewx[400]: cheetahgenerator: Generated 1 files for report simple in 3.34 seconds
Apr 13 17:50:29 raspberrypi weewx[400]: genimages: Generated 16 images for simple in 8.76 seconds
Apr 13 17:50:29 raspberrypi weewx[400]: cheetahgenerator: Generated 1 files for report cmon in 0.19 seconds
Apr 13 17:52:45 raspberrypi weewx[400]: genimages: Generated 16 images for cmon in 136.35 seconds
Apr 13 17:52:47 raspberrypi weewx[400]: historygenerator.pyc: Generated 4 tables in 0.98 seconds
Apr 13 17:52:47 raspberrypi weewx[400]: cheetahgenerator: Generated 1 files for report HTMLPages in 1.21 seconds
Apr 13 17:52:47 raspberrypi weewx[400]: reportengine: copied 0 files to /var/tmp
Apr 13 17:53:31 raspberrypi weewx[400]: reportengine: ftp'd 36 files in 44.29 seconds

Of course I have setup raspi to log to ramdisk, but it would be easier to pinpoint the possible failures if logging level could be adjusted.

Tried to search if there were any previous discussion but search did not produce any hits.

mwall

unread,
Apr 13, 2016, 11:12:59 AM4/13/16
to weewx-user
On Wednesday, April 13, 2016 at 11:01:32 AM UTC-4, Jarmo Seppänen wrote:
Hi,

Just wondering would it be possible to limit weewx logging only to errors or failures?


you can turn off the generator log messages using log_success/log_failure.  for example, in skin.conf:

[CheetahGenerator]
    log_success = False # default is True
    log_failure = True # default is True

uploaders (restful services) have similar controls, but these are in weewx.conf:

http://weewx.com/docs/usersguide.htm#[StdRESTful]

i highly recommend the use of logwatch:

https://github.com/weewx/weewx/wiki/logwatch

for pi or other puny computer, be sure to keep your log retention small (e.g., 3) in logrotate

m

vince

unread,
Apr 13, 2016, 12:00:49 PM4/13/16
to weewx-user
On Wednesday, April 13, 2016 at 8:12:59 AM UTC-7, mwall wrote:
On Wednesday, April 13, 2016 at 11:01:32 AM UTC-4, Jarmo Seppänen wrote:
Hi,

Just wondering would it be possible to limit weewx logging only to errors or failures?


you can turn off the generator log messages using log_success/log_failure.  for example, in skin.conf:

[CheetahGenerator]
    log_success = False # default is True
    log_failure = True # default is True

uploaders (restful services) have similar controls, but these are in weewx.conf:

http://weewx.com/docs/usersguide.htm#[StdRESTful]



Great stuff - is there any plan/way to have a global log_success or global log_failure that the sections can supersede perhaps, rather than requiring setting log_success=false in a bunch of places ?

 
i highly recommend the use of logwatch:

https://github.com/weewx/weewx/wiki/logwatch



The wiki page is nice.  It would be nice to show an example report.

Just installed logwatch to try it - wow.  Pretty slick. Looks like I have an error to look into....


 ################### Logwatch 7.4.0 (05/02/12) ####################
        Processing Initiated: Wed Apr 13 08:52:20 2016
        Date Range Processed: yesterday
                              ( 2016-Apr-12 )
                              Period is day.
        Detail Level of Output: 0
        Type of Output/Format: stdout / text
        Logfiles for Host: debian
 ##################################################################

 --------------------- weewx Begin ------------------------

 average station clock skew: 2.955
   min: 0 max: 4.99 samples: 6

 counts:
   archive: records added                          2304
   engine: garbage collected                      19990
   filegenerator: files generated                 10656
   forecast: downloads                               16
   forecast: prunings                                17
   forecast: records generated                     2235
   genimages: images generated                     9305
   restful: records published                       720

 forecast downloads:
   NWS                                                8
   WU                                                 8

 forecast prunings:
   NWS                                                8
   WU                                                 8
   Zambretti                                          1

 forecast records generated:
   NWS                                              314
   WU                                              1920
   Zambretti                                          1

 uploads:
   CWOP                                             144
   PWSWeather                                       288
   Wunderground-PWS                                 288


 unmatched lines:
   Apr 12 07:35:15 debian weewx[13199]: wxdata: archive record, delta 15.9023060799
   Apr 12 07:36:34 debian weewx[13199]: imageStackedWindRose: Generated 1 images in 0.42 seconds
   Apr 12 07:36:43 debian weewx[13199]: reportengine: Caught unrecoverable exception in generator weewx.reportengine.RsyncGenerator
   Apr 12 07:36:43 debian weewx[13199]:         ****  global name 'log_success' is not defined
   Apr 12 07:36:43 debian weewx[13199]:         ****  Traceback (most recent call last):
   Apr 12 07:36:43 debian weewx[13199]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 140, in run
   Apr 12 07:36:43 debian weewx[13199]:         ****      obj.start()
   Apr 12 07:36:43 debian weewx[13199]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 170, in start
   Apr 12 07:36:43 debian weewx[13199]:         ****      self.run()
   Apr 12 07:36:43 debian weewx[13199]:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 263, in run
                 (and so on....)

 
 

mwall

unread,
Apr 13, 2016, 6:42:38 PM4/13/16
to weewx-user
On Wednesday, April 13, 2016 at 12:00:49 PM UTC-4, vince wrote:
Great stuff - is there any plan/way to have a global log_success or global log_failure that the sections can supersede perhaps, rather than requiring setting log_success=false in a bunch of places ?

vince

unread,
Apr 13, 2016, 6:52:00 PM4/13/16
to weewx-user
This is why we shop here :-)
Thanks.... 

mwall

unread,
Apr 13, 2016, 6:54:48 PM4/13/16
to weewx-user
On Wednesday, April 13, 2016 at 12:00:49 PM UTC-4, vince wrote:
Just installed logwatch to try it - wow.  Pretty slick. Looks like I have an error to look into....

vince, are you running weewx 3.4.0?  it looks like you are getting bit by this bug (fixed in 3.5.0):

https://github.com/weewx/weewx/commit/081fcc92acaa8e075119599a8befeed8e061557d

m

vince

unread,
Apr 13, 2016, 7:04:40 PM4/13/16
to weewx-user
yup - 3.4.0 indeed - let me upgrade....

 

Daniel Jönsson

unread,
Nov 21, 2017, 4:36:10 PM11/21/17
to weewx-user
I cant seem to get it to work. And I cant find it in the user guide (http://www.weewx.com/docs/usersguide.htm) were to put it.
I would also very much like to only have errors in my log.

mwall

unread,
Nov 21, 2017, 5:13:45 PM11/21/17
to weewx-user


On Tuesday, November 21, 2017 at 4:36:10 PM UTC-5, Daniel Jönsson wrote:
I cant seem to get it to work. And I cant find it in the user guide (http://www.weewx.com/docs/usersguide.htm) were to put it.
I would also very much like to only have errors in my log.


commit 6b461c9 applies to reports

use it like this:

[StdReport]
  log_success = False
  log_failure = True

then optionally specialize for individual reports.  for example, to log success for StandardReport but not for others:

[StdReport]
  log_success = False
  log_failure = True
    ...
  [[StandardReport]]
    log_success = True
    ...
  [[AnotherReport]]
    ...

this pattern should also work for uploaders:

[StdRESTful]
  log_succes = False
  log_failure = True

we probably should do it for the archive service as well, which would probably indicate that one should be able to specify log_success and log_failure at the root of weewx.conf then override for each service as necessary.

m

Daniel Jönsson

unread,
Nov 22, 2017, 3:02:53 AM11/22/17
to weewx-user
Thank you for quick reply. 

This was exactly how I understood it to be when I read up on the commit on github, and this is what I have been trying to do. But it does not effect the logging.
I am running version 3.6.1, and if I understand it correct this was implemented in 3.6.0, so it should work but isn't.

I have tried to put it under [StdReport] and all reports their under, and under [StdRESTful]. No matter where I put it, it does absolutely nothing to lessen the output to syslog.

mwall

unread,
Nov 23, 2017, 7:08:09 PM11/23/17
to weewx-user
On Wednesday, November 22, 2017 at 3:02:53 AM UTC-5, Daniel Jönsson wrote:

This was exactly how I understood it to be when I read up on the commit on github, and this is what I have been trying to do. But it does not effect the logging.
I am running version 3.6.1, and if I understand it correct this was implemented in 3.6.0, so it should work but isn't.

I have tried to put it under [StdReport] and all reports their under, and under [StdRESTful]. No matter where I put it, it does absolutely nothing to lessen the output to syslog.

could you post some log output and highlight the messages that you think should not be emitted?

also post a sanitized copy of the weewx.conf

m

Andrew Milner

unread,
Nov 26, 2017, 12:54:55 AM11/26/17
to weewx-user
Just updated to 3.8 and find that logwatch is logging as unmatched lines the routine entries from manager, copygenerator, imagegenerator:

 unmatched lines:
   Nov 25 08:45:49 RPi3 weewx[29522]: Stopping weewx weather system: weewx..
   Nov 25 08:59:51 RPi3 weewx[541]: engine: Platform Linux-4.9.35-v7+-armv7l-with-debian-8.0
   Nov 25 08:59:51 RPi3 weewx[504]: Starting weewx weather system: weewx.
   Nov 25 08:59:51 RPi3 weewx[701]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
   Nov 25 08:59:51 RPi3 weewx[701]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
   Nov 25 08:59:53 RPi3 weewx[701]: engine: Caught database OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")
   Nov 25 08:59:53 RPi3 weewx[701]:     ****  Waiting 2 minutes then retrying...
   Nov 25 09:01:52 RPi3 weewx[701]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
   Nov 25 09:01:52 RPi3 weewx[701]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
   Nov 25 09:01:53 RPi3 weewx[701]: manager: Starting backfill of daily summaries
   Nov 25 09:01:53 RPi3 weewx[701]: manager: Daily summaries up to date
   Nov 25 09:01:53 RPi3 weewx[701]: restx: Wunderground-PWS: Data for station ILIMASSO5 will be posted
   Nov 25 09:01:53 RPi3 weewx[701]: restx: PWSweather: Posting not enabled.
   Nov 25 09:01:53 RPi3 weewx[701]: restx: CWOP: Posting not enabled.
   Nov 25 09:01:53 RPi3 weewx[701]: restx: WOW: Posting not enabled.
   Nov 25 09:01:53 RPi3 weewx[701]: restx: AWEKAS: Posting not enabled.
   Nov 25 09:23:04 RPi3 weewx[2102]: Stopping weewx weather system: weewx..
   Nov 25 09:24:26 RPi3 weewx[2187]: engine: Platform Linux-4.9.35-v7+-armv7l-with-debian-8.0
   Nov 25 09:24:26 RPi3 weewx[2187]: engine: Locale is 'en_GB.UTF-8'
   Nov 25 09:24:26 RPi3 weewx[2177]: Starting weewx weather system: weewx.
   Nov 25 09:24:26 RPi3 weewx[2191]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, windrun=prefer_hardware, heatindex=prefer_hardware, maxSolarRad=prefer_hardware, humidex=prefer_hardware, pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, cloudbase=prefer_hardware
   Nov 25 09:24:26 RPi3 weewx[2191]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
   Nov 25 09:24:27 RPi3 weewx[2191]: manager: Starting backfill of daily summaries
   Nov 25 09:24:27 RPi3 weewx[2191]: restx: Wunderground-PWS: Data for station ILIMASSO5 will be posted
   Nov 25 09:24:27 RPi3 weewx[2191]: restx: PWSweather: Posting not enabled.
   Nov 25 09:24:27 RPi3 weewx[2191]: restx: CWOP: Posting not enabled.
   Nov 25 09:24:27 RPi3 weewx[2191]: restx: WOW: Posting not enabled.
   Nov 25 09:24:27 RPi3 weewx[2191]: restx: AWEKAS: Posting not enabled.
   Nov 25 09:25:00 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:25:00 EET (1511594700) to database 'archive'
   Nov 25 09:25:00 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:25:00 EET (1511594700) to daily summary in 'archive'
   Nov 25 09:25:03 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:25:03 EET (1511594703) to database 'mesowx'
   Nov 25 09:25:56 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:25:57 EET (1511594757) to database 'mesowx'
   Nov 25 09:26:50 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:26:50 EET (1511594810) to database 'mesowx'
   Nov 25 09:27:43 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:27:44 EET (1511594864) to database 'mesowx'
   Nov 25 09:28:37 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:28:37 EET (1511594917) to database 'mesowx'
   Nov 25 09:29:30 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:29:31 EET (1511594971) to database 'mesowx'
   Nov 25 09:30:23 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:30:24 EET (1511595024) to database 'mesowx'
   Nov 25 09:30:24 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:30:00 EET (1511595000) to database 'archive'
   Nov 25 09:30:24 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:30:00 EET (1511595000) to daily summary in 'archive'
   Nov 25 09:30:31 RPi3 weewx[2191]: imagegenerator: Generated 12 images for StandardReport in 0.92 seconds
   Nov 25 09:30:31 RPi3 weewx[2191]: copygenerator: copied 14 files to /home/weewx/public_html
   Nov 25 09:30:39 RPi3 weewx[2191]: imagegenerator: Generated 6 images for BigImages in 7.68 seconds
   Nov 25 09:30:42 RPi3 weewx[2191]: imagegenerator: Generated 6 images for SmallImages in 3.20 seconds
   Nov 25 09:31:00 RPi3 weewx[2191]: copygenerator: copied 4 files to /home/weewx/public_html/Bootstrap
   Nov 25 09:31:18 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:31:18 EET (1511595078) to database 'mesowx'
   Nov 25 09:32:11 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:32:12 EET (1511595132) to database 'mesowx'
   Nov 25 09:33:05 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:33:05 EET (1511595185) to database 'mesowx'
   Nov 25 09:34:03 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:34:03 EET (1511595243) to database 'mesowx'
   Nov 25 09:34:56 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:34:57 EET (1511595297) to database 'mesowx'
   Nov 25 09:35:50 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:35:50 EET (1511595350) to database 'mesowx'
   Nov 25 09:35:50 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:35:00 EET (1511595300) to database 'archive'
   Nov 25 09:35:50 RPi3 weewx[2191]: manager: Added record 2017-11-25 09:35:00 EET (1511595300) to daily summary in 'archive'
   Nov 25 09:35:55 RPi3 weewx[2191]: imagegenerator: Generated 12 images for StandardReport in 0.91 seconds
   Nov 25 09:35:55 RPi3 weewx[2191]: copygenerator: copied 0 files to /home/weewx/public_html
   Nov 25 09:36:02 RPi3 weewx[2191]: imagegenerator: Generated 6 images for BigImages in 7.66 seconds
   Nov 25 09:36:05 RPi3 weewx[2191]: imagegenerator: Generated 6 images for SmallImages in 3.20 seconds
   Nov 25 09:36:21 RPi3 weewx[2191]: copygenerator: copied 0 files to /home/weewx/public_html/Bootstrap
   

Have I forgotten to do something on the update, as these lines used to be dropped from my logwatch reports??

Andrew Milner

unread,
Nov 26, 2017, 4:22:49 AM11/26/17
to weewx-user
Problem SOLVED - There were some subtle text changes to the logwatch script with the upgrade (Added instead of added, imagegenerator instead of Genimages) so I needed to copy the logwatch script into /etc/logwatch and add some custom edits I had made to the old script to ignore raw: and historygenerator: . But I think it is now all ok

Any chance of getting setup.py to update the logwatch scripts (and keep custom edits) - or perhaps making the update remind one of what else needs to be checked/changed manually - just a thought .....

gjr80

unread,
Nov 26, 2017, 4:28:24 AM11/26/17
to weewx-user
Andrew, just out of interest what version did you upgrade from? I did not think that 3.8.0 introduced any significant changes to log output.

> Any chance of getting setup.py to update the logwatch scripts (and keep custom edits) - or perhaps
> making the update remind one of what else needs to be checked/changed manually - just a thought .....

Do you want the standard answer....

Gary

Andrew Milner

unread,
Nov 26, 2017, 5:05:16 AM11/26/17
to weewx-user
Gary

I think I was running 3.6.1 - and those were the only two subtle (and they were subtle - a capital A and a change of name) differences I found apart from what I assume to be my own edits.

It seems to me that logging to a separate weewx log file, implementing logrotate and using logwatch should really be part of the 'standard' weewx installation (possibly with a disabling option in weewx.conf/setup) and handled/updated accordingly rather than being 'nice to have' extras obtained via the wiki.

.... and my standard response is "What's a pull request?" as I still haven't got to grips with them or GitHub, and the weewx documentation bible is no help!!!!
Andrew

gjr80

unread,
Nov 26, 2017, 5:23:11 AM11/26/17
to weewx-user
I'll have a look, at the very least we should have updated the logwatch script if the log output was changed. I too run a custom logwatch script (but haven't noticed any 3.8.0 issues) so I know having to change it is annoying. Matthew is the logwatch expert, will have to metaphoricall pick his brains.

As for bibles, you probably want the GitHub bible rather than the weeWX bible. But to make it easier for simple changes involving one file only I am sure a marked up source would be fine.

Gary

Andrew Milner

unread,
Nov 26, 2017, 5:45:01 AM11/26/17
to weewx-user
Gary
The script had been updated correctly, but I had not copied it over into /etc/logwatch (I use setup.py to install), so I only had the two extra lines to add for raw and historygenerator once I'd copied it over.  The problem I reported in this thread was caused by the fact that I had forgotten/not realised I had to do the copy when I did the upgrade.
Andrew

gjr80

unread,
Nov 26, 2017, 6:26:19 AM11/26/17
to weewx-user
So I guess the issue is making sure there is a means of users being able to customise the logwatch script such that the user customisations are retained across upgrades and at the same time the stock logwatch script is upgraded as required during an upgrade. At the moment users that use the stock logwatch script should not experience any problems across upgrades.

Gary

Andrew Milner

unread,
Nov 26, 2017, 6:53:02 AM11/26/17
to weewx-user
Ah - not quite true Gary.  The stock messages for added archive record were what highlighted my problem in the first place.  As I have installed via setup.py, I copied the script to /etc/logwatch - and this copy is not updated in the upgrade.  Maybe I should have 'installed' the script a different way originally and not copied the script to /etc/logwatch as I was instructed to do at the time when I first installed logwatch many moons ago!!  If I change to a symlink in /etc/logwatch/scripts/services pointing to /home/weewx/util/logwatch/scripts/services does this mean that the new script is automagically picked up each upgrade?? 
Right - have changed to a symlink back to /home/weewx/util/logwatch/scripts/services/weewx - but now I assume that when there is a new upgrade the update will overwrite the util directory, and so I will lose my changes - correct??

gjr80

unread,
Nov 26, 2017, 8:38:11 AM11/26/17
to weewx-user
Well I think it is true. If users follow the wiki (and link to the filter script) they will have no issues unless they run a modified filter script. If they run a modified filter script they will have issues (loss of local changes or weeWX upgrade filter script changes not being applied/used) irrespective of whether they link to the filter script (loss of local changes) or use a modified copy of the filter script saved elsewhere (distributed updates will not be applied).

Solve the situation for users running custom filter scripts and all will be fine.

Gary

vince

unread,
Nov 26, 2017, 2:25:25 PM11/26/17
to weewx-user
On Sunday, November 26, 2017 at 3:26:19 AM UTC-8, gjr80 wrote:
So I guess the issue is making sure there is a means of users being able to customise the logwatch script such that the user customisations are retained across upgrades 


I don't think you should even attempt to support that.  If the user essentially hacks on the logwatch script, they are doing so at their own risk.

gjr80

unread,
Nov 26, 2017, 6:13:27 PM11/26/17
to weewx-user
Ok, poor choice of words on my behalf. I am not advocating users changing the actual filter script, as I have pointed out this will lead to problems. Rather users need to have a mechanism to customise the logwatch filter, similar in broad concept to how users can use search list extensions to add to the available tags in a report.

Gary

Andrew Milner

unread,
Nov 26, 2017, 10:25:20 PM11/26/17
to weewx-user
In my instance the need for the filter changes were raw and historygenerator parts of the meso and bootstrap skin extensions to weewx.

Daniel Jönsson

unread,
Dec 6, 2017, 4:48:14 AM12/6/17
to weewx-user
Sorry for late answer. Been away for work.

All this I see in syslog:

Dec  6 10:40:28 server weewx[30571]: fousb: synchronising to the weather station (quality=1)
Dec  6 10:41:14 server weewx[30571]: manager: added record 2017-12-06 10:37:01 CET (1512553021) to database 'weewx'
Dec  6 10:41:14 server weewx[30571]: manager: added record 2017-12-06 10:37:01 CET (1512553021) to daily summary in 'weewx'
Dec  6 10:41:15 server weewx[30571]: GaugeGenerator: Generated 6 images for HTMLPages in 0.19 seconds
Dec  6 10:41:15 server weewx[30571]: translategenerator.pyc: Language is swedish
Dec  6 10:41:15 server weewx[30571]: historygenerator.pyc: Generated 6 tables in 0.33 seconds
Dec  6 10:41:16 server weewx[30571]: cheetahgenerator: Generated 10 files for report HTMLPages in 1.68 seconds
Dec  6 10:41:16 server weewx[30571]: reportengine: copied 0 files to /var/www/weewx/Bootstrap
Dec  6 10:41:16 server weewx[30571]: translategenerator.pyc: Language is swedish
Dec  6 10:41:17 server weewx[30571]: genimages: Generated 6 images for BigImages in 0.80 seconds
Dec  6 10:41:17 server weewx[30571]: translategenerator.pyc: Language is swedish
Dec  6 10:41:17 server weewx[30571]: genimages: Generated 6 images for SmallImages in 0.33 seconds
Message has been deleted

Daniel Jönsson

unread,
Dec 6, 2017, 2:29:25 PM12/6/17
to weewx-user
Something strange here with Google groups. Sorry if double posting now, but my messages are dispersing..

I have now made changes to rsyslog config, to only log messages with level 5 and above, and weewx is still flooding my syslog with the following two lines every 5 minutes:

Dec  6 19:51:03 server weewx[30571]: manager: added record 2017-12-06 19:51:00 CET (1512586260) to database 'weewx'
Dec  6 19:51:03 server weewx[30571]: manager: added record 2017-12-06 19:51:00 CET (1512586260) to daily summary in 'weewx'

How could this be considered "normal but significant condition" as per RFC 5424? I would vote for that to be level 6 (info)

And here is my weewx.conf:
# WEEWX CONFIGURATION FILE
#
# Copyright (c) 2009-2015 Tom Keffer <tke...@gmail.com>
# See the file LICENSE.txt for your rights.

##############################################################################

# This section is for general configuration information.

# Set to 1 for extra debug info, otherwise comment it out or set to zero
debug = 0

# Root directory of the weewx data file hierarchy for this station
WEEWX_ROOT = /

# How long to wait before timing out a socket (FTP, HTTP) connection
socket_timeout = 20

# Do not modify this. It is used when installing and updating weewx.
version = 3.6.1

##############################################################################

#   This section is for information about the station.

[Station]
    
    # Description of the station location
    location = "Munka Ljungby, Sweden"
    
    # Latitude and longitude in decimal degrees
    latitude = 56.261
    longitude = 12.988
    
    # Altitude of the station, with unit it is in. This is downloaded from
    # from the station if the hardware supports it.
    altitude = 53, meter
    
    # Set to type of station hardware. There must be a corresponding stanza
    # in this file with a 'driver' parameter indicating the driver to be used.
    station_type = FineOffsetUSB
    
    # If you have a website, you may specify an URL
    #station_url = http://www.example.com
    
    # The start of the rain year (1=January; 10=October, etc.). This is
    # downloaded from the station if the hardware supports it.
    rain_year_start = 1
    
    # Start of week (0=Monday, 6=Sunday)
    week_start = 0

##############################################################################

[FineOffsetUSB]
    # This section is for the Fine Offset series of weather stations.
    
    # The station model, e.g., WH1080, WS1090, WS2080, WH3081
    model = WS1080
    
    # How often to poll the station for data, in seconds
    polling_interval = 60
    
    # The driver to use:
    driver = weewx.drivers.fousb
    
    # Filter unreal values:
    max_rain_rate = 15    # measured in cm/hr

##############################################################################

#   This section is for uploading data to Internet sites

[StdRESTful]
    log_success = False         # default is True
    log_failure = True          # default is True
    
    [[StationRegistry]]
        # To register this weather station with weewx, set this to true
        register_this_station = false
    
    [[AWEKAS]]
        # This section is for configuring posts to AWEKAS.
        
        # If you wish to do this, set the option 'enable' to true,
        # and specify a username and password.
        enable = false
        username = replace_me
        password = replace_me
    
    [[CWOP]]
        # This section is for configuring posts to CWOP.
        
        # If you wish to do this, set the option 'enable' to true,
        # and specify the station ID (e.g., CW1234).
        enable = false
        station = replace_me
    
    # If this is an APRS (radio amateur) station, uncomment
    # the following and replace with a passcode (e.g., 12345).
    #passcode = replace_me (APRS stations only)
    
    [[PWSweather]]
        # This section is for configuring posts to PWSweather.com.
        
        # If you wish to do this, set the option 'enable' to true,
        # and specify a station and password.
        enable = false
        station = replace_me
        password = replace_me
    
    [[WOW]]
        # This section is for configuring posts to WOW.
        
        # If you wish to do this, set the option 'enable' to true,
        # and specify a station and password.
        enable = false
        station = replace_me
        password = replace_me
    
    [[Wunderground]]
        # This section is for configuring posts to the Weather Underground.
        
        # If you wish to do this, set the option 'enable' to true,
        # and specify a station (e.g., 'KORHOODR3') and password.
        enable = false
        station = replace_me
        password = replace_me
        
        # Set the following to True to have weewx use the WU "Rapidfire"
        # protocol. Not all hardware can support it. See the User's Guide.
        rapidfire = False

##############################################################################

#   This section specifies what reports, using which skins, to generate.

[StdReport]
    log_success = False         # default is True
    log_failure = True          # default is True
    
    # Where the skins reside, relative to WEEWX_ROOT
    SKIN_ROOT = /etc/weewx/skins
    
    # Where the generated reports should go, relative to WEEWX_ROOT
    HTML_ROOT = /var/www/weewx
    
    # The database binding indicates which data should be used in reports.
    data_binding = wx_binding
    
    # Each of the following subsections defines a report that will be run.
    [[HTMLPages]]
        HTML_ROOT = /var/www/weewx/Bootstrap
        skin = Bootstrap
    [[BigImages]]
        HTML_ROOT = /var/www/weewx/Bootstrap/big_images
        skin = Images
    [[[ImageGenerator]]]
        bottom_label_font_size = 14
        axis_label_font_size = 14
        image_width = 900
        top_label_font_size = 18
        image_height = 600
        anti_alias = 2
        unit_label_font_size = 18
    [[SmallImages]]
        HTML_ROOT = /var/www/weewx/Bootstrap/images
        skin = Images
    
    [[FTP]]
        # FTP'ing the results to a webserver is treated as just another report,
        # albeit one with an unusual report generator!
        skin = Ftp
        
        # If you wish to use FTP, uncomment and fill out the next four lines.
        #user = replace with the ftp username
        #password = replace with the ftp password
        #server = replace with the ftp server name, e.g, www.threefools.org
        #path = replace with the ftp destination directory (e.g., /weather)
        
        # Set to True for a secure FTP (SFTP) connection. Not all servers
        # support this.
        secure_ftp = False
        
        # To upload files from something other than what HTML_ROOT is set
        # to above, specify a different HTML_ROOT here.
        #HTML_ROOT = /var/www/weewx
        
        # Most FTP servers use port 21
        port = 21
        
        # Set to 1 to use passive mode, zero for active mode
        passive = 1
    
    [[RSYNC]]
        # rsync'ing to a webserver is treated as just another report
        skin = Rsync
        
        # If you wish to use rsync, you must configure passwordless ssh using
        # public/private key authentication from the user account that weewx
        # runs as to the user account on the remote machine where the files
        # will be copied.
        #
        # The following three lines determine where files will be sent.
        #server = replace with the rsync server name, e.g, www.threefools.org
        #path = replace with the rsync destination directory (e.g., /weather)
        #user = replace with the rsync username
        
        # Rsync can be configured to remove files from the remote server if
        # they don't exist under HTML_ROOT locally. USE WITH CAUTION: if you
        # make a mistake in the remote path, you could could unintentionally
        # cause unrelated files to be deleted. Set to 1 to enable remote file
        # deletion, zero to allow files to accumulate remotely.
        delete = 0

##############################################################################

#   This service acts as a filter, converting the unit system coming from
#   the hardware to a unit system in the database.

[StdConvert]
    
    # The target_unit affects only the unit system in the database. Once
    # chosen it cannot be changed without converting the entire database.
    # Modification of target_unit after starting weewx will result in
    # corrupt data - the database will contain a mix of US and METRIC data.
    #
    # The value of target_unit does not affect the unit system for
    # reporting - reports can display US, Metric, or any combination of units.
    #
    # In most cases, target_unit should be left as the default: US
    #
    # In particular, those migrating from a standard wview installation
    # should use US since that is what the wview database contains.
    
    # DO NOT MODIFY THIS VALUE UNLESS YOU KNOW WHAT YOU ARE DOING!
    target_unit = US    # Options are 'US', 'METRICWX', or 'METRIC'

##############################################################################

#   This section can adjust data using calibration expressions.

[StdCalibrate]
    
    [[Corrections]]
        # For each type, an arbitrary calibration expression can be given. 
        # It should be in the units defined in the StdConvert section.
        # Example:
        # foo = foo + 0.2
        barometer = barometer + 0.21557
        pressure = pressure + 0.21557
        altimeter = altimeter + 0.21557

##############################################################################

#   This section is for quality control checks. If units are not specified,
#   values must be in the units defined in the StdConvert section.

[StdQC]
    
    [[MinMax]]
        barometer = 26, 32.5, inHg
        outTemp = -40, 120, degree_F
        inTemp = 10, 120, degree_F
        outHumidity = 0, 100
        inHumidity = 0, 100
        windSpeed = 0, 120, mile_per_hour

##############################################################################

#   This section controls the origin of derived values.

[StdWXCalculate]

    [[Calculations]]
    # Derived quantities are calculated by this service. Possible values are:
    #  hardware        - use the value provided by hardware
    #  software        - use the value calculated by weewx
    #  prefer_hardware - use value provide by hardware if available,
    #                      otherwise use value calculated by weewx
    
        pressure = prefer_hardware
        barometer = prefer_hardware
        altimeter = prefer_hardware
        windchill = prefer_hardware
        heatindex = prefer_hardware
        dewpoint = prefer_hardware
        inDewpoint = prefer_hardware
        rainRate = prefer_hardware

##############################################################################

#   For hardware that supports it, this section controls how often the
#   onboard clock gets updated.

[StdTimeSynch]
    
    # How often to check the weather station clock for drift (in seconds)
    clock_check = 14400
    
    # How much it can drift before we will correct it (in seconds)
    max_drift = 5

##############################################################################

#   This section is for configuring the archive service.

[StdArchive]
    
    # If the station hardware supports data logging then the archive interval
    # will be downloaded from the station. Otherwise, specify it (in seconds).
    archive_interval = 300
    
    # How long to wait (in seconds) before processing new archive data. Must
    # be greater than zero.
    archive_delay = 15
    
    # If possible, new archive records are downloaded from the station
    # hardware. If the hardware does not support this, then new archive
    # records will be generated in software.
    # Set the following to "software" to force software record generation.
    record_generation = hardware
    
    # Whether to include LOOP data in hi/low statistics
    loop_hilo = True
    
    # The data binding used to save archive records
    data_binding = wx_binding

##############################################################################

#   This section binds a data store to a database.

[DataBindings]
        
    [[wx_binding]]
        # The database must match one of the sections in [Databases].
        database = archive_mysql
        # The name of the table within the database
        table_name = archive
        # The manager handles aggregation of data for historical summaries
        manager = weewx.wxmanager.WXDaySummaryManager
        # The schema defines the structure of the database.
        # It is *only* used when the database is created.
        schema = schemas.wview.schema

##############################################################################

#   This section defines various databases.

[Databases]
    
    # A SQLite database is simply a single file
    [[archive_sqlite]]
        database_type = SQLite
        database_name = weewx.sdb
    
    # MySQL
    [[archive_mysql]]
        database_type = MySQL
        database_name = weewx

##############################################################################

#   This section defines defaults for the different types of databases.

[DatabaseTypes]
    
    # Defaults for SQLite databases
    [[SQLite]]
        driver = weedb.sqlite
        # Directory in which the database files are located
        SQLITE_ROOT = /var/lib/weewx
    
    # Defaults for MySQL databases
    [[MySQL]]
        driver = weedb.mysql
        # The host where the database is located
        host = localhost
        # The user name for logging in to the host
        user = weewx
        # The password for the user name
        password = SECRET

##############################################################################


#   This section configures the internal weewx engine.

[Engine]
    
    [[Services]]
        # This section specifies the services that should be run. They are
        # grouped by type, and the order of services within each group
        # determines the order in which the services will be run.
        prep_services = weewx.engine.StdTimeSynch
        data_services = ,
        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate
        archive_services = weewx.engine.StdArchive
        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS
        report_services = weewx.engine.StdPrint, weewx.engine.StdReport



Andrew Milner

unread,
Dec 6, 2017, 10:41:25 PM12/6/17
to weewx-user
Just a thought for you to think about.  You may find it helpful to retain the 'record added' log information just to make it easier to see if weewx is running normally or if it has stopped (as in the case of an obscure bug that did not cause an error message) - although I accept that that could be determined from examination of the database.

</fo

Daniel Jönsson

unread,
Dec 7, 2017, 1:04:54 AM12/7/17
to weewx-user
No. To make shore processes are running you use a monitoring tool. For example Monit.
The problem with one process flooding the syslog, is that errors not declared in monitoring tools, are missed because you cant find them by reading the log.
And for shore no other programs/servers/process I have ever had running on the server behaves this way (except Home Assistant, which is also written i Python). 

For examepl now (since I changed the conf for rsyslog 10 hours ago) in my syslog I have 120 lines. All are from weewx and looks like this:
Dec  6 11:50:50 server weewx[30571]: manager: added record 2017-12-06 11:47:01 CET (1512557221) to database 'weewx'
Dec  6 11:50:50 server weewx[30571]: manager: added record 2017-12-06 11:47:01 CET (1512557221) to daily summary in 'weewx'

Exept ONE line that says:
Dec  6 11:46:50 server weewx[30571]: fousb: get_observations failed: ignoring suspected bogus data from 0xbe40 (delay=5 interval=5)

This one I want to be in sys log. And the only reason I saw this is because I was looking for an example of important messages getting lost. I would never have seen this if I was reading through the log next week. Then there would be 1 000 lines of successful reading, and 1 line of an error.

Andrew Milner

unread,
Dec 7, 2017, 1:18:20 AM12/7/17
to weewx-user
Another method is to not use syslog at all!! 
Diverted weewx log messages to a separate log file (see the wiki) so they never clutter syslog and receive a daily weewx email giving the log summary for the previous day (see logwatch in wiki).  syslog thus remains unaffected by weewx.  This has an added advantage in that debugging levels can be set also without cluttering the syslog.

A monitoring program is obviously the best to maintain integrity - my option was intended to mean that having regular logged messages does allow one to pinpoint and spot possible issues caused by situations which do not necessarily result in weewx stopping - weewx is very reliable and rarely, if ever, crashes without recovering itself and there being no need for manual intervention.

Daniel Jönsson

unread,
Dec 7, 2017, 3:00:31 AM12/7/17
to weewx-user
Agreeing on weewx being reliable. It has been running non-stop for I think 3 years now. This is why I don't want it to fill up my syslog.. :-) I trust it to work, and if not then I want to be notified.
And of course it is useful having plenty of logging available. But I think it should be as little as possible as per default, and then you can set more logging if you like that.

And of course you could redirect all messages to a separate log, but then you need to keep track of more log files.

I want all important messages to end up in syslog, but nothing else. All less important logging should be put in a separate logfil, and the default way should be no logging.
I am running weewx on a server with "unlimited" storage, memory and cpu. But if you would run it on ie a rasberry you wear out the sd card faster.
Reply all
Reply to author
Forward
0 new messages