weewx v4 need for syslog daemon

[Vince Skahan]

I'm running into adventures with v4 and docker images for the various os due to the new logging setup.

Typical example is starting v4 (rpm) up in a centos8 container:

--- Logging error ---

Traceback (most recent call last):

  File "/usr/share/weewx/weewx/engine.py", line 188, in run

    for packet in self.console.genLoopPackets():

  File "/usr/share/weewx/weewx/drivers/simulator.py", line 148, in genLoopPackets

    time.sleep(sleep_time)

KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/lib64/python3.6/logging/handlers.py", line 936, in emit

    self.socket.send(msg)

OSError: [Errno 9] Bad file descriptor

On ubuntu-20.04 and debian-10 I worked around this by running rsyslogd under supervisor, so I had two processes (syslog daemon and weewxd) running in one container, which is kinda a hack.   I haven't been able to get this workaround working on centos7 or centos8 yet.

In v4 is there any way to tell weewx to not log via sockets ?

[Tom Keffer]

The default handler is SysLogHandler, which uses a socket to communicate with syslogd, but there are lots of other handlers.

How about RotatingFileHandler? Instructions are in the WeeWX wiki. NB: there is a bug in the WeeWX logging facility that prevents this handler from working if you specify options maxBytes and backupCount. If you want to use RotatingFileHandler, either avoid these two options, or use the patched weeutil.logger module.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/bec06cb1-7b38-42e2-a99b-f5df871eba85%40googlegroups.com.

[Vince Skahan]

On Sunday, May 17, 2020 at 12:01:15 PM UTC-7, Tom Keffer wrote:

The default handler is SysLogHandler, which uses a socket to communicate with syslogd, but there are lots of other handlers.

How about RotatingFileHandler? Instructions are in the WeeWX wiki. NB: there is a bug in the WeeWX logging facility that prevents this handler from working if you specify options maxBytes and backupCount. If you want to use RotatingFileHandler, either avoid these two options, or use the patched weeutil.logger module.

No difference.  I grabbed the patched module from your link.  Still all kinds of barfing about syslog sockets no matter what I do.

Does 'disable_existing_loggers' work ?  I can't see any difference in effect regardless of True/False or unspecified.

What I'm looking is to disable all loggers that are built-in, and log only to the console (for a container this makes most sense to me).

Here's the weewx.conf settings I'm failing with currently...

[Logging]

    version = 1

    disable_existing_loggers = True

    [[loggers]]

        [[[root]]]

           # the trailing comma here seems required 

           handlers = console,

    [[handlers]]

        [[[console]]]

          level = DEBUG

          formatter = verbose

          class = logging.StreamHandler

          # Alternate choice is 'ext://sys.stderr'

          stream = ext://sys.stdout

 

[Tom Keffer]

The default logging configuration includes a syslog handler. While you're trying to override that with a weewx.conf spec, the override gets merged into the default configuration. So, the syslog handler never goes away.

Unfortunately, the way the Python handlers work, all specified handlers get instantiated whether or not they will actually get used. No just-in-time creation here. So the mere presence of the syslog handler could be causing the problem. Or, maybe not.

A possible workaround. Put this in user/extensions.py. It basically removes completely the specification of a syslog handler from the default handler.

import weeutil.logger

weeutil.logger.LOGGING_STR = """[Logging]

    version = 1
    disable_existing_loggers = True

    [[loggers]]

        # Root logger
        [[[root]]]
          level = {log_level}
          propagate = 1
          handlers = console,

    # Definitions of possible logging destinations
    [[handlers]]

        # Log to console

        [[[console]]]
            level = DEBUG
            formatter = verbose
            class = logging.StreamHandler
            # Alternate choice is 'ext://sys.stderr'
            stream = ext://sys.stdout

    # How to format log messages
    [[formatters]]
        [[[simple]]]
            format = "%(levelname)s %(message)s"
        [[[standard]]]
            format = "{process_name}[%(process)d] %(levelname)s %(name)s: %(message)s"
        [[[verbose]]]
            format = "%(asctime)s  {process_name}[%(process)d] %(levelname)s %(name)s: %(message)s"
            # Format to use for dates and times:
            datefmt = %Y-%m-%d %H:%M:%S
"""

See if that helps.

As for disable_existing_loggers, its exact function isn't very clear from the documentation, at least to me, but it seems like in this case you want it True.

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/2e9d6f6b-9324-46b3-815b-f4dbde3f7df7%40googlegroups.com.

[Vince Skahan]

Minor update - I found that 'setup.py install' will not complete without a working syslog daemon on the build system.   Really odd.

Workaround on ubuntu/debian is to install rsyslogd and start the daemon before the 'setup.py install' step, which is good enough for those platforms.  I got my docker build stuff working for debian10 and ubuntu 20.04 in both setup.py and dpkg variants, and also did debian10 with setup.py and python2 successfully (for a driver author who wants his driver tested with python2 and weewx v4).

No joy on anything centos yet.   Hope to get to your patch posted yesterday this week sometime.

Dumb question - could the 'console' be the default and have syslog be the added handler ?

[Tom Mitchell]

I like this idea a lot - maybe it could be stdout. I would also love to not depend on a running syslogd.

[Tom Keffer]

Fair enough, but before I go implementing this... did it work? Did my workaround disable syslog thoroughly enough that you could get started?

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/06d0bc13-ea81-463b-b512-59917006d4cb%40googlegroups.com.

[Vince Skahan]

On Tuesday, May 19, 2020 at 11:52:42 AM UTC-7, Tom Keffer wrote:

Fair enough, but before I go implementing this... did it work? Did my workaround disable syslog thoroughly enough that you could get started?

update -  I got centos7 python2 setup.py to build/install/run ok by 'replacing' logger.py in the sources with a patched variant (using your code snippet) before doing the setup.py build+install.   The only downside in brief testing is the container doesn't log anything, as there is no logging subsystem in the distributed centos7 base Docker image.

Weewx 'does' run fine however.

Perhaps we might ask the other Docker users and the usual developer folks for their thoughts ?

• one option is switch the logging for everybody, but I don't know what they does for default installs, VMs, etc. as a result
• another option is to just require superseding logger.py for RedHat(ish) Docker users only, and note it somehow in the docs (?)

Updated repo is at https://github.com/vinceskahan/weewx-docker if anybody wants to take a look.  Centos7 build Dockerfile and patched logger.py is under the 'build' directory.  Dockerfile should be pretty obvious.

Maybe get some more eyeballs on this one before messing with the core weewx quite yet ?

[Tom Keffer]

What I had in mind was to trim the syslog handler out of the default weeutil.logger.LOGGING_STR, and, instead, put it in weewx.conf. You'd end up in the same place, but not without risk. The problem is that the installer would have to patch old weewx.conf files, inserting the syslog handler. 

If this is a rare instance, and it does seem to be, then I'd prefer my workaround. I'd rather explain to Docker users who, by and large, are sophisticated, how to do this, than have the system log inexplicably disappear for less experienced users.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/1811cab3-d5e8-40d5-a036-bcf5a07d81a0%40googlegroups.com.

[Tom Mitchell]

FWIW, I have been using phusion/baseimage as a base for my Docker builds and it pipes syslog to stdout by default - that works fine for me.

That said, I think it would be better if weewx were to log to stdout by default since it is kind of the default for Unix/variants. Not depending on syslogd seems like a nice simplification of the runtime.

It sounds like with the workaround I would be able to override the syslogd in weewx.conf which would be fine. Not ivory tower perfect but I understand the user impact.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-de...@googlegroups.com.

[Vince Skahan]

On Tuesday, May 19, 2020 at 5:05:09 PM UTC-7, Tom Keffer wrote:

What I had in mind was to trim the syslog handler out of the default weeutil.logger.LOGGING_STR, and, instead, put it in weewx.conf. You'd end up in the same place, but not without risk. The problem is that the installer would have to patch old weewx.conf files, inserting the syslog handler. 

I'm of course fine with doing so in weewx.conf as an optional thing, but I am concerned that as currently written you can't do a "setup.py install" successfully on some supported operating systems under their official Docker base images.   They're of course not going to change those images for us to alter that issue.

Underlying thing seems to be systemd under the hood doing logging in centos7 and later.  (awaiting Matthew going boom :-)

The Docker images of course don't run systemd as their init system.

Don't we have some users doing things like running weewx on NAS boxes and the like ?   Has this issue popped up for those kind of atypical users ?

[mwall]

On Tuesday, May 19, 2020 at 9:00:18 PM UTC-4, Tom Mitchell wrote:

FWIW, I have been using phusion/baseimage as a base for my Docker builds and it pipes syslog to stdout by default - that works fine for me.

That said, I think it would be better if weewx were to log to stdout by default since it is kind of the default for Unix/variants. Not depending on syslogd seems like a nice simplification of the runtime.

it is probably incorrect for weewx to assume that syslog exists, but not because it is not the 'unix way'.  on the contrary, syslog IS the unix way.  systemd is perverting the 'unix way', and not in a helpful or good way.

it is quite helpful to have weewx log messages in syslog - if there are hardware issues, or clock skew, or usb devices plugging/unplugging, or other system stuff happening, you see it right there with the weewx messages.  makes diagnostics much easier.

we had to make a syslog shim to do the windows port of weewx, so not depending on syslog has other benefits too.

emitting log messages to stdout is not appropriate, and is definitely not the 'unix way'.  log messages should go to the log, whether that is syslog, syslog-redirected-to-rsyslog-server, a file in /var/log, or whatever.

when you run weewx directly, the observation data go to stdout.

it would be more appropriate to default the log output to a file, with log rotation (preferably date-time stamped names, not incrementing index names).  for example, for a setup.py install the log would go to /home/weewx/log/weewx and for a deb/rpm install the log would go to /var/log/weewx

of course, it must be easy to change the name of the basename of the log file so that multiple instances of weewx can send to different files, each with their own rollovers

m

[Chris Alemany]

May I please put a hand way up for default file-based logging? This would be wonderful for MacOS X.  The log viewed through the standard “Console” in MacOS X is chock full and terrible. A dedicated file would make troubleshooting so much cleaner. (/var/log is a common spot in MacOS X but weewx also installs in its own space so people the log there would be nice too). I realize it’s not a voting thing… just my 2cs from the apple shaped seats. :)

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/3aea5121-69a6-436c-8ddf-f51636142de6%40googlegroups.com.

[Tom Keffer]

On the Mac, a rotating file handler will be the default starting with V4.0.1.

3-may-2020 commit 6348741

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/018C9240-38A2-4619-A96A-8E55C4C60CE0%40gmail.com.

[mwall]

here are typical use cases for weewx log output:

1) emit to system log (nominally syslog, but varies by platform)

2) emit to separate log file, one file per weewx instance.  optionally do log rotation (in case it is not handled by the os).  location varies by platform, but nominally /var/log or /home/weewx/log

are there others?

this might just be a configuration issue.  if you package weewx for platform x, then modify weewx.conf so that the log uses conventions for platform x.

[Vince Skahan]

The question I guess I have is why does weewx 'install' require a working syslog answering ?   That seems wrong to me.

This happens on installing via setup.py and installing packages.  Guessing there's a call under the hood to weeutil/logger.py but I didn't dig into the gory details.  If we could figure that out, then the normal runtime emit logging info gets more mainstream.

[mwall]

On Wednesday, May 20, 2020 at 12:15:20 PM UTC-4, Vince Skahan wrote:

The question I guess I have is why does weewx 'install' require a working syslog answering ?   That seems wrong to me.

This happens on installing via setup.py and installing packages.  Guessing there's a call under the hood to weeutil/logger.py but I didn't dig into the gory details.  If we could figure that out, then the normal runtime emit logging info gets more mainstream.

install should not require syslog (or any other) logging.  deb/rpm just extracts assets from the .deb/.rpm.  setup.py has an explicit import of 'log' from 'distutils' - not sure what logging mechanism that uses, but probably not syslog.

configure does require logging, but probably should not.  configuration is done by invoking wee_config, and that imports weeutil.logger.

it should be possible to refactor wee_config and the code it uses so that there are no more weeutil.logger dependencies.

[Vince Skahan]

update - I found an old article by Dan Walsh from Project Atomic on how to get rsyslog working in a container vs. RHEL7, which works in quick testing for me...

If I edit rsyslog.conf per https://www.projectatomic.io/blog/2014/09/running-syslog-within-a-docker-container/ and start rsyslogd 'before' starting weewx, then weewx works and logs as expected to /var/log/messages within the container without any patches to 4.0.0 - in 'very' brief testing.

I'll integrate that in my Docker stuff tonight maybe and verify it's good enough vs. setup.py and rpm installations for runtime weewx in a container.  Stay tuned.

Still think 'installing' should not require a listening syslog service of course....

[Vince Skahan]

On Wednesday, May 20, 2020 at 9:48:13 AM UTC-7, mwall wrote:

install should not require syslog (or any other) logging.  deb/rpm just extracts assets from the .deb/.rpm.  setup.py has an explicit import of 'log' from 'distutils' - not sure what logging mechanism that uses, but probably not syslog.

configure does require logging, but probably should not.  configuration is done by invoking wee_config, and that imports weeutil.logger.

it should be possible to refactor wee_config and the code it uses so that there are no more weeutil.logger dependencies.

Are we far enough along in the discussion to create a couple/few github issues and move things there ?

 - setup.py requiring logging

 - wee_config requiring logging

 - I think wee_util also requires it, if I remember one report correctly

I got further along dockerizing, albeit with workarounds - current status in https://github.com/vinceskahan/weewx-docker - still to do: get rpms to install in docker successfully and run ok with logging.   Working on it.....

[Tom Keffer]

Gee, I sat down, getting ready to track down and eliminate all uses of logging in wee_config, just to discover... it doesn't do any. It just uses print statements. 

However, what it does do, is a gratuitous 'import weeutil.logger', but then never uses it. Easy fix.

setup.py never did any logging. However, it executes wee_config as a subprocess, so that's where the logging calls came from.

Don't know what 'wee_util' is.

Commit 791de74.

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/ca24fbe5-09f7-4226-9410-b710394fb9f7%40googlegroups.com.

[Vince Skahan]

Sorry - I misspoke earlier.  It was wee_import that had the same issue.

Look in weewx-user for a thread named "wee_import" from May-19th.

[Vince Skahan]

On Thursday, May 21, 2020 at 2:23:33 PM UTC-7, Tom Keffer wrote:

Commit 791de74.

Tests ok for setup.py install on centos8 (python3), ubuntu2020.04 (python3), centos7 (python2), debian10 (python3)

Thanks !

You forgot to merge --no-prompt up to master :-)

[Tom Keffer]

It's a new feature, so it's in the development branch.

But, I can cherry-pick it into the master branch.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/139534d9-1428-4f5b-b371-79c1d3120292%40googlegroups.com.