4.x logging format

[Graham Eddy]

i am trying to change the weewx log format. i have added the following to my weewx.conf, but it has no effect. how should i do it? (weewx 4.0.0 on macos 10.13.6 high sierra). and can i use { style?

[Logging]

  [[formatters]]

    [[[verbose]]]

      format = '%(asctime)s %(levelname)s %(name)s|%(message)s'

      datefmt = %Y-%m-%d %H:%M:%S

    [[[standard]]]

      format = '%(asctime)s %(levelname)s %(name)s|%(message)s'

      datefmt = %Y-%m-%d %H:%M:%S

    [[[simple]]]

      format = '%(asctime)s %(levelname)s %(name)s|%(message)s'

      datefmt = %Y-%m-%d %H:%M:%S

but it gives me the weewx default

May  2 21:59:44 pebble weewx[1663] INFO __main_] <Info>: Platform Darwin-17.7.0-x86_64-i386-64bit

[Graham Eddy]

i reported this incorrectly. instead of giveing me an unchanged logger format, it gives me no lines at all (from the point the weewx.conf log format is parsed).  even the following does the same (no logger output). commenting out the “format” line reverts to lines being produced, in original format of course

[Logging]

  [[formatters]]

    [[[standard]]]

      format = '%(message)s'

[Tom Keffer]

Logging from Python under MacOS High Sierra and later is a complete mystery. I have been unable to get it to work and, it seems, neither have other users.

Wish I could be of more help...

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/55F10FED-0D65-4010-9D74-D61203848291%40gmail.com.

[Graham Eddy]

i think we all agree that syslog on macos is a lost cause.

the corollary is that pointing python logging at syslog (via logging.handlers.SysLogHandler) is doomed

i accept that syslog works fine on most platforms, but it is not feasible for macos

but there is the simple alternative for macos users of using a different handler:

i patched log_dict in bin/weeutil/logger.py to add a logging.handlers.RotatingFileHandler and it worked fine.

adding it via weewx.conf gives a type conversion error that seems simple to fix.

with the fix, i would suggest this be added to the macos installation instructions for user to add to weewx.conf (or make it part of the installation process)

details follow...

weewx.conf (yes, using that ‘root’ hack introduced to work around a ConfigObj limitation):

[Logging]

  [[loggers]]

    [[[root]]]

      level = DEBUG

      propogate = True

      handlers = syslog, rotate

  [[handlers]]

    [[[rotate]]]

      level = DEBUG

      formatter = standard

      class = logging.handlers.RotatingFileHandler

      filename = /tmp/weewx.log

      maxBytes = 10000000

      backupCount = 2

gives the following error, which on the face of it would take a simple fix of type conversions:

Traceback (most recent call last):

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 563, in configure

    handler = self.configure_handler(handlers[name])

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 736, in configure_handler

    result = factory(**kwargs)

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/handlers.py", line 146, in __init__

    if maxBytes > 0:

TypeError: '>' not supported between instances of 'str' and ‘int’ < - - - SEE HERE

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "bin/weewxd", line 261, in <module>

    main()

  File "bin/weewxd", line 136, in main

    weeutil.logger.setup(options.log_label, config_dict)

  File "/opt/weewx/bin/weeutil/logger.py", line 203, in setup

    logging.config.dictConfig(log_dict)

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 800, in dictConfig

    dictConfigClass(config).configure()

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 571, in configure

    '%r' % name) from e

ValueError: Unable to configure handler 'rotate'

the log_dict it produces before crashing seems correct:

{'version': 1, 'disable_existing_loggers': False, 'loggers': {'': {'level': 'DEBUG', 'propagate': True, 'handlers': ['syslog'], 'propogate': 'True'}}, 'handlers': {'syslog': {'level': 'DEBUG', 'formatter': 'standard', 'class': 'logging.handlers.SysLogHandler', 'address': '/var/run/syslog', 'facility': 'local1'}, 'console': {'level': 'DEBUG', 'formatter': 'verbose', 'class': 'logging.StreamHandler', 'stream': 'ext://sys.stdout'}, 'rotate': {'level': 'DEBUG', 'formatter': 'standard', 'class': 'logging.handlers.RotatingFileHandler', 'filename': '/tmp/weewx.log', 'maxBytes': '10000000', 'backupCount': '2'}}, 'formatters': {'simple': {'format': '%(levelname)s %(message)s'}, 'standard': {'format': 'weewx[%(process)d] %(levelname)s %(name)s: %(message)s'}, 'verbose': {'format': '%(asctime)s  weewx[%(process)d] %(levelname)s %(name)s: %(message)s', 'datefmt': '%Y-%m-%d %H:%M:%S'}}}

cheers

[Tom Keffer]

Great ideas! Thanks for your thoughts.

Try this version of weeutil/logger.py. It includes a shim that does the proper type conversion before passing on arguments to logging.handlers.RotatingFileHandler.

To use should be as simple as adding this to weewx.conf (alas, NOT TESTED):

[Logging]
  [[loggers]]
    [[[root]]]

      handlers = syslog, rotate
  [[handlers]]
    [[[rotate]]]

      filename = /tmp/weewx.log

By default, it logs to /var/log/weewx.log. Hence, the need to override that option if you want to log to /tmp/weewx.log.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/85C23BE1-912C-4EAA-80AC-F7E81F8EE021%40gmail.com.

[Graham Eddy]

simplified logging stanza doesn’t work and shows type conversion fix needed.

details follow...

using that simplified stanza in weewx.conf gives runtime error:

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 381, in resolve

    name = s.split('.')

AttributeError: 'NoneType' object has no attribute 'split'

adding class line back into config viz:

[Logging]

#  [[loggers]]

#    [[[root]]]

#      level = DEBUG

#      propogate = True

#      handlers = syslog, rotate

  [[handlers]]

    [[[rotate]]]

#      level = DEBUG

#      formatter = standard

      class = logging.handlers.RotatingFileHandler

      filename = /tmp/weewx.log

#      maxBytes = 10000000

#      backupCount = 2

does not give immediate runtime error

but also adds no logging messages to /tmp/weewx.log

RotatingFileHandler documentation says defaults for maxBytes and backupCount are zero

and either of these being zero means logging is not done.

experimentation shows documentation is correct i.e. this config does not work

therefore both maxBytes and backupCount must be definable to be non-zero

so we need the type conversion fix

note: i am using /tmp/weewx.log here because i am presently fiddling with new version on test system and don’t want weewxd running rampant with root privilege. at least, not yet :-)

[Graham Eddy]

woops! i missed your reference to replacement weeutil/logger.py…

putting that in place, it works fine on macos.

it achieves the two things i was after: logs entries in a file (a basic function missing from weewx on macos for such a long time!), and allows customising message formats

i have two critiques about this fix:

  (1) linux would want [syslog] handlers only but macos would want [rotate] for default. easiest is to default correctly for one in the code and override it in weewx.conf for the other (applied either by user or installation script). if both enabled, harmless (if not duplicating entries in one logfile on linux) but wasteful

  (2) shim approach is per- logging handler class only. would be pretty easy for python user to add another shim in user/extensions.py but… alternatively, do the string->int conversions e.g. add after line 130 in weeutil/logger.py “log_config['Logging'].walk(_fix)” something like:

    def _intify_string_leaf(section, key):

        if isinstance(section[key], str):

            # The value is a string...

            try:

                value = int(section[key])

            except ValueError:

                pass

            else:

                # ... looks like an int, so make it one

                section[key] = value

    log_config['Logging'].walk(_intify_string_leaf)

(this works.) conversely, intifying all leaves could have unintended consequences...

thanks for all that! cheers

On 3 May 2020, at 5:07 am, Tom Keffer <tke...@gmail.com> wrote:

[Tom Keffer]

I like your fix better. Added in commit 6348741. 

I also made the rotating log handler the default for MacOS.

One problem I've discovered: if you 'declare' a handler, then the logging facility goes ahead and instantiates it, whether or not you actually use it. That means the program crashes when it tries to create /var/log/weewx.log unless its run with sudo privileges.

So, for non-Mac systems, I took the handler back out, and put instructions on how to declare it in the wiki documentation.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/AB7F41D1-FD08-400D-B80A-01E53BD42D5A%40gmail.com.

[Chris Alemany]

Just like to say I updated to 4.x and just added

 

[Logging]

  [[handlers]]

    [[[rotate]]]

      filename = /tmp/weewx.log

To the top section of my weewx.conf file (after version) and it is WONDERFUL to finally have real log output from weewx.  It's beautiful! I might cry for not having to stare at Console.app and hope for something.  :)

Thank you all for your continued efforts. 

[Doug Jenkins]

Not to bring up old issues but, I am trying to format my logs. I read the wiki attached to this thread and I have this configuration working in my weewx.conf:

# ===== LOGGING SETUP (Configured 09/04/2021) =====

[Logging]]

  [[root]]

    handlers = rotate,

  [[loggers]]

    [[handlers]]

      [[[rotate]]]

        level = INFO

        formatter = standard

        class = logging.handlers.RotatingFileHandler

        filename = /mnt/sandisk_512_ssd/weewx/logs/weewx.log

        maxBytes = 1000000

        backupCount = 10

# ===== END LOGGING SETUP =====

But I want to format the logging out using a custom formatter as such:
# ==== PROPOSED CHANGE TO MY WEEWX.CONF =====

[Logging]

  [[formatters]]

    [[[ddj]]]

      format = '%(asctime)s [%(levelname)s] [%(threadName)s] [%(name)s : %(funcName)s] %(message)s'

  [[root]]

    handlers = rotate,

  [[loggers]]

    [[handlers]]

      [[[rotate]]]

        level = INFO

        formatter = ddj

        class = logging.handlers.RotatingFileHandler

        filename = /mnt/sandisk_512_ssd/weewx/logs/weewx.log

        maxBytes = 1000000

        backupCount = 10

# ===== END LOGGING SETUP =====

I am running Weewx 4.5.1 on a Raspberry Pi using Raspbian O/S (32bit) which is essentially Debian buster.

Is there a way to express a formatter in the weewx.conf file so the logging object will format the results for me?

[Tom Keffer]

It should be possible. Did you try it? What happened?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/819f1a6c-2ae0-4861-a5c1-726512243c5an%40googlegroups.com.

[Doug Jenkins]

Good morning Tom:

sorry about the late reply on this thread. I did try this on my setup. and I got the following error when it tried to report:

# ==== FOR REFERENCE : weewx.conf logging setup =====

# Do not modify this. It is used when installing and updating weewx.

version = 4.5.1

[Logging]

  [[formatters]]

      [[[ddj]]]

        format = '%(asctime)s [%(levelname)s] [%(threadName)s] [%(name)s : %(funcName)s] %(message)s'

  [[root]]

    handlers = rotate,

  [[loggers]]

    [[handlers]]

      [[[rotate]]]

        level = INFO

        formatter = ddj

        class = logging.handlers.RotatingFileHandler

        filename = /var/log/weewx/weewx.log

        maxBytes = 1000000

        backupCount = 10

# ===== END : weewx.conf logging setup =====

# ===== ERROR message when I execute wee_reports. =====

Using configuration file /etc/weewx/weewx.conf

Traceback (most recent call last):

  File "/usr/share/weewx/wee_reports", line 103, in <module>

    main()

  File "/usr/share/weewx/wee_reports", line 63, in main

    weeutil.logger.setup('wee_reports', config_dict)

  File "/usr/share/weewx/weeutil/logger.py", line 178, in setup

    log_config['Logging'].walk(_fix)

  File "/usr/lib/python3/dist-packages/configobj.py", line 919, in walk

    out[entry] = self[entry].walk(

  File "/usr/lib/python3/dist-packages/configobj.py", line 919, in walk

    out[entry] = self[entry].walk(

  File "/usr/lib/python3/dist-packages/configobj.py", line 894, in walk

    val = function(self, entry, **keywargs)

  File "/usr/share/weewx/weeutil/logger.py", line 164, in _fix

    if isinstance(section[key], (list, tuple)):

  File "/usr/lib/python3/dist-packages/configobj.py", line 557, in __getitem__

    return self._interpolate(key, val)

  File "/usr/lib/python3/dist-packages/configobj.py", line 549, in _interpolate

    return engine.interpolate(key, value)

  File "/usr/lib/python3/dist-packages/configobj.py", line 352, in interpolate

    value = recursive_interpolate(key, value, self.section, {})

  File "/usr/lib/python3/dist-packages/configobj.py", line 330, in recursive_interpolate

    k, v, s = self._parse_match(match)

  File "/usr/lib/python3/dist-packages/configobj.py", line 417, in _parse_match

    value, section = self._fetch(key)

  File "/usr/lib/python3/dist-packages/configobj.py", line 386, in _fetch

    raise MissingInterpolationOption(key)

configobj.MissingInterpolationOption: missing option "asctime" in interpolation.

# ==== END ERROR MESSAGE =====

any help is always appreciated!

DDJ

[Tom Keffer]

Hi, Doug

I was able to replicate this. The problem is that ConfigObj, the utility that we use to read the configuration files, is trying to interpolate your 'format', rather than simply pass it on to the logging facility.

Replace your version of logger.py with the attached. You should find it in /usr/share/weewx/weeutil/logger.py

Let me know if that helps.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/41b027e6-29a4-45c6-8bdb-8cb93a0a3640n%40googlegroups.com.

[Doug Jenkins]

Tom:

Thanks for the hotfix! I do appreciate it such a quick turnaround on a small item like this! :)

I switched the logging.py file at /usr/share/weewx/weeutil/ directory and tested it with both wee_reports and the standard weewx service. In both cases, Weewx formatted the log using my formatter expression. I made some changes to the formatter in my weewx.conf to ensure it was responding to the changes. The log reported the changes as expected.

I did not test wee_database, wee_import, or wee_device, but they probably will utilize this logging.py change and passthrough the formatter as expected.

Please let me know if you need any additional testing for this file or a issue/PR logged for the item. 

Again thanks for the great solution and continuous development of weewx! 

DDJ