4.x logging format

211 views
Skip to first unread message

Graham Eddy

unread,
May 2, 2020, 8:02:20 AM5/2/20
to weewx...@googlegroups.com
i am trying to change the weewx log format. i have added the following to my weewx.conf, but it has no effect. how should i do it? (weewx 4.0.0 on macos 10.13.6 high sierra). and can i use { style?

[Logging]
  [[formatters]]
    [[[verbose]]]
      format = '%(asctime)s %(levelname)s %(name)s|%(message)s'
      datefmt = %Y-%m-%d %H:%M:%S
    [[[standard]]]
      format = '%(asctime)s %(levelname)s %(name)s|%(message)s'
      datefmt = %Y-%m-%d %H:%M:%S
    [[[simple]]]
      format = '%(asctime)s %(levelname)s %(name)s|%(message)s'
      datefmt = %Y-%m-%d %H:%M:%S

but it gives me the weewx default
May  2 21:59:44 pebble weewx[1663] INFO __main_] <Info>: Platform Darwin-17.7.0-x86_64-i386-64bit

Graham Eddy

unread,
May 2, 2020, 8:26:42 AM5/2/20
to weewx...@googlegroups.com
i reported this incorrectly. instead of giveing me an unchanged logger format, it gives me no lines at all (from the point the weewx.conf log format is parsed).  even the following does the same (no logger output). commenting out the “format” line reverts to lines being produced, in original format of course

[Logging]
  [[formatters]]
    [[[standard]]]
      format = '%(message)s'

Tom Keffer

unread,
May 2, 2020, 8:42:57 AM5/2/20
to weewx-user
Logging from Python under MacOS High Sierra and later is a complete mystery. I have been unable to get it to work and, it seems, neither have other users.

Wish I could be of more help...

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/55F10FED-0D65-4010-9D74-D61203848291%40gmail.com.

Graham Eddy

unread,
May 2, 2020, 11:05:49 AM5/2/20
to weewx...@googlegroups.com
i think we all agree that syslog on macos is a lost cause.
the corollary is that pointing python logging at syslog (via logging.handlers.SysLogHandler) is doomed

i accept that syslog works fine on most platforms, but it is not feasible for macos
but there is the simple alternative for macos users of using a different handler:

i patched log_dict in bin/weeutil/logger.py to add a logging.handlers.RotatingFileHandler and it worked fine.
adding it via weewx.conf gives a type conversion error that seems simple to fix.
with the fix, i would suggest this be added to the macos installation instructions for user to add to weewx.conf (or make it part of the installation process)

details follow...

weewx.conf (yes, using that ‘root’ hack introduced to work around a ConfigObj limitation):

[Logging]
  [[loggers]]
    [[[root]]]
      level = DEBUG
      propogate = True
      handlers = syslog, rotate
  [[handlers]]
    [[[rotate]]]
      level = DEBUG
      formatter = standard
      class = logging.handlers.RotatingFileHandler
      filename = /tmp/weewx.log
      maxBytes = 10000000
      backupCount = 2

gives the following error, which on the face of it would take a simple fix of type conversions:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 736, in configure_handler
    result = factory(**kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/handlers.py", line 146, in __init__
    if maxBytes > 0:
TypeError: '>' not supported between instances of 'str' and ‘int’ < - - - SEE HERE

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "bin/weewxd", line 261, in <module>
    main()
  File "bin/weewxd", line 136, in main
    weeutil.logger.setup(options.log_label, config_dict)
  File "/opt/weewx/bin/weeutil/logger.py", line 203, in setup
    logging.config.dictConfig(log_dict)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 800, in dictConfig
    dictConfigClass(config).configure()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 571, in configure
    '%r' % name) from e
ValueError: Unable to configure handler 'rotate'

the log_dict it produces before crashing seems correct:

{'version': 1, 'disable_existing_loggers': False, 'loggers': {'': {'level': 'DEBUG', 'propagate': True, 'handlers': ['syslog'], 'propogate': 'True'}}, 'handlers': {'syslog': {'level': 'DEBUG', 'formatter': 'standard', 'class': 'logging.handlers.SysLogHandler', 'address': '/var/run/syslog', 'facility': 'local1'}, 'console': {'level': 'DEBUG', 'formatter': 'verbose', 'class': 'logging.StreamHandler', 'stream': 'ext://sys.stdout'}, 'rotate': {'level': 'DEBUG', 'formatter': 'standard', 'class': 'logging.handlers.RotatingFileHandler', 'filename': '/tmp/weewx.log', 'maxBytes': '10000000', 'backupCount': '2'}}, 'formatters': {'simple': {'format': '%(levelname)s %(message)s'}, 'standard': {'format': 'weewx[%(process)d] %(levelname)s %(name)s: %(message)s'}, 'verbose': {'format': '%(asctime)s  weewx[%(process)d] %(levelname)s %(name)s: %(message)s', 'datefmt': '%Y-%m-%d %H:%M:%S'}}}

cheers

Tom Keffer

unread,
May 2, 2020, 3:07:50 PM5/2/20
to weewx-user
Great ideas! Thanks for your thoughts.

Try this version of weeutil/logger.py. It includes a shim that does the proper type conversion before passing on arguments to logging.handlers.RotatingFileHandler.

To use should be as simple as adding this to weewx.conf (alas, NOT TESTED):

[Logging]
  [[loggers]]
    [[[root]]]

      handlers = syslog, rotate
  [[handlers]]
    [[[rotate]]]
      filename = /tmp/weewx.log

By default, it logs to /var/log/weewx.log. Hence, the need to override that option if you want to log to /tmp/weewx.log.

-tk


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
logger.py

Graham Eddy

unread,
May 2, 2020, 9:43:00 PM5/2/20
to weewx...@googlegroups.com
simplified logging stanza doesn’t work and shows type conversion fix needed.
details follow...

using that simplified stanza in weewx.conf gives runtime error:

  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/config.py", line 381, in resolve
    name = s.split('.')
AttributeError: 'NoneType' object has no attribute 'split'

adding class line back into config viz:

[Logging]
#  [[loggers]]
#    [[[root]]]
#      level = DEBUG
#      propogate = True
#      handlers = syslog, rotate
  [[handlers]]
    [[[rotate]]]
#      level = DEBUG
#      formatter = standard
      class = logging.handlers.RotatingFileHandler
      filename = /tmp/weewx.log
#      maxBytes = 10000000
#      backupCount = 2

does not give immediate runtime error
but also adds no logging messages to /tmp/weewx.log

RotatingFileHandler documentation says defaults for maxBytes and backupCount are zero
and either of these being zero means logging is not done.
experimentation shows documentation is correct i.e. this config does not work
therefore both maxBytes and backupCount must be definable to be non-zero
so we need the type conversion fix

note: i am using /tmp/weewx.log here because i am presently fiddling with new version on test system and don’t want weewxd running rampant with root privilege. at least, not yet :-)

Graham Eddy

unread,
May 3, 2020, 3:11:51 AM5/3/20
to weewx...@googlegroups.com
woops! i missed your reference to replacement weeutil/logger.py…
putting that in place, it works fine on macos.
it achieves the two things i was after: logs entries in a file (a basic function missing from weewx on macos for such a long time!), and allows customising message formats

i have two critiques about this fix:

  (1) linux would want [syslog] handlers only but macos would want [rotate] for default. easiest is to default correctly for one in the code and override it in weewx.conf for the other (applied either by user or installation script). if both enabled, harmless (if not duplicating entries in one logfile on linux) but wasteful

  (2) shim approach is per- logging handler class only. would be pretty easy for python user to add another shim in user/extensions.py but… alternatively, do the string->int conversions e.g. add after line 130 in weeutil/logger.py “log_config['Logging'].walk(_fix)” something like:
    def _intify_string_leaf(section, key):
        if isinstance(section[key], str):
            # The value is a string...
            try:
                value = int(section[key])
            except ValueError:
                pass
            else:
                # ... looks like an int, so make it one
                section[key] = value
    log_config['Logging'].walk(_intify_string_leaf)
(this works.) conversely, intifying all leaves could have unintended consequences...

thanks for all that! cheers

On 3 May 2020, at 5:07 am, Tom Keffer <tke...@gmail.com> wrote:

Tom Keffer

unread,
May 3, 2020, 7:54:13 AM5/3/20
to weewx-user
I like your fix better. Added in commit 6348741

I also made the rotating log handler the default for MacOS.

One problem I've discovered: if you 'declare' a handler, then the logging facility goes ahead and instantiates it, whether or not you actually use it. That means the program crashes when it tries to create /var/log/weewx.log unless its run with sudo privileges.

So, for non-Mac systems, I took the handler back out, and put instructions on how to declare it in the wiki documentation.

-tk


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Chris Alemany

unread,
Aug 21, 2020, 3:27:49 PM8/21/20
to weewx-user
Just like to say I updated to 4.x and just added
 
[Logging]
  [[handlers]]
    [[[rotate]]]
      filename = /tmp/weewx.log

To the top section of my weewx.conf file (after version) and it is WONDERFUL to finally have real log output from weewx.  It's beautiful! I might cry for not having to stare at Console.app and hope for something.  :)
Thank you all for your continued efforts. 

Doug Jenkins

unread,
Sep 4, 2021, 11:50:46 AMSep 4
to weewx-user
Not to bring up old issues but, I am trying to format my logs. I read the wiki attached to this thread and I have this configuration working in my weewx.conf:

# ===== LOGGING SETUP (Configured 09/04/2021) =====
[Logging]]
  [[root]]
    handlers = rotate,
  [[loggers]]
    [[handlers]]
      [[[rotate]]]
        level = INFO
        formatter = standard
        class = logging.handlers.RotatingFileHandler
        filename = /mnt/sandisk_512_ssd/weewx/logs/weewx.log
        maxBytes = 1000000
        backupCount = 10
# ===== END LOGGING SETUP =====

But I want to format the logging out using a custom formatter as such:
# ==== PROPOSED CHANGE TO MY WEEWX.CONF =====
[Logging]
  [[formatters]]
    [[[ddj]]]
      format = '%(asctime)s [%(levelname)s] [%(threadName)s] [%(name)s : %(funcName)s] %(message)s'
  [[root]]
    handlers = rotate,
  [[loggers]]
    [[handlers]]
      [[[rotate]]]
        level = INFO
        formatter = ddj
        class = logging.handlers.RotatingFileHandler
        filename = /mnt/sandisk_512_ssd/weewx/logs/weewx.log
        maxBytes = 1000000
        backupCount = 10
# ===== END LOGGING SETUP =====


I am running Weewx 4.5.1 on a Raspberry Pi using Raspbian O/S (32bit) which is essentially Debian buster.

Is there a way to express a formatter in the weewx.conf file so the logging object will format the results for me?

Tom Keffer

unread,
Sep 6, 2021, 8:43:21 AMSep 6
to weewx-user
It should be possible. Did you try it? What happened?

Doug Jenkins

unread,
Sep 22, 2021, 6:50:13 AMSep 22
to weewx-user
Good morning Tom:

sorry about the late reply on this thread. I did try this on my setup. and I got the following error when it tried to report:

# ==== FOR REFERENCE : weewx.conf logging setup =====
# Do not modify this. It is used when installing and updating weewx.
version = 4.5.1

[Logging]
  [[formatters]]
      [[[ddj]]]
        format = '%(asctime)s [%(levelname)s] [%(threadName)s] [%(name)s : %(funcName)s] %(message)s'
  [[root]]
    handlers = rotate,
  [[loggers]]
    [[handlers]]
      [[[rotate]]]
        level = INFO
        formatter = ddj
        class = logging.handlers.RotatingFileHandler
        filename = /var/log/weewx/weewx.log
        maxBytes = 1000000
        backupCount = 10
# ===== END : weewx.conf logging setup =====


# ===== ERROR message when I execute wee_reports. =====
Using configuration file /etc/weewx/weewx.conf
Traceback (most recent call last):

  File "/usr/share/weewx/wee_reports", line 103, in <module>
    main()
  File "/usr/share/weewx/wee_reports", line 63, in main
    weeutil.logger.setup('wee_reports', config_dict)
  File "/usr/share/weewx/weeutil/logger.py", line 178, in setup
    log_config['Logging'].walk(_fix)
  File "/usr/lib/python3/dist-packages/configobj.py", line 919, in walk
    out[entry] = self[entry].walk(
  File "/usr/lib/python3/dist-packages/configobj.py", line 919, in walk
    out[entry] = self[entry].walk(
  File "/usr/lib/python3/dist-packages/configobj.py", line 894, in walk
    val = function(self, entry, **keywargs)
  File "/usr/share/weewx/weeutil/logger.py", line 164, in _fix
    if isinstance(section[key], (list, tuple)):
  File "/usr/lib/python3/dist-packages/configobj.py", line 557, in __getitem__
    return self._interpolate(key, val)
  File "/usr/lib/python3/dist-packages/configobj.py", line 549, in _interpolate
    return engine.interpolate(key, value)
  File "/usr/lib/python3/dist-packages/configobj.py", line 352, in interpolate
    value = recursive_interpolate(key, value, self.section, {})
  File "/usr/lib/python3/dist-packages/configobj.py", line 330, in recursive_interpolate
    k, v, s = self._parse_match(match)
  File "/usr/lib/python3/dist-packages/configobj.py", line 417, in _parse_match
    value, section = self._fetch(key)
  File "/usr/lib/python3/dist-packages/configobj.py", line 386, in _fetch
    raise MissingInterpolationOption(key)
configobj.MissingInterpolationOption: missing option "asctime" in interpolation.

# ==== END ERROR MESSAGE =====

any help is always appreciated!

DDJ

Tom Keffer

unread,
Sep 22, 2021, 12:16:41 PMSep 22
to weewx-user
Hi, Doug

I was able to replicate this. The problem is that ConfigObj, the utility that we use to read the configuration files, is trying to interpolate your 'format', rather than simply pass it on to the logging facility.

Replace your version of logger.py with the attached. You should find it in /usr/share/weewx/weeutil/logger.py

Let me know if that helps.

-tk



logger.py

Doug Jenkins

unread,
Sep 22, 2021, 3:43:43 PMSep 22
to weewx-user
Tom:

Thanks for the hotfix! I do appreciate it such a quick turnaround on a small item like this! :)

I switched the logging.py file at /usr/share/weewx/weeutil/ directory and tested it with both wee_reports and the standard weewx service. In both cases, Weewx formatted the log using my formatter expression. I made some changes to the formatter in my weewx.conf to ensure it was responding to the changes. The log reported the changes as expected.

I did not test wee_database, wee_import, or wee_device, but they probably will utilize this logging.py change and passthrough the formatter as expected.

Please let me know if you need any additional testing for this file or a issue/PR logged for the item. 

Again thanks for the great solution and continuous development of weewx! 

DDJ

Reply all
Reply to author
Forward
0 new messages