4 beta test report

[Greg Troxel]

I just updated to the tip of master (plus a local commit to change
prefix to /usr/weewx). After installing, weewxd crashes on boot trying
to set up the backup logger. I am unclear on what's going on here and
starting to read code.

I wonder if there are some Linux path assumptions hard-coded in, but
that's an unsubstantiated guess. Does this work ok for everybody else?

My system in NetBSD8, earmv7hf-el (RPI3), and has been running weewx
3.9.x and earlier just fine for just over 2 years.

The system does have working syslog, and lines have been appearing in
/var/log/messages via syslog, when I was running 3.9.2:

Mar 7 20:25:28 wx weewx[1249]: rsyncupload: rsync'd 17 files (83,033 bytes) in 0.82 seconds
Mar 7 20:30:14 wx weewx[1249]: engine: Main loop exiting. Shutting engine down.
Mar 7 20:30:14 wx weewx[1249]: engine: Shutting down StdReport thread
Mar 7 20:30:15 wx weewx[1249]: engine: Terminating weewx version 3.9.2

The crash looks like:

$ bin/weewxd
Traceback (most recent call last):
File "bin/weewxd", line 261, in <module>
main()
File "bin/weewxd", line 80, in main
weeutil.logger.setup(options.log_label, {})
File "/usr/home/gdt/SOFTWARE/WEEWX/weewx/bin/weeutil/logger.py", line 148, in setup
logging.config.dictConfig(log_dict)
File "/usr/pkg/lib/python2.7/logging/config.py", line 794, in dictConfig
dictConfigClass(config).configure()
File "/usr/pkg/lib/python2.7/logging/config.py", line 576, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'syslog': [Errno 2] No such file or directory

[Thomas Keffer]

Hi, Greg

weewx V4 uses a new logging facility, Python's module 'logging', which requires that one set up logging handlers. The handler "syslog" is designed to (roughly) emulate the previous syslog logging. To set it up, you must specify a destination (called 'address') and a 'facility', which vary from OS to OS. I suspect that the defaults that weewx is picking for NetBSD are wrong.

Take a look at the top of file weeutil/logger.py where these destinations are coded in. It looks like:

if sys.platform == "darwin":
    address = '/var/run/syslog'
    facility = 'local1'
elif sys.platform.startswith('linux'):
    address = '/dev/log'
    facility = 'user'
elif sys.platform.startswith('freebsd'):
    address = '/var/run/log'
    facility = 'user'
else:
    address = ('localhost', 514)
    facility = 'user'

I don't know which branch NetBSD would take. Either 'freebsd' or the default, I imagine. Obviously, what it's choosing isn't working out. Perhaps you can supply an alternative for NetBSD? It may be as simple as a different file path for 'address'.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/rmiwo7v1uzc.fsf%40s1.lexort.com.

[mwall]

greg,

i have switched to openbsd on two of my production weewx systems (too many hassles with systemd and other recent linux-isms that have made the systems unstable and too difficult to manage).

so there will be more direct bsd testing as i bring those systems online.

m

[Greg Troxel]

[Thanks very much for the logging hint; I think that fixes 99%; patch to follow]

Also saw this; haven't looked into it yet.

Traceback (most recent call last):

File "/usr/weewx/bin/weewx/reportengine.py", line 197, in run
obj.start()
File "/usr/weewx/bin/weewx/reportengine.py", line 280, in start
self.run()
File "/usr/weewx/bin/weewx/imagegenerator.py", line 41, in run
self.genImages(self.gen_ts)
File "/usr/weewx/bin/weewx/imagegenerator.py", line 180, in genImages
aggregate_interval=aggregate_interval)
File "/usr/weewx/bin/weewx/xtypes.py", line 86, in get_series
aggregate_interval)
File "/usr/weewx/bin/weewx/xtypes.py", line 138, in get_series
for stamp in weeutil.weeutil.intervalgen(startstamp, stopstamp, aggregate_interval):
File "/usr/weewx/bin/weeutil/weeutil.py", line 339, in intervalgen
stamp2 = int(time.mktime(dt2.timetuple()))
OverflowError: mktime argument out of range

[Greg Troxel]

Thomas Keffer <tke...@gmail.com> writes:

> weewx V4 uses a new logging facility, Python's module 'logging', which
> requires that one set up logging handlers. The handler "syslog" is designed
> to (roughly) emulate the previous syslog logging. To set it up, you must
> specify a destination (called 'address') and a 'facility', which vary from
> OS to OS. I suspect that the defaults that weewx is picking for NetBSD are
> wrong.

Thanks for the hint. Patch attached. With this weewx runs, gets
archive records, posts them to mqtt, and creates and rsyncs the html
pages. And log messages appear in /var/log/messages (which is where
syslog puts them per local config, tos that's fine.).

I am still seeing an issue, but I am 99.9% sure it is unrelated.

> Take a look at the top of file weeutil/logger.py where these destinations
> are coded in. It looks like:
>
> if sys.platform == "darwin":
> address = '/var/run/syslog'
> facility = 'local1'
> elif sys.platform.startswith('linux'):
> address = '/dev/log'
> facility = 'user'
> elif sys.platform.startswith('freebsd'):
> address = '/var/run/log'
> facility = 'user'
> else:
> address = ('localhost', 514)
> facility = 'user'
>
> I don't know which branch NetBSD would take. Either 'freebsd' or the
> default, I imagine. Obviously, what it's choosing isn't working out.
> Perhaps you can supply an alternative for NetBSD? It may be as simple as a
> different file path for 'address'.

It was.

I would suggest that if there isn't a match, that the code throw an
exception rather than assume a default. The UDP port 514 might be for
Linux, and if so I think it would be better to just put that first and
be explicit.

OpenBSD is likely also /var/run/log; this layout is inherited from
somewhere along the 4.3BSD to 4.4BSD lineage (syslogd originated in
4.3BSD, but I think 4.4 regularized some paths into the modern BSD
hier(7) layout.)

[mwall]

On Sunday, March 8, 2020 at 10:52:37 AM UTC-4, Greg Troxel wrote:

Thomas Keffer writes:

> weewx V4 uses a new logging facility, Python's module 'logging', which
> requires that one set up logging handlers. The handler "syslog" is designed
> to (roughly) emulate the previous syslog logging. To set it up, you must
> specify a destination (called 'address') and a 'facility', which vary from
> OS to OS. I suspect that the defaults that weewx is picking for NetBSD are
> wrong.

Thanks for the hint.  Patch attached.  With this weewx runs, gets
archive records, posts them to mqtt, and creates and rsyncs the html
pages.   And log messages appear in /var/log/messages (which is where
syslog puts them per local config, tos that's fine.).  

changes pushed to master at commit 89918d28 

also added change to support openbsd, which uses /dev/log

port 514 is the default for remote system logging (rsyslog).  not sure why anyone would ever do localhost:514...

[mwall]

On Sunday, March 8, 2020 at 10:52:37 AM UTC-4, Greg Troxel wrote:

OpenBSD is likely also /var/run/log; this layout is inherited from
somewhere along the 4.3BSD to 4.4BSD lineage (syslogd originated in
4.3BSD, but I think 4.4 regularized some paths into the modern BSD
hier(7) layout.)

here is a nice summary of bsd/linux log behavior, thanks to stackexchange:

https://unix.stackexchange.com/questions/205883/understand-logging-in-linux/294206 

[vince]

On Sunday, March 8, 2020 at 10:50:20 AM UTC-7, mwall wrote:

port 514 is the default for remote system logging (rsyslog).  not sure why anyone would ever do localhost:514...

rsyslog to me doesn't mean remote, it's just one of many possible syslog daemons you could have installed, so I could see it being used that way if you wanted to force logging through a daemon that can do rewrite rules etc.

[Greg Troxel]

So I don't understand why it makes sense to have this compiled in, if
it's sysadmin's choice vs the standard interface on a particular OS.

FWIW, on NetBSD syslogd can be made to listen on UDP port 514. Actually,
this is the default beahvior of syslogd, but the standard startup
scripts pass -s ("secure") to not listen to the network.

(This being in weewx at all is surprising to me; I would expect the
logging module to have any necessary per-os defaults. But I am very
likely not understanding something. On BSDs there is syslog(3), a
library interface to syslog, and programs using that do not have to know
how to talk to syslogd. But that is not in POSIX, I'm pretty sure.)

[Thomas Keffer]

The problem is that the OS is not generally known until runtime. Well, you could pick it at install time, but then that would require the same branching "if statement," done earlier, which is not a better deal. Or, you could make the user pick it in weewx.conf, but most users wouldn't have a clue what to pick.

Nevertheless, the user can change these options by using an undocumented feature to override the handlers. For example, you could have put this in weewx.conf and ended up in the same place:

[Logging]
    [[handlers]]    
        # System logger
        [[[syslog]]]
            address = '/dev/log/user'
            facility = 'user'

I want to see how this interface works out before documenting it.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/rmimu8qxwtz.fsf%40s1.lexort.com.

[vince]

On Sunday, March 8, 2020 at 7:50:51 PM UTC-7, Thomas Keffer wrote:

I want to see how this interface works out before documenting it.

Just curious, but could you explain what changed re: logging in v4 vs. v3 ?

Typically userspace apps just let the os's syslog functionfigure things out based on the facility/severity of the message being logged....or they write directly to their own logfile someplace.

What problems did switching gears attempt to solve ?

[Thomas Keffer]

It's all in the Wiki white paper.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d5780813-e152-46ec-ba7f-978348e077a5%40googlegroups.com.

[vince]

On Monday, March 9, 2020 at 12:55:43 PM UTC-7, Thomas Keffer wrote:

It's all in the Wiki white paper.

Interesting stuff - thanks.....

At the very bottom of the wiki there is a line saying "filepile:"  (?)

You might want to make it a little clearer that is just an example, before folks copy+paste that literally as-is....

Nice writeup !

[Thomas Keffer]

LOL! Good eye! I've replaced it with something more general.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/724e1e5b-b2b3-428f-8db4-20e78b2cc05f%40googlegroups.com.