V5.0.0b17 available

[Tom Keffer]

Hopefully, we are getting very close to a final release for V5.0!

Warning!

But, until then, this is still very much a beta release. You could break your installation, and it might take skills to recover. If you're using pip in a virtual environment, the risk is low of breaking other things, but you will have to shut down your production instance temporarily. For a package installer, try it in a VM, or on another computer, first.

Pip

For pip installs, please delete your old virtual environment, then install from scratch by following the new pip install instructions. While upgrading should work, we are particularly interested in the experience of a new install, including setting up a daemon and udev files. Make sure to follow the new instructions that use a daemon setup script.

Debian

For Debian package installs, modify /etc/apt/sources.list as follows:

echo "deb [arch=all] https://weewx.com/apt-test/python3 buster main" | sudo tee /etc/apt/sources.list.d/weewx.list

Note the "apt-test". This tells apt to look there for the beta release, instead of the normal repository. You may want to change it back when you're done.

Red Hat

For Red Hat package installs, put this in /etc/yum.repos.d/weewx.repo

[weewx]
name=weewx
baseurl=http://weewx.com/yum-test/weewx/el9
enabled=1

Note the "yum-test". This tells yum to look there for the beta release, instead of the normal repository. Again, you may want to change things back when you're done.

What we're looking for

1. We are interested in your upgrade experience. Did the installers get your configuration file right? Skins? Is the daemon configured correctly?

2. With this release, we have included udev rules to set the correct permissions for devices. Did it work? If not, did unplugging then replugging the device work?

3. How about logging? Version 5 will log to a dedicated weewx file, generally /var/log/weewx/weewx.log. Is that what you're seeing?

Thanks!

-Tom & Matt

[Vince Skahan]

For pip installs, please delete your old virtual environment, then install from scratch by following the new pip install instructions. While upgrading should work, we are particularly interested in the experience of a new install, including setting up a daemon and udev files. Make sure to follow the new instructions that use a daemon setup script.

No issues here on a pi with current RaspiOS and a pip installation but a question...

Do the udev rules assume a 'Davis' Vantage usb logger ?

pi@pi4:/etc/udev/rules.d$ grep -i vantage 60-weewx.rules
# make any davis vantage connected via usb-serial accessible to anyone
SUBSYSTEM=="tty",ATTRS{idVendor}=="10c4",ATTRS{idProduct}=="ea60",MODE="0666",SYMLINK+="vantage"

Should there perhaps be a udev rule for typical serial2usb converters perhaps, or is that something not stable depending on which serial2usb we use ?

pi@pi4:/etc/udev/rules.d$ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 0557:2008 ATEN International Co., Ltd UC-232A Serial Port [pl2303]
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I'm fine here with sticking with the console coming up as /dev/ttyUSB0 since that's stable for me here, but just asking to (re)learn the udev stuff...

Also - reminder to others -  if you're had an old weewx-venv remember to also install your added python modules.  For me here, I add paho-mqtt and requests to the 'pip install weewx' command to have the additional python modules my skins+extensions require.   It might be wise to run 'pip3 list' before you stash the previous venv aside to do a pip install of b17 from scratch...

[Joel Bion]

Good point re: checking package list. I locally need requests and cryptography. 

Sent from my iPhone

[pa...@pauland.net]

item 3. How about logging? Version 5 will log to a dedicated weewx file, generally /var/log/weewx/weewx.log. Is that what you're seeing?

Debian package install under a fresh Debian 12 OS will only log to Systemd's journald. Due to  fact that logrotate and  rsyslog are no longer installed by default.

apt install weewx  creates:

/etc/logrotate.d/weewx

/etc/rsyslog.d/weewx.conf
/var/log/weewx

Very similar behavior with a pip install under Debian 12 if user runs setup-daemon.systemd

But these are all ineffectual as /var/log/weewx/weewx.log will never be created as there is no rsyslog or logrotate running.  

Logging is only done to Systemd's journald and can be viewed with something like sudo journalctl -f -u weewx

Big fear I have is the number 1 support issue of WeeWX V5 is going to become "/var/log/weewx/weewx.log is missing".

Maybe a caveat  is needed that WeeWx V5 will try to create a dedicated log file, but may not under some OS.

Currently under the WeeWx Debian set up  Verify section there is this 

check the log file:

tail -50 /var/log/weewx/weewx.log

Rather than be helpful this will trigger confusion when the command fails because .var/log/weewx/weewx.log does not exist.

BTW just to add to the confusion folks with a fresh Debian 12 install will have the logging issue, while folks that upgraded from a previous version of Debian will not.

Oh bother.

Paul

[matthew wall]

On Friday, December 15, 2023 at 9:25:40 AM UTC-5 pa...@pauland.net wrote:

Maybe a caveat  is needed that WeeWx V5 will try to create a dedicated log file, but may not under some OS.

Currently under the WeeWx Debian set up  Verify section there is this 

check the log file:

tail -50 /var/log/weewx/weewx.log

Rather than be helpful this will trigger confusion when the command fails because .var/log/weewx/weewx.log does not exist.

thank you paul!

we might have to remove all the log-related recipes from the docs and refer to wiki pages instead; one cannot create timeless docs when the os removes tried-and-true methods for interacting with the system.

we had to create a wiki page exactly for this reason:

https://github.com/weewx/weewx/wiki/view-logs

the logrotate and rsyslog conf files are *supposed* to be conditional for the pip installs.  does this mean that a "pure systemd-journald" debian still has a directory /etc/rsyslog.d?  existence of that directory is what we use to determine whether to deploy the syslog and logrotate confs.

we will probably have to move them to the postinst for debian packages (along with the init stuff, since not every system uses systemd).  that would be a hassle, since doing things in post removes all of the file comparisons and version-checking that come with debian packaging.

we considered using python's logging system exclusively, that way we avoid the systemd-journald borgification of logging.  but python's logging has other issues, most notably that it cannot ensure a consistent log when multiple processes try to write to the same file.

on the other hand, we might be safer to just deploy them anyway, because anyone serious about logging will install syslog.

m

[Paul R Anderson]

I saw the conditional checks for

/etc/logrotate.d

/etc/rsyslog.d

Yes those directories do in fact exist  in a "pure systemd-journald" debian 12. Presumably they remain in place so older setup scripts don't fail. But their contents are silently ignored.

Also noted you check later to see if /var/log/syslog exist.

On Debian 12 "pure systemd-journald" /var/log/syslog Does Not EXIST 

If you checked that first , then nestled your /etc/logrotate.d and /etc/rsyslog.d under it would work in Debian 12, can't speak for any other OS.

Paul

 

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/47679f66-1c7a-428d-b3ba-b60c62c54b33n%40googlegroups.com.

[matthew wall]

On Friday, December 15, 2023 at 10:05:13 AM UTC-5 Paul R Anderson wrote:

I saw the conditional checks for

/etc/logrotate.d

/etc/rsyslog.d

Yes those directories do in fact exist  in a "pure systemd-journald" debian 12. Presumably they remain in place so older setup scripts don't fail. But their contents are silently ignored.

Also noted you check later to see if /var/log/syslog exist.

On Debian 12 "pure systemd-journald" /var/log/syslog Does Not EXIST 

If you checked that first , then nestled your /etc/logrotate.d and /etc/rsyslog.d under it would work in Debian 12, can't speak for any other OS

the stat on /var/log/syslog is to ensure that we get the right permissions on the /var/log/weewx directory, since debian, ubuntu, and redhat each have different conventions for that.

so far we have been able to have a single deb package work for a wide range of debian major releases. contrast this with the redhat world, where we *must* have a separate package for each major release.

i think redhat is doing this syslog removal too, although they might have been less aggressive about eliminating syslog since so many enterprise installations have toolchains that depend heavily on syslog.  journald is a poor (and less performant) substitute for syslog, ELK, or any infrastructure that uses syslog, not to mention the obfuscation and grief it causes when you try to diagnose systems that fail early in the boot process. yes, i have bias :)

i'll revisit the conditionals.  hopefully there is a path that serves both the windows-like, one-program-does-everything, reboot-if-it-fails, intended-for-a-laptop direction of systemd without sacrificing the needs of those who want a system to run reliably for a long time with minimal maintenance or update.

m

[Vince Skahan]

5.0b17 looks like it takes a fine approach already to me.

• systemd systems can always use journalctl to read the logs.  Add the '-u weewx' flag to see just weewx entries.

• if you 'also' have rsyslogd installed and enabled, you can 'also' use /var/log/weewx/weewx.log to view the same entries as of 5.0b17

• if you do 'not' have rsyslogd installed+enabled it's no harm that weewx drops config files into place for rsyslogd.  They are tiny and wound not be used.

So it looks perfect to me.  Other than systemd.  I am not on 'team systemd' to say the least although I have to accept that borg won :-)

Users of any and all systemd-laden operating systems just have to accept that if they see references to /var/log/syslog or /var/log/messages that 'in general' they need to either use journalctl to view logs 'the systemd way', or they alternately need to 'add' rsyslogd to use legacy logging that has worked for over 40 years before the systemd borg attacked and won.

FWIW - the official debian12 vagrant box has rsyslog installed, configured, and running with a very normal looking rsyslog.conf, so that's already done for that setup.  logrotate is also present and normally configured with typical entries under /etc/rsyslog.d

If you do official debian12 under 'docker' you get a very minimal setup where neither logrotate nor rsyslogd are installed.  Neither make a lot of sense in docker anyway, but in any event if weewx writes config files for things that don't exist in the os, it's tiny and benign at worst.

[michael.k...@gmx.at]

I am still on b15, but found out about the udev rules for ws23xx just shortly after you added them (see https://groups.google.com/g/weewx-user/c/kol0udZNuyc/m/1WhtZF0kBAAJ). My ws28xx rules are different to the ones  in https://github.com/weewx/weewx/blob/V5/src/weewx_data/util/udev/rules.d/ws28xx.rules

Here is mine:

$ cat /etc/udev/rules.d/99-ws28xx.rules
SUBSYSTEM=="usb", ATTR{idVendor}=="6666", ATTR{idProduct}=="5555", MODE="666"

There are weewx':

SUBSYSTEM=="usb", ATTRS{idVendor}=="6666", ATTRS{idProduct}=="5555", MODE="0666"

Note the Differences: ATTR vs ATTRS "666" vs. "0666"

Since I can access the device as a non-root user this way, ATTR should be sufficient, but I can't tell if there are downsides on also match for parent devices when using ATTRS.

What I don't know: do I have to delete my udev rules for ws28xx, when installing b17?

[Joel Bion]

Here’s a thought:

Why not write a short bash or Python script that looks to see where weewx is logging and when finding it either:

1) prints out help text telling the user where to find it

Or

2) starts dumping the log to stdout. 

And on error, reports “can’t find it”

Is this reasonable? Or are the places so varied and undeterminable at runtime that such a script would be unwritable?

Joel

On Dec 15, 2023, at 9:27 AM, Vince Skahan <vince...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/a2e034c1-22a0-4009-9575-1831506db96fn%40googlegroups.com.

[Vince Skahan]

On Friday, December 15, 2023 at 11:02:59 AM UTC-8 Joel Bion wrote:

Why not write a short bash or Python script that looks to see where weewx is logging and when finding it either:

1) prints out help text telling the user where to find it

Or

2) starts dumping the log to stdout. 

And on error, reports “can’t find it”

Is this reasonable? Or are the places so varied and undeterminable at runtime that such a script would be unwritable?

Think not.  Print out to 'which' user and 'where' ?  There isn't a user to print out to when you run as a daemon.  That's what syslogs are for.

The answer here is not specific at all to weewx:

• on a modern os that uses systemd you need to learn how to use journalctl to see your logs and make that transition in how you do things
• if you chose to 'add' a syslog daemon (there are many to pick from) you need to know where 'that' will log to and how to configure it and view its logs
• if you are running via docker you already have a problem with getting running containers to log someplace in your infrastructure

This is a documentation issue.  Unfortunately logging is complicated because it can be done in so many ways.

Logging in v5 betas has been evolving a bit so I suspect the docs have not caught up completely there.

• https://www.weewx.com/docs/5.0/usersguide/where/#log-files still presumes syslogd is installed and running and there are some other errors there regarding permissions.    Given the os planned to be supported, they should all reference systemd in my opinion.
  
• https://github.com/weewx/weewx/wiki/WeeWX-v4-and-logging is too low-level for me but it is good for how to customize logging of what's logged when by weewx once it's logging 'somewhere'
• https://github.com/weewx/weewx/wiki/view-logs is much more correct to my eyes in the general sense although I disagree a lot there too.  The default is not syslog.  The default is the operating system's logging infrastructure.  That is journald mostly these days and especially for the os variants+versions v5 will support for packages.

As I said, logging is complicated and documenting it clearly and completely for all the os variants is hard to do.

But ultimately you need to know how your os does things.

[Vince Skahan]

(sorry if this is posted twice.  Google groups got crazy on me in the browser)

On Friday, December 15, 2023 at 11:01:51 AM UTC-8 michael.k...@gmx.at wrote:

What I don't know: do I have to delete my udev rules for ws28xx, when installing b17?

5.0b17 is a beta release.  How about doing some testing and let us know what you find !!!!!

Save your old rules for safekeeping.

Install b17.

Tell us what you found.

 

[Joel Bion]

That’s why one option I was suggesting was to just print out the location of the log vs its contents. The idea is to avoid requiring the user to figure out where their log is located, and just either have a tool tell them, or just grab the current log from wherever it was stored and dump it out. I’m aware that daemons use logs for output. But the raised concern was, I thought, that users wouldn’t know where their log was located. If that isn’t a concern, there’s no need for such a script. 

On Dec 15, 2023, at 11:51 AM, Vince Skahan <vince...@gmail.com> wrote:

(sorry if this is posted twice.  Google groups got crazy on me in the browser)

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/09ad9f8b-7dd9-4013-973b-edf968192dc6n%40googlegroups.com.

[Vince Skahan]

Joel - print to 'where' ?  They how do you tell them which tool to use and how to use it ?  There is a limit to how much hand-holding makes sense.

The top paragraphs in https://www.weewx.com/docs/5.0/usersguide/installing/ pretty clearly indicate that users need to have some basic skills.  The next section https://www.weewx.com/docs/5.0/usersguide/where/ mentions where to find logs.

Users need to actually 'read' the Users' Guide and sometimes surf the wiki to supplement that information.  That's why so many hours were spent on the docs.

[Joel Bion]

one more thing, there are tools that log to custom locations and don’t go through the system’s “logging infrastructure” by default.  An example is Apache httpd. As a result, while I run syslog-ng on my hardware, I don’t assume the location of logging for any daemon, but figure it out each time I install something. It’s annoying. 

Sent from my iPhone

On Dec 15, 2023, at 11:59 AM, Joel Bion <jpb...@gmail.com> wrote:



[Joel Bion]

I sympathize 100% with this. I’ll drop my point then :)
> Joel - print to 'where' ? They how do you tell them which tool to use and how to use it ? There is a limit to how much hand-holding makes sense.

[pa...@pauland.net]

MATTHEW

Pondering more on weewx set up creating

/etc/logrotate.d/weewx
/etc/rsyslog.d/weewx.conf
/var/log/weewx
While these are all ineffectual as /var/log/weewx/weewx.log will never be created, there harmless and silently ignored.
There may be a good reason to create them anyway.
If a user decides to install the debian rsyslog package AFTER they have install WeeWx it will install logrotate as well as a dep. Then logging will start to be written to /var/log/weewx/weewx.log
I have verified logging this on a test system. I assume that logrotate will honor the settings in /etc/rsyslog.d/weewx.conf and work as well, I did force a manual log rotate on weewx and it worked

 

Paul

[Greg Troxel]

"pa...@pauland.net" <pa...@pauland.net> writes:

> Pondering more on weewx set up creating
> /etc/logrotate.d/weewx
> /etc/rsyslog.d/weewx.conf
> /var/log/weewx
> While these are all ineffectual as /var/log/weewx/weewx.log will never be
> created, there harmless and silently ignored.

Generally, all of this should be conditional on figuring out if the OS
has these concepts. And, modern packaging has a strong bias to only
writing within your prefix. (I have yet to try out weewx 5 myself.)

[Vince Skahan]

In effect they are so.  Config files that are never read because the component that 'uses' the file is not present are pretty benign considering how tiny they are. 

[matthew wall]

that is an argument to put everything into post, since a generic package declaration has no way to know whether the target system uses upstart, sysV, or systemd, much less whether the target uses only systemd-journald, a combination of systemd-journald and (rsyslog|syslog-ng), or just syslog. and that is not even considering the case that paul points out, where you install a syslog-less base os then install rsyslog later so that logging will happen as <deity> intended ;)

it is even more complicated by the way systemd is sort-of-but-not-really compatible/interoperable - it keeps many of the other artifacts around but ignores them.  or, like networkmanager, fights with them.

anyway, i think we are zeroing in on two approaches:

1) deploy the syslog and logrotate conf files if those directories exist, whether or not the operating system has syslog enabled.  uninstall will remove those files.  the post-pip script does the same thing.  (note to self: we need to make that script have both 'install' and 'uninstall' options)

2) make weewx do its own logging to file in ~/weewx-data/log (or /var/log/weewx for deb/rpm), independent of any system logging. postgres, grafana, influx, ELK, atlassian, and many others do it this way, with a configuration option to send their messages to syslog.  that way repackagers, administrators, or anyone gets a good out-of-box experience, but can integrate into their toolchains with little fuss (ok, a lot of fuss for atlassian, thank you java).

i have updated the docs at https://weewx.com/docs/5.0 to reflect (1), as we are not ready to change weewx to the (2) pattern.  its hard enough to tell someone how to find a system log, let alone distinguish between weewxd.log, weectl-import.log, weectl-database.log, etc.

[pa...@pauland.net]

V5.0.0b17 pip install
setup-daemon.systemd fails when run with sudo with this error
Cannot find utility files at location '/root/weewx-data/util'
Because when run with sudo
UTIL_ROOT=$HOME/weewx-data/util
Becomes
UTIL_ROOT=/root/weewx-data/util

Paul

On Thursday, December 14, 2023 at 5:52:51 PM UTC-5 Tom Keffer wrote:

[Tom Keffer]

Did you run the version of setup-daemon.systemd in ~/weewx-data/scripts? It should have expanded $HOME to the location of the location of weewx-data.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/9cdd016b-ecba-4ea6-9b08-45506f7d6cafn%40googlegroups.com.

[Paul R Anderson]

Sorry not sure what I'm doing wrong.

panders@cont-weewx-git:~$ sudo sh /home/panders/weewx-data/scripts/setup-daemon.systemd

Cannot find utility files at location '/root/weewx-data/util'

panders@cont-weewx-git:~$ 

Paul 

[Vince Skahan]

This worked for me - deb12 in a vagrant vm running as user 'vagrant'.

I followed the docs, just added --no-prompt to the station setup to not have it ask questions

sudo apt update
sudo apt install python3-pip -y
sudo apt install python3-venv -y
python3 -m venv ~/weewx-venv
source ~/weewx-venv/bin/activate
python3 -m pip install weewx
source ~/weewx-venv/bin/activate
weectl station create --no-prompt
sudo sh ~/weewx-data/scripts/setup-daemon.systemd
sudo systemctl start weewx
sudo systemctl status weewx

What precise os are you running ?

Are you trying to do it in docker perhaps ?

We'd need to see your exact steps start to finish please.

[pa...@pauland.net]

Good sleuthing Vince !
Running Debian 12 on a test LXD Container. System was already set and running was only testing setup-daemon.systemd
Verified  setup-daemon.systemd works properly on a physical machine running Debian 12. Attribute the error to  ENV variables and their handling in the LXD container.
So sorry for raising the alarm bell. If I think something seems weird, and it's in a LXD test container I always verify on real hardware... except for today. 

Paul

[Tom Keffer]

No, Paul, it's actually a bug. The variable UTIL_ROOT should have been expanded. 

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/f0bc9c46-7965-4e59-b46d-4d68f8bdad44n%40googlegroups.com.

[Tom Keffer]

Fixed in commit 4bf9ccd .

[pa...@pauland.net]

Confirmed commit 4bf9ccd

setup-daemon.systemd runs as expected with no errors. Even on my edge case LXD container install.

BTW to confirm this I used the instructions for Running WeeWX from a git repository, against the development branch. And I can confirm that the it worked perfectly.

Paul