v5.0.0a13 Observations

[G Hammer]

I spun up a clean Rocky Linux 9.1 container on Proxmox today to give v5 install a shot.

I created a user so as to not be installing as root, though I am going to use sudo, so...

I used a variety of resources to get the general flow.

This one was good, but didn't hold your hand.

https://github.com/weewx/weewx/wiki/pip-install-strategies

This one had a bit more handholding, but was incomplete and was debian oriented.

https://github.com/weewx/weewx/blob/V5/docs/quickstarts/pip.md

Between the two, I thought I had enough to install.

sudo python3 -m pip install weewx

That resulted in a good install, but a few items were not done with pip as they needed wheel to be available.

There were also PATH warnings that look to be in error as the directories are in the path. Plus, weewxd and utilities ran fine without explicit full paths.

I'm attaching the output from the install as weewx5-install.txt mainly to show these items.

I have installed weewx-mqtt and weewx-weatherlink-live, reconfigured for these and all is running fine.

Next I will add the systemd service file and enable it.

I was kinda hesitant to go down the pip path. For no reason it seems. This was easier than my traditional setup.py install.

[Tom Keffer]

Thanks for that!

The error "Using legacy 'setup.py install' for pyephem, since package 'wheel' is not installed." is exactly that. You need to install 'wheel' before using pip:

sudo pip install wheel.

That little detail should really be in the quickstart guide.

I suspect the "PATH" warning is because the path when running 'sudo' is different from the path without. Try

sudo -l

and see if "secure_path" is set for your user and, if so, whether it includes /usr/local/bin.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/98edb954-2e38-490c-850c-a90f01f785fdn%40googlegroups.com.

[G Hammer]

As this is a container, I will reset and incorporate the lessons learned today, wheel, etc.

You are correct, the path does not contain /usr/local/bin when using sudo.

secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin

I imagine visudo could solve that. But, are there other possibilities?

[Tom Keffer]

The warning only happened because pip did a path check after it had been invoked using 'sudo', so it would not see anything that was not in secure_path. 

When it comes time to actually run weewxd, you most likely would not be using 'sudo', but rather simply run a daemon as the root user. The root user probably has /usr/local/bin in its path, if not simply add it in .profile (or equivalent).

End result: it's probably a false warning that can be ignored.

Wanted to ask you: earlier you said that the pip install instructions are "incomplete." I'm too close to this thing, so it's good to have some other eyes take a look. Anything in particular that you found incomplete or confusing?

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/97464335-dcf9-4f12-af83-0351e5f6dbcan%40googlegroups.com.

[G Hammer]

I will start fresh and document what I see sometime over the weekend.

However, root user running directly, would need the entire path to the config passed. Attempting simply weewxd results in a failure to find weewx.conf.

See weewx5-root.txt

[Tom Keffer]

Need a little more information. 

Did you run "weectl station create" before attempting to run weewxd? It creates weewx.conf so, without it, there isn't any configuration file to be found.

If you ran it without any parameters it will put the file in ~/weewx-data, which, in this case, would be /root/weewx-data.

If you ran it with a file path, it will set it up at that path.

Let me know what you did.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/af039e68-4514-4773-894f-581b3fe940c5n%40googlegroups.com.

[G Hammer]

Tom,

The install was done with a regular account (wx) using sudo, not root.

sudo python3 -m pip install weewx

In this case the conf is in /home/wx/weewx-data
I took a look at the systemd service file in weewx-data/utils and it is configured with the correct path to config.

When root uses the command line as written in the service file, weewx runs properly.

/usr/local/bin/weewxd /home/wx/weewx-data/weewx.conf

[Vince Skahan]

I did a little rocky/9 testing in a vagrant VM and came up with a script that works for a one-step install 'after' you log in as a non-privileged user who can sudo.  The result is that weewx is started via systemd and runs as the non-privileged user.

# required for building pyephem
sudo yum install -y python3-devel gcc python3-pip
pip3 install --user wheel

# install weewx in Simulator mode as an unprivileged user
pip3 install --user weewx
weectl station create --no-prompt

# selinux will prevent systemd from starting weewxd if enforcing
#  this forces the context to match /bin/ls which seems to work
#  although I'm not sure which context really should be applied here

#  nor whether future updates 'unprivileged' require this too
chcon -R --reference /bin/ls ~/.local/bin

# put the systemd service into place and start weewx up
cd ~/weewx-data
sudo cp util/systemd/weewx.service /etc/systemd/system
sudo systemctl daemon-reload
sudo systemctl enable weewx
sudo systemctl start weewx

[...Tom - let me know if you want a PR to update the v5 RH quickstart to use systemd and to fix up selinux (ugh) getting in the way...]

[Tom Keffer]

Good to know. In that case, weewxd is acting as expected. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/4806ff3d-3883-45e1-b1ee-e6f2dca348a3n%40googlegroups.com.

[Tom Keffer]

Maybe I'm wrong, but I doubt very many people will be using SELinux and, if they do, they should be prepared for these kinds of problems. I don't want to clutter up the quick guides for these rare cases.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/d478e71c-ac60-4fdf-abd6-d55377b186f4n%40googlegroups.com.

[Vince Skahan]

Disagree....

• On modern RH systems, selinux is enabled/enforcing by default.  RHish users are victims here. Their vendor gave them the os set up with selinux enforcing for better or worse as the starting point.  Much like RH systems have to use 'yum' rather than 'apt', they also have to deal with selinux one way or the other.  Few RH users know this, nor know how.

• On a modern RH system,  no --user installation can possibly work at bootup without fixing the selinux context (the recommended way) or disabling selinux completely (not recommended, and requires editing a file in /etc/sysconfig as root and rebooting before proceeding).

Bottom line is we're talking about needing to add 'two' lines to the quickstart (one to install wheel, one to run chcon), and using the system init file contents (copy the debian tab verbatim and make it appear in the RH tab too).

This also lets you be able to finally kill using init.d for modern RH systems and move to systemd like  for modern Debian systems.  Matthew's already said he wants to go systemd everywhere in the packages.  Why not get there for RH systems (via pip) too ?

Again, I'm happy to do the two minute PR to make the RH --user instructions actually correct and able to succeed.  I've already tested that they work.

[Tom Keffer]

Thanks for the education --- I know next to nothing about Red Hat systems. I'll take you up on the PR.

If you look in pip.md under "Preparation", you'll see a tab for Debian. Add one for RH systems. That will keep the clutter to a minimum.

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/f6bd6251-2ab5-489b-a787-a3dcee4c268cn%40googlegroups.com.

[G Hammer]

Leaving the selinux issue alone (my Rocky Linux 9.1 lxc installs have no enforcement out of the box).

I needed to clean up my archive as it had data from old PWSes in it that caused silly entries in Seasons.

I changed the data to NULL (as they were all battery readings of one sort or another).

Then I was going to drop daily and rebuild daily

wee_database doesn't like either of the commands though it did work for --check.

wee_database-5.txt is attached to show the errors.

[Vince Skahan]

Same errors here on my rocky 9 VM and a matching debian-11 VM...

[Tom Keffer]

Fixed, I think, in v5.0.0a14.

Upgrade with

python3 -m pip install weewx --upgrade --user

On Sun, Jan 29, 2023 at 8:52 AM Vince Skahan <vince...@gmail.com> wrote:

Same errors here on my rocky 9 VM and a matching debian-11 VM...

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/1687c1d9-5213-43e6-8c46-fcbfa2974a13n%40googlegroups.com.

[G Hammer]

It is working.

Thanks.

[Vince Skahan]

Seems to work on rocky9.  I did notice that my weewx.conf still shows 5.0.0a13 for the version after the upgrade.

--

-----  vince...@gmail.com ----

[G Hammer]

My weewx.conf also shows a13 but seasons shows

WeeWX version 5.0.0a14
Skin Seasons 5.0.0a13

[Tom Keffer]

Unlike earlier versions of WeeWX, install and upgrade is a two-step process. You need to finish things off with

weectl station upgrade

-tk

[G Hammer]

The weectl station upgrade ran ok, but has a disconcerting prompt.

[wx@wx-pip ~]$ weectl station upgrade

Upgrade station at None? (Y/n) y

[Vince Skahan]

That's going to confuse folks.   An upgrade can reasonably be assumed to actually do it all shouldn't it ?

Is there any way to make it do the right thing in one step ?

[Tom Keffer]

On Sun, Jan 29, 2023 at 5:51 PM Vince Skahan <vince...@gmail.com> wrote:

That's going to confuse folks.   An upgrade can reasonably be assumed to actually do it all shouldn't it ?

Is there any way to make it do the right thing in one step ?

Nope. pip does not allow you to freely roam the file system making changes. Recall our earlier conversation about this.

The only possibility is to come up with a shell script that invokes pip, then follows up with "weectl station upgrade". Let me think about that.

[Tom Keffer]

On Sun, Jan 29, 2023 at 5:50 PM G Hammer <gary....@gmail.com> wrote:

The weectl station upgrade ran ok, but has a disconcerting prompt.

[wx@wx-pip ~]$ weectl station upgrade

Upgrade station at None? (Y/n) y

Thanks for that. I think I see what happened.

-tk

[Joel Bion]

The only possibility is to come up with a shell script that invokes pip, then follows up with "weectl station upgrade". Let me think about that.

--

Exactly. IMO, trying to shoehorn everything into the capabilities of “pip/wheel” with its model of virtual environments leads to extremely inefficient and complicated results - all because Python is optimizing itself for use cases that don’t quite match a single-instance system service. Fail2ban is another example. I have a hunch that a bash script that invokes pip/wheel for part of the stuff, and does the other stuff itself - outside of the python install paradigm - is going to lead to a simpler (and perhaps cleaner?) result.

[Tom Keffer]

Fixed in commit8cb5481

[G Hammer]

Small spelling error in docs/upgrading.htm

You may then wish to upgrade your configuration file, documentation, examples, and utility files by usign

using

[Vince Skahan]

Sure.  So a weewx update requires:

• updating weewx in python-land via pip
• resynching the weewx-data tree to catch up to the pip-installed content
• restarting the daemon (maybe/possibly/usually)

FWIW, this one seems like an opportunity for a wrapper script, regardless of language selected.  The catch-22 of course is what if a weewx upgrade changes the wrapper itself too in the first step, as well as dealing with what if step-N failed for whatever reason.

But a "weectl software upgrade" command for a wrapper would be kind of nice if that's reasonable.

I also wonder if 'weewx station upgrade' isn't super clear terminology either, but I don't have a better suggestion there.   What bad happens if they answer 'no' to the prompt, for example ?  Also, can we pass it a --yes switch like apt-get so we can fully script an upgrade and/or upgrade-skins ?

[matthew wall]

On Sunday, January 29, 2023 at 9:38:02 PM UTC-5 vince...@gmail.com wrote:

Sure.  So a weewx update requires:

• updating weewx in python-land via pip
• resynching the weewx-data tree to catch up to the pip-installed content
• restarting the daemon (maybe/possibly/usually)

FWIW, this one seems like an opportunity for a wrapper script, regardless of language selected.  The catch-22 of course is what if a weewx upgrade changes the wrapper itself too in the first step, as well as dealing with what if step-N failed for whatever reason.

please, no!

 

But a "weectl software upgrade" command for a wrapper would be kind of nice if that's reasonable.

upgrading the software should be completely separate from 'upgrading' a skin, database, non-core driver, or any other part of weewx that is not in the core.

when you do an upgrade of the software, your other stuff should just work.  if it does not, then the software is broken - it is not backward-compatible.

changes in weewx core that break extensions or skins should be indicated by the appropriate semantic version change, and drivers/skins should be released and documented separately as appropriate.

the 'weewx version' should come from weewx code, *not* the weewx conf file.  (imho, the 'version' variable should not be in any weewx conf file, since it cannot be trusted)

each skin may have a version number that is independent of anything in weewx code (we default to putting the weewx release version number in the skins that ship with weewx, since at this time we cannot be bothered to track per-skin versions for skins that ship with weewx)

 

I also wonder if 'weewx station upgrade' isn't super clear terminology either, but I don't have a better suggestion there.   What bad happens if they answer 'no' to the prompt, for example ?  Also, can we pass it a --yes switch like apt-get so we can fully script an upgrade and/or upgrade-skins ?

lets keep it simple and start with the fundamental tools.  if you want layers of wrappers, use a java app like matlab or something from atlassian ;)

to upgrade weewx:  pip install weewx --upgrade

(or 'apt install weewx' or 'yum install weewx', with options to pin at a specific version in each case)

if you want to 'upgrade' your skins, config file, or database, then there must be separate commands to do that.  but that is a corner case that should almost never have to happen, since the weewx software is supposed to be backward-compatible.  in fact, doing an 'upgrade' to a config file is really just a "tidy up and remove older syntax, but does not change functionality in any way.

non-core skins and drivers will require upgrades from time to time, but that is done using wee_extension.  skins should not be 'upgraded' when i do an upgrade to weewx!  nor do i want weewx mucking with my data when i upgrade the software.

so that leaves us with:

* use 'weectl' to explicitly 'tidy up' a conf file (should never be necessary, except maybe for major releases)

* use 'wee_extension' to upgrade your non-core skins/drivers

* use 'weectl station upgrade' to upgrade your core skins

now this *could* use some massaging to make it more consistent - lets figure this out instead of adding layers of wrapper scripts.

btw, 'weectl station upgrade' might better be 'weectl conf upgrade'.  we started with 'station' because the intent was to encapsulate conf file and any other assets into a 'station'.  but it might be better semantics to just call it a config.

m 

[G Hammer]

As a user with zero programming ability and limited ability to follow code, I'd like to say that the two step process is fine with me.

As this is all new, I missed the requirement for the second step. Once pointed out by Tom, no problem.

FWIW I used to only install with setup.py. Upgrades were 'simple' once you downloaded the file, untared, changed directory, did the build and install commands. Oh wait, not that simple at all.

So far, the pip method has proven to be simpler for me to install and to understand.

[G Hammer]

Should I be able to install a non-core skin in the alpha?

I just attempted to install the belchertown skin and it failed.

Attaching btown-weewx5.txt for the result.

[matthew wall]

On Monday, January 30, 2023 at 10:32:40 AM UTC-5 gary....@gmail.com wrote:

Should I be able to install a non-core skin in the alpha?

not yet.

 

[Tom Keffer]

On Mon, Jan 30, 2023 at 7:32 AM G Hammer <gary....@gmail.com> wrote:

Should I be able to install a non-core skin in the alpha?

I just attempted to install the belchertown skin and it failed.

You should be able to, but Belchertown uses an older method of importing the ExtensionInstaller, which V5 was not prepared to accept. This has now been fixed in v5.0.0a15.

-tk

[Vince Skahan]

Thanks Tom.  My ecowitt skin (link) now installs ok.  If there are pointers to the current way to import it, let us know :-)

[Tom Keffer]

You mean, besides the docs for weectl extension install?

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/270ed790-2416-4c6f-91f8-6ed3aa3b747cn%40googlegroups.com.

[Vince Skahan]

yes - something else.

You mentioned that previously the installer didn't work because the skin "uses an older method of importing the ExtensionInstaller", as all of mine do too.  I was asking what the newer/current method looks like.  Is that documented someplace I missed finding ?

[Tom Keffer]

Ah! Sorry, now I get your question.

Not documented, but the example "basic" follows it. 

Old style:

from setup import ExtensionInstaller

New style:

from weecfg.extension import ExtensionInstaller

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/a19d8ded-952f-4ef8-9518-3ed5615ffb48n%40googlegroups.com.

[G Hammer]

Thanks. All is working well on my install now.

The alpha is in use for my home display feed now.