Package weewx like other python projects

[Till Maas]

Hi,

currently, I am looking into writing a driver for the tinkerforge
outdoor weather station[0] and I have most of the code and
understanding. However, since weewx is using its own conventions to
install python code, it makes it hard for me to use standard approaches
(using a CI for tests, using pylint, ...).

I wonder if there is any interest/support for patches to make weewx
follow common python packaging methods, such as

- installing python modules into /usr/lib/pythonX.Y/site-packages
- make it possible to install weewx with pip
- make weewx support normal python modules as extensions
instead/additionally of a custom installation format

Since the dev notes[1] mention a pythonic code base as a goal so I hope
extending this to the packaging would be welcome.

What do you think?

Kind regards
Till


[0]
https://www.tinkerforge.com/en/shop/outdoor-weather-station-ws-6147.html
[1]
http://weewx.com/docs/devnotes.htm

[Tom Keffer]

Hello, Till

As you can imagine, there have been many discussions both offline and in weewx-development about these topics. The biggest problem is that weewx is a program, not a module, which is what pip typically expects to install.

Maybe you're more of a distutils whiz than I am, but, try as I may, I could not find a combo that allowed pip to install weewx. So, we gave up and went our own way.

But, I'm curious when you say that you cannot use "standard approaches" with weewx. For example, I use PyCharm and Eclipse. Both work fine with our configuration. Any particular problems?

-tk

[Till Maas]

Hi Tom,

On Mon, May 04, 2020 at 08:36:57AM -0700, Tom Keffer wrote:

> As you can imagine, there have been many discussions both offline and in
> weewx-development about these topics. The biggest problem is that weewx is
> a program, not a module, which is what pip typically expects to install.
>
> Maybe you're more of a distutils whiz than I am, but, try as I may, I could
> not find a combo that allowed pip to install weewx. So, we gave up and went
> our own way.

sure, I can give it a try. For this, it might be necessary/simpler to
rename the current setup.py script to make it available for a setuptools
script that follows normal conventions. Then we can adjust until
everything is migrated. Would it be important to be able to install
everything into /home/weewx in the future? Would it suffice to be able
to install into /home/weewx/.local?

> But, I'm curious when you say that you cannot use "standard approaches"
> with weewx. For example, I use PyCharm and Eclipse. Both work fine with our
> configuration. Any particular problems?

What is hurting most is that pip install does not work. Therefore a
custom installation script would be needed to make it available when
creating a tox config that would run unit tests or pylint/pyflake.
Also after installing, it needs a custom PYTHONPATH setting.

Therefore it would be nice to start by moving ./bin/weewx to ./weewx in
the source code. Then adding a build script for creating a module for it
would be simple. Then it looks like it would be good to move:

./bin/daemon.py -> ./weewx/daemon.py
./bin/six.py should be removed in the future or moved into a
external_lib directory since it seems to be a copy of the six python
module.

All the directories like ./bin/weedb could be moved to ./weewx/weedb or
./weewx/db to make them part of the module

The code from the tools like ./bin/wee_util could be moved into
./weewx/weeutil/main.py

Alternatively, the sub packages for the tools could also be moved along
weewx, so

./weeutil/main.py
./weewx/...

etc.

I need to look more into the extension and bin/user to propose a change
for this.

After moving the modules and the main methods, it should be easily
supported to let setuptools create stubs that will run the main method
for the modules.

The code from setup.py to create a new config or to update configs
should probably be moved into their own helper scripts.

To develop drivers, it is most useful to have weewx available as a
pip-installable module, so I suggest to prepare an example with this, if
you like the overall idea.

What do you think?

Thanks
Till

[Tom Keffer]

Sounds like it would break a LOT of stuff! 

It has been painful enough getting extension writers to embrace Python 3 and WeeWX 4. But, that transition required them only to rewrite an extension in Python 3. Old extensions continued to work just fine under Python 2 and WeeWX V4.

If I'm understanding your proposal, something like

import weedb.sqlite

would become

import weewx.weedb.sqlite

That would require hundreds of changes across dozens of extensions. The upside just doesn't seem worth it.

-tk

[Vince Skahan]

I wonder if there is any interest/support for patches to make weewx
follow common python packaging methods, such as

- installing python modules into /usr/lib/pythonX.Y/site-packages
- make it possible to install weewx with pip
- make weewx support normal python modules as extensions
  instead/additionally of a custom installation format
 

Personally I'm not seeing any real benefit but I certainly see a 'lot' of risk and pain.

I'm still not getting the 'why' part.  You want to tear apart everything because you need to set PYTHONPATH ?

Why does anybody 'need' weewx installable via pip anyway ?  Seems to me that setup.py is a bit of a solved problem at this point.  Same with the extension installer.  At this point there are dozens of extensions and drivers and skins installable in that format.  Another solved problem as far as I can tell.   FWIW, I would not be willing to repackage all my personal extensions + skins etc. into another format at this point.  They work fine as-is.  The extension installer scratches that itch for me.

Why again would it be wise to turn everything upside down at this point ?

[Tom Keffer]

Slow down there, everyone. Let's give Till a chance to respond. It's possible I'm totally misunderstanding what he's thinking.

Install via pip would be very nice if it was possible. One repository for all platforms.  V4 was an opportunity to try it, so I probably spent a solid week banging away at the problem. But, in the end, I didn't like the results.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/aaf77e00-4470-4c58-bf56-42346b1e410d%40googlegroups.com.

[Vince Skahan]

Certainly. Too many scope-increase meetings on the project from hell at work I guess.  Sorry if I jumped the gun :-)

Anyway, I guess I always try to think "it it was my money funding this, would I invest here at this time", because time+blood pressure are things we all have limited budgets for too.

So I guess I'd be interested in understanding:

• What problems does pip solve ?   What problems does it cause ?

• Same questions re: where things are installed to.   How does switching the installed locations help/hurt the as-is ?
  

• Does switching to pip change how painful (or not) it is to release an updated distribution ?   Digitally sign them ?   Etc....
  
  
• Does switching to pip alter the 'git pull ; python3 setup.py build ; python3 setup.py install' kind of use case the folks more on the developer bleeding edge side tend to work ?
  

• Does it alter anything re: extension installation tools or extension structure ?

Lots of moving parts in weewx's ecosystem to think about....

[Till Maas]

Hi Tom,

On Mon, May 04, 2020 at 02:26:11PM -0700, Tom Keffer wrote:
> Sounds like it would break a LOT of stuff!
>
> It has been painful enough getting extension writers to embrace Python 3
> and WeeWX 4. But, that transition required them only to rewrite an
> extension in Python 3. Old extensions continued to work just fine under
> Python 2 and WeeWX V4.
>
> If I'm understanding your proposal, something like
>
> import weedb.sqlite
>
> would become
>
> import weewx.weedb.sqlite
>
> That would require hundreds of changes across dozens of extensions. The
> upside just doesn't seem worth it.

thank you for the feedback. The naming was just a proposal to be able to
discuss something. There are some options here to avoid this problem for
extension authors:

1) copy ./bin/weedb to ./weedb etc or

2) Add some extra code to ensure the old paths still work and add
deprecation warnings. For example the python path could be adjusted
before importing packages or a shim with a warning could be added.

I am sure we can find a solution that allows both to make it a proper
python module and keep it backwards compatible.

Thanks
Till

[Greg Troxel]

My $0.02, coming from packaging many things:

People who develop a particular package are very used to how it is
built, and so more or less however that works is fine for them.

People that are coming to something new, or packaging systems that
have say 20K packages, would like to see a small number of common
build systems, and for any package that can reasonably fit into an
existing approach to use that approach.

Users generally would like to "apt install" or "pkg_add" something,
and be unaware of all of this. This means that a one-off install
process has to be accomodated in each packaging system.

So therefore I would like to be able to have weewx behave like most
other things and invoke it with the usual prefix arg for where I want
it, and have everything be in a many-things-in-one-prefix normal
place.

My impression is that pip is basically a packaging system (that only
works for python) that expects the setuptools interface. So making
things work nicely with setuptools for various packaging systems and
making it work for pip is probably more or less the same thing.

My observations of weewx are that

1) what it is needing to do is fundamentally not odd. A bunch of
python libs, some programs that need those libs (bin), some config
files (etc) some place to write data (var), docs, and so on.

2) It seems that 99.5+ % of python stuff using some sort of
distutils/setuptools. I am not entirely clear on the details any
more, but I did figure it out to spiff up pkgsrc support for this and
the fine points have left my head. I don't see any reason that using
the standard approach wouldn't work for everything except updating the
config file

3) config files are awkward, partially because there are often
packaging systems in the middle. It seems best to have the
installation and the config merging script be separate. packages can
hook the merging script for package install time if that makes sense,
but the package build will do an "install" operation to a destdir
before tarring it up.

4) The current weewx build basically demands its own prefix. I would
expect to see libs in the usual place and scripts in ${PREFIX}/bin.
That means config in ${PREFIX}/etc/weexx/, and various other things in
the right compartmentalized names for many things in a PREFIX. I
raised this earlier and my understanding is that various Linux
distribution packages adjust the standard build. But this seems
suboptimal; there's no problem with a standard layout even if one has
only weewx in a prefix

Of course, there is the question of "Ideally how would this be if we
were starting from scratch" and then the questions of "how hard is it to
get from here to there" and "is it worth it".

[Vince Skahan]

On Tuesday, May 5, 2020 at 9:39:45 AM UTC-7, Greg Troxel wrote:

Of course, there is the question of "Ideally how would this be if we
were starting from scratch" and then the questions of "how hard is it to
get from here to there" and "is it worth it".

Yup - same things I was thinking.

Problem is that every os, and every packaging system, expect 'their' paths to be 'the' standard.

Obligatory xkcd reference - https://xkcd.com/927/

[Greg Troxel]

Vince Skahan <vince...@gmail.com> writes:

> Problem is that every os, and every packaging system, expect 'their' paths
> to be 'the' standard.

There are some very slight variations, but the venerable BSD hier(7) was
changed fairly little into FHS. And, autoconf is the ur-typ of how this
is done, with --etcdir and various other flags to set particular
directories. Packaging systems are very used to passing those to meet
their norms.

[Till Maas]

On Mon, May 04, 2020 at 04:53:13PM -0700, Vince Skahan wrote:
>
> Certainly. Too many scope-increase meetings on the project from hell at
> work I guess. Sorry if I jumped the gun :-)
>

> Anyway, I guess I always try to think "*it it was my money funding this,
> would I invest here at this time*", because time+blood pressure are things

> we all have limited budgets for too.
>
> So I guess I'd be interested in understanding:
>

> - What problems does pip solve ? What problems does it cause ?

pip is the standard installer for python modules. It allows to

Install python tools (system-wide or per-user), tracks what it installed
and allows to remove and uninstall the files. Also it supports
dependencies, so that it can install all the necessary python modules
that are required (for example configobj, cheetah, pil ...)

Together with tools like venv, it allows to create custom python
development versions in specific directories. This allows to scope the
installation even finer. For example, I could have a different
development environment for weewx than for another project that might
require different python versions or module versions.

Then with tools like tox, one can easily create different test
environments to run unit tests or static code analyzers for different
python versions. This allows to check for unused variables, code style
violations, missing/unnecessary imports etc.

This could all be included in a system that then automatically performs
these checks for each change to the code.

I would like to use this for the driver that I want to develop. Since
every other module than weewx follows the standard approach, I can
specify them as requirements for the project and it the system installs
them automatically without any special code for weewx.

Consider that all modules would use their custom installation method,
then it would need quite some effort to install all the modules that
weewx uses (configobj, cheetah, pil, serial, usb, pip, ephem, ...).

> - Same questions re: where things are installed to. How does switching

> the installed locations help/hurt the as-is ?

By using the standard method, it would be also very easy to package for
different distributions, since they are optimised for the default
approach. For example Fedora supports automatically generating the
python dependences based from the information of the pip-compatible
package. Currently, the weewx spec contains a manually created list of
RPM dependencies. Also the installation instructions would be simpler
because it would just set the default paths for the distribution. I
assume it would be similar for other distributions since using a common
format makes it easier for distributions to simplify things.

Since I do not know the reasons, why weewx does differently, we are
having this conversation to see if it is feasible. From my
understanding, the biggest hurt would be to learn the new paths for
existing developers. For new developers, they would be hurt less because
weewx works like other existing python projects.

> - Does switching to pip change how painful (or not) it is to release an

> updated distribution ? Digitally sign them ? Etc....

My understanding is that it makes things easier for packaging since it
will make weewx be more like other projects.

> - Does switching to pip alter the '*git pull ; python3 setup.py build ;
> python3 setup.py install*' kind of use case the folks more on the

> developer bleeding edge side tend to work ?

It should still work. If there are interactive steps that are done as
part of the install script, they might need to run a different, tool,
too.
Otherwise, you should be able to benefit from using pip with this, too:
git pull; pip install --user --upgrade .

Also, python3 setup.py install* might not remove files from older weewx
installations that would not be installed anymore but could cause
problems. So this might also change.

> - Does it alter anything re: extension installation tools or extension
> structure ?

It is not necessary to change this but a logical next step would be for
(future) extensions to also be pip-installable instead of using a custom
weewx method to copy files.

> Lots of moving parts in weewx's ecosystem to think about....

Yes, thank you for providing this detailed feedback.

Thanks
Till

[Chris Alemany]

I was actually looking at the “brew” system, which is Linux and MacOS friendly, just yesterday to see how packages were created in that system. It might be a good candidate?

Cheers
Chris

> --
> You received this message because you are subscribed to the Google Groups "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/rmipnbip9mo.fsf%40s1.lexort.com.

[Vince Skahan]

On Tuesday, May 5, 2020 at 11:26:02 AM UTC-7, Chris Alemany wrote:

I was actually looking at the “brew” system, which is Linux and MacOS friendly, just yesterday to see how packages were created in that system. It might be a good candidate?

I'd lean toward 'no' on brew for packaging weewx, as it's yet-another-non-standard packager/installer ala that XKCD comic I mentioned.

If we went pip, the Mac users would already be covered.

FWIW, I installed python3 on my Macbook via brew, and it's great having a /usr/local/bin/python3 interpreter there that is safe from os carnage if Apple gives us any 'features' down the road.  It also lets me mess up my python3 setup without risk of hosing the os's python the other way (which I've done twice in the last decade due to pip dependency hell and openssl changes over the years).

[Greg Troxel]

Chris Alemany <chri...@gmail.com> writes:

> I was actually looking at the “brew” system, which is Linux and MacOS
> friendly, just yesterday to see how packages were created in that
> system. It might be a good candidate?

I have said "packaging system", by which I mean

every Linux distribution
pkgsrc (10+ operating systems, esp. NetBSD and illumos)
FreeBSD ports
OpenBSD ports
brew
macports
fink
[probably a dozen more]

and I didn't really mean

pip (which is a python-only packaging system)
[all the rest of the single-language systems]

I don't think it makes sense at all to talk about using a particular
packaging system for weewx. Even if so, brew seems nowhere near
portable enough to suffice, covering a mere 2 operating systems.
Certainly brew should *contain* weewx, but weewx should not be a
brew-only thing.

Till has explained things super well, and I concur with those comments.
This is about doing things the normal python way so that

weewx is normal in python
therefore pip is smooth
therefore *all* of the above systems can wrap weewx easily

[mwall]

On Tuesday, May 5, 2020 at 2:24:47 PM UTC-4, Till Maas wrote:

By using the standard method, it would be also very easy to package for
different distributions, since they are optimised for the default
approach. For example Fedora supports automatically generating the
python dependences based from the information of the pip-compatible
package.

it would be nice if this actually worked.  for example, python3-cheetah is available in debian10 but not in debian9.  python-pyserial and python-pyusb are not available in centos7.  so there are some operating system releases in which you must do a 'pip install' (debian9) or pull something from epel (centos8) before weewx will run.

unless the package does all of the dependencies for them (or if they want additional python modules), users face the dilemma of "do i use the native packaging or do i use pip" when manually installing python dependencies.

there seem to be many options, but no unified approach

[Greg Troxel]

mwall <mw...@users.sourceforge.net> writes:

> it would be nice if this actually worked. for example, python3-cheetah is
> available in debian10 but not in debian9. python-pyserial and python-pyusb
> are not available in centos7. so there are some operating system releases
> in which you must do a 'pip install' (debian9) or pull something from epel
> (centos8) before weewx will run.
>
> unless the package does all of the dependencies for them (or if they want
> additional python modules), users face the dilemma of "do i use the native
> packaging or do i use pip" when manually installing python dependencies.
>
> there seem to be many options, but no unified approach

In pkgsrc, the notion is that each python module is built for all
supported python versions (without attempting to delete 27 while it is
still in widespread use), and each package simply depends on packages
for everything it needs. On my weewx installation, I have all the
needed python modules installed via pkgsrc.

Your comment that a number of packaging systems/distributions lack
things that arguably they ought to have is of course a fair point.

[Tom Keffer]

As I said earlier, I put quite a bit of time trying to get pip to work. You can see the results in the 'pip' branch of the repository. NB: it's a work in progress, so the branch is not a "finished" dead end, but rather an "idea" dead end. It follows your ideas: executables and libraries are in different locations, and the install destinations can be changed via setup.cfg or command line arguments to setup.py. I couldn't get it to work, but this was a couple months ago, and I don't recall the details, although it revolved around weewx being an application, not a library. 

In fact, I would appreciate it if you could point me to any non-trivial application that pip installs. I couldn't find any. Just libraries.

Requirements for a pip install:

1. Ability to install into one directory, such as /home/weewx. It's OK if it takes some contorted setting of install directories in order to do this, but I like having all state, all libraries, all code, in one spot. It makes my life easier.
2. Ability to install in "standard" locations.
3. If we install in standard locations, and if the names of the weewx modules are not changed (and I don't think they should be), then there is a risk of a name collision for common names like 'schema' and 'user. A strategy for this.
4. A "one pass" install, rather than pip, followed by invoking yet another program that asks the user for station settings. Right now, we do this by invoking wee_config with appropriate parameters from within setup.py (this part is not finished in the pip branch).
5. Pip allows you to upload a given version number only once. This would require an automated version naming tool, otherwise it's a righteous pain the butt. I don't know anything about tox. Can it do something like this?
6. Support for our "standard platforms," which generally includes the platforms listed in Matthew's compatibility matrix.

I am very open to switching if a good value proposition can be made for doing so, and if you, or someone, can own the results. 

Definitely no NIH on my part around this!

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/20200505182444.GF16242%40genius.invalid.

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> In fact, I would appreciate it if you could point me to any non-trivial
> application that pip installs. I couldn't find any. Just libraries.

home assistant

which is sort of like weewx, but more complicated.

My home assistant install, via pip in a venv, has 119 modules.

> Requirements for a pip install:
>

> 1. Ability to install into one directory, such as /home/weewx. It's OK

> if it takes some contorted setting of install directories in order to do
> this, but I like having all state, all libraries, all code, in one spot. It
> makes my life easier.

Is that really a pip requirement? pip can install in a virtualenv, and
for instance I have all homeassistant stuff in a venv in ~/HASS.



I have not been arguing for 'pip is the only way', but 'install in
normal path layout so that using this from pip and every other packaging
system is easy'. Packaging systems generally want to natively wrap
things. (I am not sure if TIll agrees with this view.)


As for namespaces, weewx may need to install all modules as weewx/foo
instead of foo. That doesn't seem terrible.

[Tom Keffer]

On Wed, May 6, 2020 at 6:30 AM Greg Troxel <g...@lexort.com> wrote:

Tom Keffer <tke...@gmail.com> writes:

> In fact, I would appreciate it if you could point me to any non-trivial
> application that pip installs. I couldn't find any. Just libraries.

home assistant

which is sort of like weewx, but more complicated.

My home assistant install, via pip in a venv, has 119 modules.

 

Thanks. Can you point me to its page on pypi? All I'm seeing is this one, and it seems to be a front-end framework, not an application: https://pypi.org/project/home-assistant-frontend/

> Requirements for a pip install:
>
>    1. Ability to install into one directory, such as /home/weewx. It's OK
>    if it takes some contorted setting of install directories in order to do
>    this, but I like having all state, all libraries, all code, in one spot. It
>    makes my life easier.

Is that really a pip requirement?  pip can install in a virtualenv, and
for instance I have all homeassistant stuff in a venv in ~/HASS.

Yes, that can work, although it has a different set of problems. 

• Another tool to learn.
• Yes, everything in one virtual environment, but the bits and pieces are scattered about within that environment.
• Venvs tend to be "disposable", so it's no place to put the database, which would still have to be put somewhere else.
• Same with skins

As for namespaces, weewx may need to install all modules as weewx/foo
instead of foo.  That doesn't seem terrible.

Yes, that's the solution, but it would require patching all *.conf files to reflect the new location of user modules and schemas. Or, we can install a custom __import__ that first tries (for example) weewx.schema, then schema.

I hope I'm not coming across as passive-aggressive, because, really, I'm not. It's just that all these solutions have been considered in the past and rejected for one reason or another. This isn't the first time we've looked at this. The requirements I listed in an earlier email are there for a reason.

-tk

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> On Wed, May 6, 2020 at 6:30 AM Greg Troxel <g...@lexort.com> wrote:
>
>> Tom Keffer <tke...@gmail.com> writes:
>>
>> > In fact, I would appreciate it if you could point me to any non-trivial
>> > application that pip installs. I couldn't find any. Just libraries.
>>
>> home assistant
>>
>> which is sort of like weewx, but more complicated.
>>
>> My home assistant install, via pip in a venv, has 119 modules.
>>
>
> Thanks. Can you point me to its page on pypi? All I'm seeing is this one,
> and it seems to be a front-end framework, not an application:
> https://pypi.org/project/home-assistant-frontend/

pip show prints out

Name: homeassistant
Version: 0.108.6
Summary: Open-source home automation platform running on Python 3.
Home-page: https://www.home-assistant.io/
Author: The Home Assistant Authors
Author-email: he...@home-assistant.io
License: Apache License 2.0
Location: /d0/u0/home/gdt/HASS/lib/python3.7/site-packages
Requires: astral, ruamel.yaml, voluptuous, python-slugify, importlib-metadata, requests, attrs, pyyaml, voluptuous-serialize, jinja2, ciso8601, aiohttp, pip, pytz, cryptography, async-timeout, PyJWT, certifi, bcrypt
Required-by:

and I think this is:

https://pypi.org/project/homeassistant/

(The -frontend package is I think the stuff that gets downloaded to your
browser to access the server.)


I have been upgrading this via pip for quite a while.

>
>> Requirements for a pip install:
>> >
>> > 1. Ability to install into one directory, such as /home/weewx. It's OK
>> > if it takes some contorted setting of install directories in order to
>> do
>> > this, but I like having all state, all libraries, all code, in one
>> spot. It
>> > makes my life easier.
>>
>> Is that really a pip requirement? pip can install in a virtualenv, and
>> for instance I have all homeassistant stuff in a venv in ~/HASS.
>
> Yes, that can work, although it has a different set of problems.
>

> - Another tool to learn.
> - Yes, everything in one virtual environment, but the bits and pieces

> are scattered about within that environment.

> - Venvs tend to be "disposable", so it's no place to put the database,

> which would still have to be put somewhere else.

> - Same with skins

I was not suggesting that everybody use pip/venv. I was just commenting
about the pip way.

I favor having a distribution that follows norms and then having that
distribution and all dependencies wrapped by a packaging system (each
one, really), so that on any given system, the native package tools work
for weewx, just like they do for the large number of other packages.

My understanding, pretty fuzzy, is that there is a debian package that
rearranges a bunch of the install stuff.

>> As for namespaces, weewx may need to install all modules as weewx/foo
>> instead of foo. That doesn't seem terrible.
>
> Yes, that's the solution, but it would require patching all *.conf files to
> reflect the new location of user modules and schemas. Or, we can install a
> custom __import__ that first tries (for example) weewx.schema, then schema.

I don't follow this. weewx.conf is not in python and I don't see module
names in it. I'm sure I'm missing something here.

Is this also about third-party modules that want to import things?

> I hope I'm not coming across as passive-aggressive, because, really, I'm
> not. It's just that all these solutions have been considered in the past
> and rejected for one reason or another. This isn't the first time we've
> looked at this. The requirements I listed in an earlier email are there for
> a reason.

You are not coming across as unreasonable. It is rather seeming like
the historical path has led to a bad situation, where weewx isn't doing
thigns the normal way, leading to it needing special manual attention,
and it's also awkward to adjust it.

I would be inclined to not so much focus on pip per se but to move
towards a bin, lib/pythonN.Y, etc/weewx, var/weewx, and so on split and
namespace cleanup as necessary (perhaps limited to things in
lib/pythonN.Y all being under weewx). That is probably 90% of what pip
needs.

[Tom Keffer]

Not sure, what exactly the application needs to be to be non-trivial.
This is a rather big project with several CLI tools:
https://pypi.org/project/ansible/

Oooh, that's a good example. Largeish, and it installs with a simple "pip install ansible". Wish I had found it sooner.

If I install using

pip3 install ansible --root=/home/ansible --prefix=""

I get

/home/ansible
├── bin
│   ├── ansible
│   ├── ansible-config -> ansible
│   ├── ansible-connection
│   ├── ansible-console -> ansible
│   ├── ansible-doc -> ansible
│   ├── ansible-galaxy -> ansible
│   ├── ansible-inventory -> ansible
│   ├── ansible-playbook -> ansible
│   ├── ansible-pull -> ansible
│   ├── ansible-test
│   └── ansible-vault -> ansible
└── lib
    └── python3.5
        └── site-packages
            ├── ansible
            ├── ansible-2.9.7-py3.5.egg-info
            └── ansible_test

Close to what we're looking for --- you'd have to specify a rather complicated PYTHONPATH when running 'ansible'. 

> Requirements for a pip install:
>

>    1. Ability to install into one directory, such as /home/weewx. It's OK

>    if it takes some contorted setting of install directories in order to do
>    this, but I like having all state, all libraries, all code, in one spot. It
>    makes my life easier.

In my opinion, it would be nice to be able to run weewx from a GIT
working copy and it being pip-installable does not contradict this. So
in general this should work. Not sure, if you also consider helper
configuration like logrotate or init config to be in that directory, but
I guess, not.

I almost always run weewxd from a git repository, and just specify the path to the configuration file weewx.conf, which is off somewhere else. Because its modules are located under the executable, it can find its own libraries. Specifying a PYTHONPATH is not necessary. Kind of 'node' style. Frankly, I don't know why more python programs aren't configured this way.

>    2. Ability to install in "standard" locations.
>    3. If we install in standard locations, and if the names of the weewx

>    modules are not changed (and I don't think they should be), then there is a
>    risk of a name collision for common names like 'schema' and 'user. A
>    strategy for this.

What is depending on these names? Are these only extensions that weewx
will install and import/execute or also something external? What
locations do things expect them to be? There could be compat package that
contains files in /usr/share/weewx/ that provide the new names. Also the
python path could be adjusted for extensions that weewx loads.

Both weewx proper and extensions depend on these names. Right now, the are usually accessed through dynamic loading, specified by the configuration file. For example, a driver might be configured as:

[Acme]

  driver = user.acme

  port = /dev/ttyUSB0

So, we would either have to patch the configuration file to change all instances of user.acme to weewx.user.acme, or have a fancy import which tries to import weewx.xxx first, then, if that files, import xxx.

 

>    4. A "one pass" install, rather than pip, followed by invoking yet

>    another program that asks the user for station settings. Right now, we do
>    this by invoking wee_config with appropriate parameters from within
>    setup.py (this part is not finished in the pip branch).

As long as the "one pass" install does not need to be called setup.py,
it would be rather easy to add a helper like "local-install" that will
then call setup.py/pip and the script to configure the station settings.
Would this be enough?

I don't care what it's called. What I'm looking for is

pip install weewx

and the program ready to go, without a second "configuration" pass. But, this requirement is not absolute. We already have a configuration utility, so requiring the user to run it after installing is not so bad. But, I guarantee users will not always do it, and it will be a source of support questions!

>    5. Pip allows you to upload a given version number only once. This would

>    require an automated version naming tool, otherwise it's a righteous pain
>    the butt. I don't know anything about tox. Can it do something like this?

Not sure, what the problem is, here. You will only release for example
4.0.0 only once, wont't you? tox is not the tool to address something
like this, it simplifies running commands for different python
versions/environments.

No problem. I just found it was tedious to have to always assign new version numbers while testing the pip installation machinery. But, once you get it working, it becomes less of a burden.

>    6. Support for our "standard platforms," which generally includes the

>    platforms listed in Matthew's compatibility matrix

>    <https://github.com/weewx/weewx/wiki/weewx4-compatibility-matrix>.

This is not a problem. I am a little bit sad that Fedora is not
considered a standard platform, though. ;-)

Matthew would know better than me, but the vast majority of users use CentOS, not Fedora.

-tk

[Vince Skahan]

On Thursday, May 7, 2020 at 4:06:41 PM UTC-7, Tom Keffer wrote:

This is not a problem. I am a little bit sad that Fedora is not
considered a standard platform, though. ;-)

Matthew would know better than me, but the vast majority of users use CentOS, not Fedora.

For Fedora 32 just follow the CentOS 8 instructions.   The rpm installs just fine.   Weewx seems to run just fine.

One oddness - the vagrant box for fedora/32-cloud-base doesn't seem to have a syslog rpm installed by default.  Anybody know what the syslog daemon is for current Fedora ? 

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> If I install using
>
> *pip3 install ansible --root=/home/ansible --prefix=""*

>
> I get
>
> /home/ansible
> ├── bin
> │ ├── ansible
> │ ├── ansible-config -> ansible
> │ ├── ansible-connection
> │ ├── ansible-console -> ansible
> │ ├── ansible-doc -> ansible
> │ ├── ansible-galaxy -> ansible
> │ ├── ansible-inventory -> ansible
> │ ├── ansible-playbook -> ansible
> │ ├── ansible-pull -> ansible
> │ ├── ansible-test
> │ └── ansible-vault -> ansible
> └── lib
> └── python3.5
> └── site-packages
> ├── ansible
> ├── ansible-2.9.7-py3.5.egg-info
> └── ansible_test
>
> Close to what we're looking for --- you'd have to specify a rather
> complicated PYTHONPATH when running 'ansible'.

Yes, but if you had created a venv, and then run the program with the
python in the venv, it would all work.

If weewx installed in $perfix/lib/pythonN.Y/site-packages/weewx/, then
it would be doing the normal thing, aligned with above. If that prefix
is the same prefix as python was built with -- which is also the normal
thing -- then it will all work fine.

> I almost always run weewxd from a git repository, and just specify the path
> to the configuration file weewx.conf, which is off somewhere else. Because
> its modules are located under the executable, it can find its own
> libraries. Specifying a PYTHONPATH is not necessary. Kind of 'node' style.
> Frankly, I don't know why more python programs aren't configured this way.

Being able to do that from git doesn't seem in conflict with the install
using standard paths.

> Both weewx proper and extensions depend on these names. Right now, the are
> usually accessed through dynamic loading, specified by the configuration
> file. For example, a driver might be configured as:
>
> [Acme]
> driver = user.acme
> port = /dev/ttyUSB0
>
> So, we would either have to patch the configuration file to change all
> instances of user.acme to weewx.user.acme, or have a fancy import which
> tries to import weewx.xxx first, then, if that files, import xxx.

It seems easy to have the code that does import check if the driver/whatever
starts with weewx., and if not 1) warn and 2) prepend it. That
provides for a soft transition with little pain.

> I don't care what it's called. What I'm looking for is
>

> *pip install weewx*

>
> and the program ready to go, without a second "configuration" pass. But,
> this requirement is not absolute. We already have a configuration utility,
> so requiring the user to run it after installing is not so bad. But, I
> guarantee users will not always do it, and it will be a source of support
> questions!

I am actually looking for this NOT to be the case. In dealing with
packaging systems, installation of software is done to a staging
diretory, and then tarred up, laster untarred, and then that system is
configured. So it's great to have the config step invokable from the
installed copy of weewx, but in a packaging system I need to disable
that from the regular install.



I continue to not understand the sharp focus on pip proper. The real
compatibility issue is that setup.py installs to non-standard paths that
don't play well sharing a prefix with packaging systems and other
installed python software. Once that's resolved then pip should just
work, modulo installation and post-installation config being separable
steps -- even if users are directed to run a script that does setup.py
and then config, so it amounts to the same steps as now.

[Tom Keffer]

At this point, I think we've discussed this enough and it's time to experiment. From the discussion, I'd say the requirements are:

1. Ability to install into one directory, such as /home/weewx. It's OK if it takes some contorted setting of install directories in order to do this, but I like having all state, all libraries, all code, in one spot. It makes my life easier. I'm not real keen on having the libraries buried deep in this directory, but I'm willing to try it and see how it feels.

1. Ability to install in "standard" locations.

1. Skins, and the user directory cannot be touched by an upgrade.
2. weewx.conf gets automatically patched by an upgrade. This is a big feature of weewx over the standard package upgrades, which merely ask a user if they want to "merge" their old config file with an incoming, new config file.
3. The modules 'user' and 'schema' would have to move to avoid name collisions, to weewx.user and weewx.schema, respectively. To maintain backwards compatibility, dynamic imports would automatically try their new locations.
   
4. Support for our "standard platforms," which generally includes the platforms listed in Matthew's compatibility matrix.
   

Did I miss anything?

Any volunteers?

-tk

How about a volunteer?

[Rich Bell]

I know am a bit late to the party. I am also no Python expert and I know that I will be able to work with any new packaging/install. But, I want to note my workflows that are super easy with today's packaging/install

1. Debugging base WeeWx code

    1. Git fork and clone

    2. Copy known weewx.conf

    3. Update code as necessary

    4. Checkin and create pull request

Depending on how WeeWX is installed, between 3 and 4 I believe I would need to do 1 or more diffs to get the updated code back to my git repository. After that, I should probably do another install/test to make sure all my updates are in my repository.

2. Extension development

    1. Download setup.py install type of WeeWX version and untar it

    2. Install versions of python via pyenv

    3. Install prerequisites via pip in each pyenv environment

    4. Via PYTHONPATH and PYENV_VERSION settings run unit tests and integration tests for each environment. Locally this weewx4/python2.17, weewx4/python3.5, weewx4/python3.6, weewx3/python2.17. In the CI environment it is weewx4/python2.17, weewx4/python3.5, weewx4/python3.6, weewx4/python3.7, weewx4/python3.8, weewx4/python3.9, weewx3.7.1/python2.17, weewx3.9.2/python2.17.

I care less about the CI environment, because its all automated. But suppose someone reports a bug against my extension and weewx4.1. Today, I download weewx4.1. Set the PYTHONPATH and PYENV_VERSION and easily run unit tests and integration tests against 6 versions of python. If I understand correctly, assuming I want to keep my install of weewx4.0/python-x, I'd have to do another install of python-x and weewx-4.1 into it.

Again, I know I can accomplish what I need with new packaging/install of WeeWX. But, as an extension writer right now it super easy to support the matrix of WeeWX versions and python versions.

    - rich

Again, I know I can accomplish what I need with new packaging/install of WeeWX. But, as an extension writer right now it super easy to support the matrix of WeeWX versions and python versions.

[Till Maas]

Hi,

On Sat, May 09, 2020 at 07:12:46PM -0700, Tom Keffer wrote:
> At this point, I think we've discussed this enough and it's time to
> experiment. From the discussion, I'd say the requirements are:
>

> 1. Ability to install into one directory, such as /home/weewx. It's OK

> if it takes some contorted setting of install directories in order to do
> this, but I like having all state, all libraries, all code, in one spot. It
> makes my life easier. I'm not real keen on having the libraries buried deep
> in this directory, but I'm willing to try it and see how it feels.

> 2. Ability to install in "standard" locations.
> 3. Skins, and the user directory cannot be touched by an upgrade.
> 4. weewx.conf gets automatically patched by an upgrade. This is a big

> feature of weewx over the standard package upgrades, which merely ask a
> user if they want to "merge" their old config file with an incoming, new
> config file.

What exactly is happening here?

> 5. The modules 'user' and 'schema' would have to move to avoid name

> collisions, to weewx.user and weewx.schema, respectively. To maintain
> backwards compatibility, dynamic imports would automatically try their new
> locations.

There are also six.py, which I guess could be removed and users should
be asked to install it from their distribution or via pip and daemon.py
(which I guess could be moved in the weewx package).

The user directory is the directory where wee_extension adds new
extensions, isn't it?

On first thought, the best approach for me would be that the recommended
way to install extensions would be that they use pip as well for
installation. Everything that wee_extensions installs, would be best
moved to /var/lib/weewx/user or similar to ensure that it does not
conflict with the packaged files from weewx. Seems like it would be
$HOME/.local/share or $XDG_DATA_HOME if installing per-user. There does
not seem to be a better counter part for /var/lib.

> 6. Support for our "standard platforms," which generally includes the

> platforms listed in Matthew's compatibility matrix

> <https://github.com/weewx/weewx/wiki/weewx4-compatibility-matrix>.
>
> Did I miss anything?

The items in ./util are not managed by setup.py, are they?

> Any volunteers?

I would like to contribute. My challenge is that I am rather new to
weewx, so I do not have the full picture, yet.

Thanks
Till

PS: Tom, I resent this because I forgot to add the list in my first
reply.

> --
> You received this message because you are subscribed to the Google Groups "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEDvA1SKJK6c5iEJ%3DbVpyBXgQ_4FiG-HPuNmX5KJSU%3Dqzg%40mail.gmail.com.

[Till Maas]

Hi,

On Sun, May 10, 2020 at 07:50:04AM -0700, Rich Bell wrote:

> 2. Extension development
> 1. Download setup.py install type of WeeWX version and untar it
> 2. Install versions of python via pyenv
> 3. Install prerequisites via pip in each pyenv environment
> 4. Via PYTHONPATH and PYENV_VERSION settings run unit tests and

When weewx is installable with pip and declares its dependencies
properly, it would be very easy to use tox to automate this:
https://tox.readthedocs.io/en/latest/config.html#generating-environments-conditional-settings
You might still need to install python separately. Fedora packages all
important major versions, therefore I use just them.

Afterwards, you can run all tests just with tox or select certain
combinations with tox -e py27-weewx3,py27-weewx4 for example.

The additional value is that this is how it works on other python
projects, too, and you do not need to write special code to automate
this for weewx.

Thanks
Till

[Vince Skahan]

On Sunday, May 10, 2020 at 8:37:17 AM UTC-7, Till Maas wrote:

On first thought, the best approach for me would be that the recommended
way to install extensions would be that they use pip as well for
installation. Everything that wee_extensions installs, would be best
moved to /var/lib/weewx/user or similar to ensure that it does not
conflict with the packaged files from weewx. Seems like it would be
$HOME/.local/share or $XDG_DATA_HOME if installing per-user. There does
not seem to be a better counter part for /var/lib.

I use the extension installer to install my custom 'skins' as well.

How would that work if wee_extension went away ?

How would I package a custom skin so pip knew what to do ?

Currently I do something ala:

• git clone <reponame> <working_directory_name>
• wee_extension --install <working_directory_name>

[mwall]

On Sunday, May 10, 2020 at 11:37:17 AM UTC-4, Till Maas wrote:

>    4. weewx.conf gets automatically patched by an upgrade. This is a big
>    feature of weewx over the standard package upgrades, which merely ask a
>    user if they want to "merge" their old config file with an incoming, new
>    config file.

What exactly is happening here?

the weewx configuration file (nominally weewx.conf) is modified during the upgrade.  if you use setup.py and install a new release of weewx on top of an existing one, it will modify the existing weewx.conf so it will work with the latest release.  it saves the old file, of course.  this works on weewx config files all the way back to 1.0

if you use dpkg/apt-get/apt to upgrade weewx, after the install it will run wee_config and do exactly the same thing.  but the results use debian nomanclature - "do you want to use your config or use the one suggested by the software author?".  it saves whichever you do not choose so you can compare.

if you use rpm/yum/dnf it does the same thing, but with redhat nomenclature.

a long long time ago, the configuration modification code was embedded in setup.py.  it has since been moved to a module, and it can be called independently of setup.py using wee_config.

 

>    5. The modules 'user' and 'schema' would have to move to avoid name
>    collisions, to weewx.user and weewx.schema, respectively. To maintain
>    backwards compatibility, dynamic imports would automatically try their new
>    locations.

There are also six.py, which I guess could be removed and users should
be asked to install it from their distribution or via pip and daemon.py
(which I guess could be moved in the weewx package).

this is tricky.  six and daemon are just one file each, and they basically never change.  imho, it does not make sense to create a dependency for each of them - just distribute them as part of weewx, and do updates as necessary.

 

The user directory is the directory where wee_extension adds new
extensions, isn't it?

On first thought, the best approach for me would be that the recommended
way to install extensions would be that they use pip as well for
installation. Everything that wee_extensions installs, would be best
moved to /var/lib/weewx/user or similar to ensure that it does not
conflict with the packaged files from weewx. Seems like it would be
$HOME/.local/share or $XDG_DATA_HOME if installing per-user. There does
not seem to be a better counter part for /var/lib.

agree that /var/lib is the appropriate place for this kind of thing.  right now /var/lib/weewx is where the sqlite databases reside.  adding a /var/lib/weewx/user directory would not be unreasonable, but it would require more python path munging.

 

>    6. Support for our "standard platforms," which generally includes the
>    platforms listed in Matthew's compatibility matrix
>    <https://github.com/weewx/weewx/wiki/weewx4-compatibility-matrix>.
>
> Did I miss anything?

The items in ./util are not managed by setup.py, are they?

those are samples, with defaults (typically) set to a /home/weewx configuration.

in the case of deb and rpm installs, most of the util contents are copied to /etc/weewx, and paths changed appropriately.  that way all you have to do on a deb or rpm system is make a symlink to the appropriate file in /etc/weewx.  for example, for logwatch, logrotate, rsyslog, etc.

 

> Any volunteers?

I would like to contribute. My challenge is that I am rather new to
weewx, so I do not have the full picture, yet.

that is a benefit - your eyes are not blinded by looking at the thing for too long :)

i think we have a pretty healthy review process, although it sometimes takes awhile.  (i have been particularly overwhelmed the past year and a half, so i am grateful to those who have covered for me wrt weewx until i get back to it)

[Rich Bell]

Till,

Thanks.  I’ve had it on my list to learn tox. Seems like now is the time.

rich

[Vince Skahan]

On Sunday, May 10, 2020 at 8:37:17 AM UTC-7, Till Maas wrote:

...which I guess could be removed and users should
be asked to install it from their distribution or via pip....

Not naysaying, but every time we ask a user to read and think and do something based on their particular os, there 'will' be 'many' people who don't do so.

Some people won't even try, expecting everything to magically work for them with no thinking require.   Some will try but be unable to understand the instructions, as they're not Linuxy people.  Some will try and fail because they're on odd operating systems that require special instructions.

There are many edge cases too....

A good example is trying to do a dpkg or rpm that requires python3-cheetah when your distro does not 'have' such a package.  If you install cheetah with pip as a second way to get it onto the system, if your package isn't smart enough to support that, a package installation will fail and you would have to use a --force or a --ignore-dependencies (or the like) switch to force the package to install.  Those options cause risk too, and a non-linux person won't understand the risks nor how to deal with adventures if they pop up.

Packaging is hard.

[Greg Troxel]

Vince Skahan <vince...@gmail.com> writes:

> Not naysaying, but every time we ask a user to read and think and do
> something based on their particular os, there 'will' be 'many' people who
> don't do so.

To me, this is a clue that the only reasonable approach for people that are
not trying to develop weewx, is to run a package from their operating
system's package manager.

Other approaches basically require people to understand what they are
doing.

> A good example is trying to do a dpkg or rpm that requires python3-cheetah
> when your distro does not 'have' such a package. If you install cheetah
> with pip as a second way to get it onto the system, if your package isn't
> smart enough to support that, a package installation will fail and you
> would have to use a --force or a --ignore-dependencies (or the like) switch
> to force the package to install. Those options cause risk too, and a
> non-linux person won't understand the risks nor how to deal with adventures
> if they pop up.

That sounds like a broken packaging system. In order to have weewx, it
first has to have all the dependencies. People doing package
maintenance deal with this as part of package creation.


I would tend to say that if the native packaging system is unsuitable,
people should either use a portable packaging system that is, or they
should use pip.

[Tom Keffer]

Greg, I think you're coming around to where I ended up. From a user's perspective, it's hard to beat a well-constructed package for their OS. And, I think we've achieved that.

But the support burden is huge. I was hoping that distutils/setuptools/pypi would make it possible to have a single install system across all systems. 

It might be possible, but I'm maxed out dealing with issues, PRs, new features, etc. Same with Matthew. We would need some fresh blood to get it done.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/rmiy2pztwdj.fsf%40s1.lexort.com.

[mwall]

On Sunday, May 10, 2020 at 2:55:10 PM UTC-4, Vince Skahan wrote:

On Sunday, May 10, 2020 at 8:37:17 AM UTC-7, Till Maas wrote:

...which I guess could be removed and users should
be asked to install it from their distribution or via pip....

A good example is trying to do a dpkg or rpm that requires python3-cheetah when your distro does not 'have' such a package.  If you install cheetah with pip as a second way to get it onto the system, if your package isn't smart enough to support that, a package installation will fail and you would have to use a --force or a --ignore-dependencies (or the like) switch to force the package to install.  Those options cause risk too, and a non-linux person won't understand the risks nor how to deal with adventures if they pop up.

exactly!  for example, we have to say that weewx3 via apt is supported on debian9 and earlier only if you use python2.  weewx3 will run just fine with python3 on older debians, but there is no python3 package of cheetah on those systems, so the deb package dependencies will fail.  to make matters worse, the failures are cryptic (actually, they say exactly what is wrong, but that does not help 99% of the users who see the failures :)  of course, we could create a python3+cheetah3 package for older debian systems and distribute that with weewx, or users could find some backported deb, or...

the variations between ubuntu and debian have also been painful.  the pil-pillow-imaging naming and forking still cause deb package installation dependency problems, and the fact that the conditional dependencies in a deb control file are not really conditional just makes things more confusing when trying to make a universal package.

its kind of funny (in a tragi-comedic way) that many people find it easier to install an entire operating system just to run a single application (i'm looking at you docker!), and that the os developers have come up with snaps and other jails as a 'lightweight' way of doing just about the same thing.  (i know, i am simplifying)

m

 

[Vince Skahan]

On Sunday, May 10, 2020 at 12:36:58 PM UTC-7, Greg Troxel wrote:

Vince Skahan <vince...@gmail.com> writes:

> A good example is trying to do a dpkg or rpm that requires python3-cheetah
> when your distro does not 'have' such a package.  If you install cheetah
> with pip as a second way to get it onto the system, if your package isn't
> smart enough to support that, a package installation will fail and you
> would have to use a --force or a --ignore-dependencies (or the like) switch
> to force the package to install.  Those options cause risk too, and a
> non-linux person won't understand the risks nor how to deal with adventures
> if they pop up.

That sounds like a broken packaging system.  In order to have weewx, it
first has to have all the dependencies.  People doing package
maintenance deal with this as part of package creation.

It's not a broken packaging system, it's operating system differences and trying to use one package to handle all os variants+versions with one package. I've battled this for 15+ years in $work.

A python app packaged in rpm format would run on centos 5, 6, 7, 8 just fine but some versions have corequisite library xyz and some don't.   That would mean the rpm works in some os+version combinations, but not all.  Confusing to the user when a rpm refuses to install and the os gives the user a cryptic not-helpful message.

Same with dpkg installations.  We see differences between ubuntu and debian variants, versions of each os variant, and other dpkg-compatible operating systems (Mint etc.)

I can think of three ways out of this os+variation problem:

• (1) smarter packages - you can certainly build rpms with a lot of conditional logic based on os+version combinations.  I dunno about dpkg.  Downside here is you're always catching up to things people find as the os 'standard' moves due to the upstream vendor
• (2) stop using the os's python modules.  Bundle at least the add-on modules weewx needs 'with' weewx core, located under the weewx tree someplace.  Fix up PYTHONPATH or the like as needed.
• (3) build a zillion packages and host a zillion repos. Have people pick the right repo for their os variant.  See https://rpms.remirepo.net for one good example.  Scroll down in the page for all the os versions+variants he supports.  Amazing.

Option-1 is too much catchup, and too painful

Option-2 is how big apps like splunk and puppet do it.  They bundle the interpreter+libs they need under their tree.   If you think about it, it's also how RHEL chose to work the python issue for RHEL-8.  They don't use the /usr/bin/python3 stuff.  They have their own internally-used-only path with the things 'they' need.

Option-3 is how you'd do it in a CI/CD/autotest world.  The Remi repos are a great example of this.  It's certainly technically possible to build an infinite number of os+version package variants via Docker and Github hooks if we wanted to go that path.

FWIW - I lean to option-2 - just package what weewx needs 'under' the weewx tree the dpkg or rpm installs.  Don't use the os python libs at all.

The downside of option-2 is what would we do if a security vulnerability appears in one debian-ish variant (only) of configobj, as a hypothetical example.  Would you rerelease weewx for just that os+version (perhaps, bump a revision number on the same weewx-4.0.0 base name so yum update sees something needing updating) ?   All those what-if's need to be thought of.

Anyway - personally I really like option-2 as it gets us out of the "did my os provide the expected package for python3-whatever" problem.

[Tom Keffer]

All of the packages WeeWX uses are very mature. It's the same set of pre-requisites that I started with 10+ years ago. HOWEVER, not all of them are pure Python. For example, Cheetah uses an internal C library, _namemapper, for performance. It's name follows the wheel convention (for example, it's _namemapper.cpython-35m-x86_64-linux-gnu.so for Python 3.5 on an x86 64-bit machine), so we'd have to include all of those, too. Ditto with PIL, and maybe ephem.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/087f1bf9-0d44-44c4-989e-ad1e10f59b10%40googlegroups.com.

[Vince Skahan]

On Sunday, May 10, 2020 at 1:27:01 PM UTC-7, Vince Skahan wrote:

• (2) stop using the os's python modules.  Bundle at least the add-on modules weewx needs 'with' weewx core, located under the weewx tree someplace.  Fix up PYTHONPATH or the like as needed.

Just wanted to add one thought and a question for Tom/Matthew before I spend any time fiddling.

At $work we bundle 'all' our OSS software into a separate big-honkin' /opt tree and 'never' install any perl/python variants or modules/libraries to any directory owned by the operating system.   In short, we tell people to use 'our' hand-built perl and python, and we install any modules/libraries they need to under that tree.

So if hypothetically we put all of weewx under /home/weewx still......

• /home/weewx/bin would contain things executable
• /home/weewx/etc would contain config files
• /home/weewx/var would contain things more read/write
• weewx's python3 would be /home/weewx/bin/python3
• weewx's pip3 would be /home/weewx/bin/pip3
• and you would always use weewx's pip3 when you wanted to install modules using pip

Basically all you'd need to do is compile your own python ala "configure --prefix=/home/weewx/python" or something like that....

Or have two packages ?  One with weewx, one with weewx-python ?

 

[Tom Keffer]

Vince, you're the Docker expert. Would it solve some of these problems? Is it widely available?

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/edf62268-4126-47f8-a96e-e6f948f0f55b%40googlegroups.com.

[Vince Skahan]

On Sunday, May 10, 2020 at 2:07:48 PM UTC-7, Tom Keffer wrote:

Vince, you're the Docker expert. Would it solve some of these problems? Is it widely available?

Uncertain what you're asking.  Do you want to release a weewx container that is known to run ?   I think several people have done that.

Now if you're asking whether containers can help with build hell of a zillion variants - sure.   One example is how geerlingguy auto-tests all his ansible modules using TravisCI and documents them on the web - picking a random one look at the page on https://galaxy.ansible.com/geerlingguy/ansible

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> Vince, you're the Docker expert. Would it solve some of these problems? Is
> it widely available?

I don't think docker is adequately portable; it would have to run on
every single posixy system that is currently capable of running python.

And weewx really isn't that complicated. It really does not seem
reasonable to tell people that they have to put it in a container.

[Greg Troxel]

Vince Skahan <vince...@gmail.com> writes:

> It's not a broken packaging system, it's operating system differences and
> trying to use one package to handle all os variants+versions with one
> package. I've battled this for 15+ years in $work.
>
> A python app packaged in rpm format would run on centos 5, 6, 7, 8 just
> fine but some versions have corequisite library xyz and some don't. That

And some will have different openssl versions (e.g. 1.0 vs 1.1) and thus
different shlib majors, and thus any .so in the python app that is
linked to /usr/lib/libssl.so.N is going to be trouble. I realize there
is some attempt to not have this happen, but it seems impossible to
avoid completely.

> would mean the rpm works in some os+version combinations, but not all.
> Confusing to the user when a rpm refuses to install and the os gives the
> user a cryptic not-helpful message.
>
> Same with dpkg installations. We see differences between ubuntu and debian
> variants, versions of each os variant, and other dpkg-compatible operating
> systems (Mint etc.)

But "dpkg-compatible" doesn't really mean "the systems are using the
same packaging system". By packaging system, I mean the entire set of
packages that are available on a particular OS/release/cpu typle (where
"debian" and "ubuntu" are different OSs).

So the notion that a binary package in rpm, deb, whatever can be
installed on any linux flavor that has the same CPU type is basically
unsound; it would have to be built per OS/release/cpu tuple and ensure
that the dependencies are available in the same package repo that it's
going to go into. (or that, with sound optimizations.) This is your
option 3 I think, and it's what pkgsrc does.

[Vince Skahan]

On Sunday, May 10, 2020 at 4:30:46 PM UTC-7, Greg Troxel wrote:

So the notion that a binary package in rpm, deb, whatever can be
installed on any linux flavor that has the same CPU type is basically
unsound; it would have to be built per OS/release/cpu tuple and ensure
that the dependencies are available in the same package repo that it's
going to go into.

There is a limit in cost/benefit for supporting every oddball Linux variant on the planet.

I know there's a lot of demand for python3, but do we have any v4 examples yet where the python2 packages didn't work for the edge cases like Mint ?

Do we have any examples where setup.py didn't work as the ultimate fallback for odd systems ?

[Graham Eddy]

macos? fits the OS/release/cpu tuple.

g-eddy

[Till Maas]

Hi,

On Sun, May 10, 2020 at 01:27:01PM -0700, Vince Skahan wrote:

> Option-2 is how big apps like splunk and puppet do it. They bundle the
> interpreter+libs they need under their tree. If you think about it, it's
> also how RHEL chose to work the python issue for RHEL-8. They don't use
> the /usr/bin/python3 stuff. They have their own internally-used-only path
> with the things 'they' need.

Platform Python in RHEL still uses the system Python libraries. The
problem it addresses is that the operating system has a different
lifetime than Python 3.6 and the platform Python does not provide any
guarantees to users but allows to be flexible for the system tools that
use Python.

> Option-3 is how you'd do it in a CI/CD/autotest world. The Remi repos are
> a great example of this. It's certainly technically possible to build an
> infinite number of os+version package variants via Docker and Github hooks
> if we wanted to go that path.

If weewx becomes a standard Python package (pip-installable), then it
is rather easy to package it for different distributions. Using services
such as https://copr.fedorainfracloud.org/ or
https://en.opensuse.org/openSUSE:Build_Service_supported_build_targets
helps with building for different distributions.

And as a fallback, there is still the pip installation method or there
could be some generic rpm/dpkg packages if necessary for some specific
distributions.

> The downside of option-2 is what would we do if a security vulnerability

Another downside, which is what led me to start the discussion, is that
this makes weewx different from other Python projects which makes my
generic Python/Linux experience less useful because everything is
different. So somehow it seems that this discussion drifted to solve a
different problem, not sure what it is exaclty.

Thanks
Till

[Tom Keffer]

I'm with you, Till, but I still need a volunteer! 😀

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/20200512182837.GE25070%40genius.invalid.

[Greg Troxel]

Till Maas <opens...@till.name> writes:

>> The downside of option-2 is what would we do if a security vulnerability
>
> Another downside, which is what led me to start the discussion, is that
> this makes weewx different from other Python projects which makes my
> generic Python/Linux experience less useful because everything is
> different. So somehow it seems that this discussion drifted to solve a
> different problem, not sure what it is exaclty.

Packaging systems basically view a package bundling dependencies as a
bug and patch it out. That way -- the normal dependency way -- a bug in
a dependency can just be patched and that dependency updated. With
bundled ones, you have to find them all and patch them all. Really
weewx would then be obligated to make a very fast point release every
time this happened. I think it's bad practice to bundle dependencies,
and the perceived need to do so is a clue that there is a problem.

[Vince Skahan]

On Tuesday, May 12, 2020 at 11:30:42 AM UTC-7, Tom Keffer wrote:

I'm with you, Till, but I still need a volunteer! 😀

Sorry if I diverted the discussion any.

I was trying to understand what problem(s) we're trying to solve, so I can assess if I can help or not.

• If the goal is to make the /home/weewx tree less unique, that's one thing.
• If the goal is to package in pip and make setup.py go away, that's a different thing.
  
• Is there anything else ?

I think we're proven that you can't do 'one package to rule them all' given the os variations we've seen in pil-vs-pillow and python2-vs-python3 for all the ancillary modules folks commonly want. That means we'd need a list of which os+version combinations you want to be able to package (dpkg/rpm/pip) for, and having appropriate repos etc. and builds for every single combination.

So I guess I'm at a bit of a loss regarding who needs help with what.

[Tom Keffer]

I don't blame you.

To keep the discussion focused, and a little more formal, I've started issue #552. Please move the discussion there. 

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/7ade75f4-e9c1-48a3-b4f4-82680aa1204f%40googlegroups.com.

[Till Maas]

On Sun, May 10, 2020 at 10:19:41AM -0700, mwall wrote:
>
>
> On Sunday, May 10, 2020 at 11:37:17 AM UTC-4, Till Maas wrote:
> >
> >
> > > 4. weewx.conf gets automatically patched by an upgrade. This is a big
> > > feature of weewx over the standard package upgrades, which merely ask
> > a
> > > user if they want to "merge" their old config file with an incoming,
> > new
> > > config file.
> >
> > What exactly is happening here?
> >
>
> the weewx configuration file (nominally weewx.conf) is modified during the
> upgrade. if you use setup.py and install a new release of weewx on top of
> an existing one, it will modify the existing weewx.conf so it will work
> with the latest release. it saves the old file, of course. this works on
> weewx config files all the way back to 1.0

Did you consider upgrading the config when running weewx the first time
after upgrade instead of during the upgrade process? Could this be an
alternative? This could still be enabled/disabled in the config file.

Another possibility to simplify things might be to have a defaults,
read-only config file and have the user provide a config file that
contains only the changes. This does not solve renaming config options,
though.

> a long long time ago, the configuration modification code was embedded in
> setup.py. it has since been moved to a module, and it can be called
> independently of setup.py using wee_config.

Awesome, this simplifies this already. :-)

> this is tricky. six and daemon are just one file each, and they basically
> never change. imho, it does not make sense to create a dependency for each
> of them - just distribute them as part of weewx, and do updates as
> necessary.

If they are going to be bundled, they still need to be moved in a custom
weewx namespace to ensure that they do not collide with the system
package. Therefore bundling seems to add more costs than adding a
dependency. At least six is a very common package, not sure about
daemon.

> > The user directory is the directory where wee_extension adds new
> > extensions, isn't it?
> >
> > On first thought, the best approach for me would be that the recommended
> > way to install extensions would be that they use pip as well for
> > installation. Everything that wee_extensions installs, would be best
> > moved to /var/lib/weewx/user or similar to ensure that it does not
> > conflict with the packaged files from weewx. Seems like it would be
> > $HOME/.local/share or $XDG_DATA_HOME if installing per-user. There does
> > not seem to be a better counter part for /var/lib.
> >
>
> agree that /var/lib is the appropriate place for this kind of thing. right
> now /var/lib/weewx is where the sqlite databases reside. adding a
> /var/lib/weewx/user directory would not be unreasonable, but it would
> require more python path munging.

The user directory would need to be in $HOME, otherwise there would be
permission challenges.

Thanks
Till

[Tom Keffer]

Discussion (and answer) moved to issue #552.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/20200522165330.GG25070%40genius.invalid.