weewx and packaging systems

[Greg Troxel]

(I've been running weewx for about 1.5 years under NetBSD on i386 and
then RPI3, using pkgsrc to provide the dependencies.)

I created a pkgsrc package for weewx in pkgsrc-wip, and there are some
awkward things about it. Part of the reason was to capture the
dependencies to avoid having to manually manage that I need Pillow, etc.
Part of it is that I think software should in general be packaged and
that almost all uses of 99% of software is via packaging systems. I
realize that the maintainers of many pieces of software think their
software is special (all software is!) and that people should do things
by hand and pay close attention. I can of course see the point,
especially for weewx, but in general this does not scale.

The main difficulty I encountered is the handling of weewx.conf; the
install is interactive and there is a notion of merging updated config
files. From a packaging viewpoint, it would be really nice to be able
to install weewx (to an empty place) non-interactively at package
creation time, register the config file as an example, and then somehow
be able to separately invoke the config file merging when the package is
installed. I have sort of dealt with this, but not entirely.

The escond issue is that weewx does not follow the usual conventions for
installation of executable programs, python files used by programs,
config files, etc. I have dealt with this by installing to
/usr/pkg/weewx (as weewx_home), as it doesn't seem useful for me to try
to rearrange things.

From my experience, I am guessing that essentially no one is using weewx
from packaging, and that probably even in Debian, Ubuntu, FC, etc. there
are not packages. I am curious if there are packages, and if anyone who
worked on those packages has words of wisdom.

I also wonder what people think of splitting the process of installing
the default config file (I put it in weewx.conf.DIST, intending that to
be owned by the packaging system and never changed by the user) from the
process of creating the user's config file and merging that config file.
This update could then be hooked into the package installation process.
That should allow people to install weewx from packaging instead of
manually from a git checkout or tarball, and upgrade, while still
having the same semantics.

Greg

[Vince Skahan]

Huh ?

Packages have been available for debian(ish) and redhat(ish) operating systems for a number of years.  Those packages install the prerequisite/corequisite packages that are needed to make weewx work and install things where those operating systems install things.  What exactly is missing there currently ?

I'd suggest almost all of the (newbie) pi users are using the packaged version these days, based on the questions we see.

Upgrades have always just plain worked in my experience, for both debian(ish) and redhat/fedora systems, regardless of whether you were packaged or setup.py for your installation method.....as long as you stick with the method of installing weewx you picked.

The structure of weewx.conf is a bit difficult to script around due to its configobj origins, but lots of work has been done to make the items therein unique enough to let you run sed inline (or use ansible, etc.) against that file to salt to taste.   Most people shouldn't have to do anything for a vanilla weewx system other than install the package interactively and answer the prompts it provides.

The extension installer also helps a lot, permitting you to add/delete skins+extensions with one command.   So if you have well-formed skins and extensions you might never need to even edit weewx.conf - but of course some require a 'lot' of touch labor to tweak to your liking.  That's the skins/extensions bug/feature, not the core weewx product's limitations.

FWIW, I can build my whole system totally scripted using either setup.py 'or' a packaged weewx by installing weewx and running sed inline on the weewx.conf file, then resetting the daemon.  That's not bad.

(ok - I don't do it 100% scripted actually, I would have a few blocks of configs to add to some extensions needing a lot of hand-tweaking to look+act the way I want them to do, but again that's an issue with those extensions.  I can certainly go from nothing-to-done totally scripted for 98% of my setup)

While I'd also like to see a more modular weewx.conf structure, that's come up a lot in the past and the decision was made to stay the course as-is there.  But sure I'd agree that moving to a less python configobj kind of structure to something more mainstream even if it's a ini file would be cool to see, but there are a zillion plus/minus things there as well.  It was an interesting conversation way back when....

[mwall]

hi greg,

packages are typically downstream from the source.  of course, if we are doing things in the source that make packaging difficult, we'll try to address that.

we build debian, redhat, and suse packages for each weewx release.  the stuff required to build each of these is located in the pkg directory in the weewx distribution.  there are targets in the makefile that automate the process.  we have been doing this for a long time now, and the download stats show a significant number of package downloads compared to tarball downloads (sometimes the debian package has more downloads, other months the tarball has more - monthly downloads have totalled 3000-5000 for a few years now)

the debian and rpm packages use the wee_xxx utilities to help with the installation process.  that results in consistent configuration file no matter how weewx was installed.  none of them invoke setup.py to do the installation.

the locations of executables, skins, etc varies by platform - that is to be expected.

if you want to create a package for netbsd (or freebsd or openbsd), then you probably want to follow the pattern used by the deb/rpm packages.  they simply copy files from the source tarball into the appropriate places.  then they use wee_config to generate the initial weewx configuration file.  if you prefer, do not do that, and require that the user run wee_config directly.  it all depends on how much you want the installation to "just work" after install.

for example, the debian package prompts for station type and other parameters, since debian package management provides mechanisms to do that.  the rpm packages, on the other hand, do not.  both platforms result in a 'dist' version of the weewx.conf file as well as all the permutations when a user upgrades and modifies the configuration.

the packages also handle dependencies

we have considered splitting weewx.conf into separate files (for example, using the xxx.d pattern), but having *all* of the configuration in a single file has always won.

we have been very careful to keep the parts isolated - docs are in a single self-contained directory, skins are all in a single directory (but users can put them anywhere, if they want), drivers and other code extensions go in the user directory, reports end up in a separate directory tree, and the entire configuration is in a single config file.

we maintain our own debian apt repository - its too difficult to coordinate weewx releases with debian/ubuntu/mint/whatever releases, and we ensure that the weewx apt repository works with *any* version of debian or its derivatives.  we could do the same for redhat/centos and suse too, i suppose, but i have not had time to set that up.  we have the apt repository fully automated, otherwise there is no way we could manage it.

we made an explicit decision to make a 'python' install of weewx a bit non-standard, at least from a python point of view.  we don't want weewx assets to be dumped into the tree where other python code lives.  so a pip install is not really appropriate for weewx.  we have talked about using pypi, but the all-in-a-single-directory pattern for standalone weewx install does not really fit with pypi.

if you want to see a weewx package for netbsd (or maybe something that works for all of the bsd?), consider creating a document that explains how someone would create a weewx package for netbsd.  write up the steps, and enumerate the assets and their locations after an install.

m

[Greg Troxel]

Vince Skahan <vince...@gmail.com> writes:

> Huh ?
>
> Packages have been available for debian(ish) and redhat(ish) operating
> systems for a number of years. Those packages install the
> prerequisite/corequisite packages that are needed to make weewx work and
> install things where those operating systems install things. What exactly
> is missing there currently ?
>
> I'd suggest almost all of the (newbie) pi users are using the packaged
> version these days, based on the questions we see.

Thanks for taking the time to write back to me.

I was guessing this wasn't the case, because the weewx setup.py install
seemed like it would be hard to make a package out of. I now
understand that the Linux packages don't use setup.py to install, which
clears up a lot.

> Upgrades have always just plain worked in my experience, for both
> debian(ish) and redhat/fedora systems, regardless of whether you were
> packaged or setup.py for your installation method.....as long as you stick
> with the method of installing weewx you picked.

So pretty obviously the Debian/RH packages are splitting the config
updating from the "put the scripts in the destdir" script -- which was
clear to me needed to be done, but it felt like fighting city hall since
that's what setup.py install does. Now that I get that setup.py isn't
used, my fog of dissonance is lifted.

> The structure of weewx.conf is a bit difficult to script around due to its
> configobj origins, but lots of work has been done to make the items therein
> unique enough to let you run sed inline (or use ansible, etc.) against that
> file to salt to taste. Most people shouldn't have to do anything for a
> vanilla weewx system other than install the package interactively and
> answer the prompts it provides.

Agreed that the config setup/merge code works well - it has for me for
manual installs.

> The extension installer also helps a lot, permitting you to add/delete
> skins+extensions with one command. So if you have well-formed skins and
> extensions you might never need to even edit weewx.conf - but of course
> some require a 'lot' of touch labor to tweak to your liking. That's the
> skins/extensions bug/feature, not the core weewx product's limitations.

I am guessing then that these are not dealt with by the packaging
system, even though they feel to me like they are packages that depend
on weewx. Is that right?

> FWIW, I can build my whole system totally scripted using either setup.py
> 'or' a packaged weewx by installing weewx and running sed inline on the
> weewx.conf file, then resetting the daemon. That's not bad.

setup.py is interactive, so I'm guessing you are passing the flag to
avoid that and then editing weewx.conf from the release version to your
local values.

> While I'd also like to see a more modular weewx.conf structure, that's come
> up a lot in the past and the decision was made to stay the course as-is
> there. But sure I'd agree that moving to a less python configobj kind of
> structure to something more mainstream even if it's a ini file would be
> cool to see, but there are a zillion plus/minus things there as well. It
> was an interesting conversation way back when....

I can see the point, but I was not trying to address the config file
format. It's the conflation of "install" and "configure" that I was
really getting at.

[Greg Troxel]

mwall <mw...@users.sourceforge.net> writes:

> packages are typically downstream from the source. of course, if we are
> doing things in the source that make packaging difficult, we'll try to
> address that.

Thanks to you also for taking the time to write back to me. Between your
and Vince's I feel like I understand the big picture now (which if I did
earlier I wouldn't have written what I did).

> the debian and rpm packages use the wee_xxx utilities to help with the
> installation process. that results in consistent configuration file no
> matter how weewx was installed. none of them invoke setup.py to do the
> installation.

This is the part that I didn't expect. You mentioned about things that
make packaging difficult, so I'll try to explain where I'm coming from.

From a packaging viewpoint (coming from lots of other things, not about
weewx), I expect the normal install target to put files in
${DESTDIR}${PREFIX}, and to follow the conventional layouts, so that
when those bits are basically tarred up and then unpacked in the user's
filesystem, they are as expected.

> the locations of executables, skins, etc varies by platform - that is to be
> expected.

Agreed. The canonical example is autoconf that has arguments to set the
various directories. Usually I find that where stuff in /var is put
varies, and /usr/pkg/man/man1 vs /usr/pkg/share/man/man1 are the big
differences.

> if you want to create a package for netbsd (or freebsd or openbsd), then
> you probably want to follow the pattern used by the deb/rpm packages. they
> simply copy files from the source tarball into the appropriate places.
> then they use wee_config to generate the initial weewx configuration file.
> if you prefer, do not do that, and require that the user run wee_config
> directly. it all depends on how much you want the installation to "just
> work" after install.

We have a notion that installing a package is not interactive, so I'd
tend to a README.pkgsrc in the installed docs dir to tell people to run
wee_config.

> for example, the debian package prompts for station type and other
> parameters, since debian package management provides mechanisms to do
> that. the rpm packages, on the other hand, do not. both platforms result
> in a 'dist' version of the weewx.conf file as well as all the permutations
> when a user upgrades and modifies the configuration.

I was thinking the updater/merger could be hooked into install.

> the packages also handle dependencies

yes, that was actually my main goal, to be able to have a package even
if I run a manual install, just to not keep track of the N python modules.

> we have considered splitting weewx.conf into separate files (for example,
> using the xxx.d pattern), but having *all* of the configuration in a single
> file has always won.

I did not mean to raise that as an issue.

> we have been very careful to keep the parts isolated - docs are in a single
> self-contained directory, skins are all in a single directory (but users
> can put them anywhere, if they want), drivers and other code extensions go
> in the user directory, reports end up in a separate directory tree, and the
> entire configuration is in a single config file.

I can see how that was. It was the setup.py install layout not being
aligned with my perception of Linux FHS (which is hier(7) in BSD) that
confused me.

> we made an explicit decision to make a 'python' install of weewx a bit
> non-standard, at least from a python point of view. we don't want weewx
> assets to be dumped into the tree where other python code lives. so a pip
> install is not really appropriate for weewx. we have talked about using
> pypi, but the all-in-a-single-directory pattern for standalone weewx
> install does not really fit with pypi.

Thanks for clarifying - I didn't realize that having setup.py not do the
normal layout was that intentional.

> if you want to see a weewx package for netbsd (or maybe something that
> works for all of the bsd?), consider creating a document that explains how
> someone would create a weewx package for netbsd. write up the steps, and
> enumerate the assets and their locations after an install.

I already made one. Understanding hte grand plan, it has some deficiencies

it puts the setup.py install layout into ${PREFIX}/weewx

it doesn't (yet) have any auto-merging code; the new weewx.conf is in
weewx.conf.DIST and the user's config is initialized from that and
then not modified.

the skins are treated as code and thus just removed/updated rather
than merged; I know people edit these a lot, and reasonably so

I wonder if setup.py build could have a "standalone mode" like it does
now, and a "package mode" that follows the standard layout. But that
may be a lot of churn for minor common subexpression elimination.

I will study the debian/rh files before I ask any more questions!

[Vince Skahan]

Greg -  packages are just bundled up files with (optional) scripts that can run pre/post install/uninstall as needed.  For example - a rpm is a gzipped cpio archive, if that helps any.  It also supports (optional) pre/post-install scriptlets.  You can see them if you do "rpm -qp filename.rpm --scripts" to see them.  Similarly "rpm -qp filename.rpm -l" to list the files in a downloaded rpm.  Or you can extract the cpio archive via "rpm2cpio filename.rpm | cpio -imdv"  if I recall the switches correctly from memory, if you want to explode the rpm out into subdirectories in your current working directory and see the trees in a tmp directory.

Skins historically were more manual, before there was an extension installer. The extension installer can drop a multitude of formatted bundles of stuff (tar, zip, etc.) or even directories of stuff into place in one command.  Super-flexible.   It's up to the extension to provide the smarts re: what goes where in the filesystem when it's installed.  FWIW - I install all my skins+extensions via the extension installer, so it's totally non-interactive other than saying 'install skin xyz or extension abc'.

The debian packaging format lets you pre-answer (preseed) things with defined values to make things non-interactive.   The rpm format is a little harder to do this with, typically you'll see a first-run thing that tells you to run a command to initialize your rpm to salt to taste (typically you'll see this in database rpms).

But short answer is the packages do not use setup.py - they just explode archives of files onto disk and do the (hopefully) right thing to hook'em into the startup sequence for the box.  Each os type does it differently, but it's notionally all the same.....

Regardless, packages are not autoconfig/metaconfig circa 1988 type installation mechanisms.   They try to make that mess go away.  You build the packages (possibly using that stuff) but the end result of the packages is an already compiled set of stuff.   Easiest way to see what it's doing is to grab the source rpm for something like python and look at the .spec file in there.  That has the compilation stuff in there, so it's done at package 'build' time, not package 'installation' time.

If you're building a weewx package for a new os that has no package already, I'd suggest you follow the filesystem standard that os follows, whatever it is.  If it's Linux FHS compliant, that should be pretty straightforward probably.

[Greg Troxel]

Thanks.

I do understand how packages work; it was just that I have not run into
a thing like weewx that has a functioning build/install system, but
where one isn't supposed to use it when building a package. In all the
rest of the things I've dealt with (probably over 100), the package
config files do "configure/make" at build time (or equivalent), and then
"make install" into a destdir, and then tar that up. I have now been
enlightened that weewx's approach is to have packagers hand-write a
bunch of install statments in the packaging config files instead, and
that "setup.py install" is set up for non-package installs with a
different layout (that fits for easy removal, into a distinct prefix).
I see the rationale - it's just something new to me.

Interesting about debian and pre-answered questions; that seems very
helpful when you are scripting things and have already debugged how you
want it. It's easy enough to document that the user should run
wee_config, so I'm not going to worry about facilities like that.

I also understand now the non-packaged extension approach. Some
packaging systems (including pkgsrc) have a notion that files shouldn't
get written in the package-system-controlled prefix, other than by the
package manager. But I realize that's far from universal and don't mean
to rail against it - at this point I am just trying to understand, and I
think I do ow.

[Vince Skahan]

On Monday, August 5, 2019 at 4:34:07 PM UTC-7, Greg Troxel wrote:

...In all the

rest of the things I've dealt with (probably over 100), the package
config files do "configure/make" at build time (or equivalent), and then
"make install" into a destdir, and then tar that up.  

Yeah, that's "rpmbuild" and the debian equivalent do.

There's just a little more magic in there where they can do stuff pre/post install/uninstall as well.

Sure better than the oooooooooold days when you had to build everything the hard way.

 

[Leon Shaner]

Let us old fartes tell ya 'bout the oooooooooold days and hard ways.  LOL

;-)

Regards,

\Leon

--

Leon Shaner :: Dearborn, Michigan (iPad)

On Aug 5, 2019, at 7:38 PM, Vince Skahan <vince...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/517155b0-f566-479d-bbbb-164a1510d5f7%40googlegroups.com.