config file quoting and whitespace changes during upgrade

[Greg Troxel]

(I have been running WeeWX since the very end of 2017, and updated a few
times. I'm building with setup.py on NetBSD, but I'm quite used to
python.)

There is some config file auto update/merge stuff. My usual practice
is to keep my config file matching the default example in the source
code, except for things that I intend to change, and after an update (of
almost anything, not just WeeWX) I try to diff and reduce unintended
diffs. And yes, I think this is normal :-)

I have noticed that after doing a WeeWX upgrade, which tries to do
something that feels like a 3-way merge from old to new configs with
mine, and almost entirely gets it right, that the config file differs
from the example in two ways:

- a bunch of whitespace changes, not important but annoying
- quoted strings are no longer quoted

Because of this I have tended to update less often than I might.

A few minutes ago I glanced at configobj and saw the magic words "round
tripper" at https://pypi.org/project/configobj/

ConfigObj is a simple but powerful config file reader and writer: an
ini file round tripper.


So I wonder if the changes are something configobj views as
normalization, and if instead

- It should be configured with WeeWX'S whitespace and qouting rules.
- WeeWX should round-trip the default files

or if

- I'm confused.
- It's an artifact of my system (but I have configobj 5.0.6 which is current).

Do other people see this?

[Joel Bion]

I’ve also noticed this.

Sent from my iPhone

> On Nov 6, 2022, at 4:55 AM, Greg Troxel <g...@lexort.com> wrote:
>
> 

> --
> You received this message because you are subscribed to the Google Groups "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/rmitu3cw8fm.fsf%40s1.lexort.com.

[Tom Keffer]

Yes, all the time.

Semantically, 'configobj' does a round trip, but textually, it leaves much to desire. 

Going forward, the intention is to start treating weewx.conf as "readonly". The installer will not try to "update" it, not even to change the version number. Any changes would be up to you, perhaps with the aid of a tool (although, if you do use a tool, you can expect the same textual changes).

The setup.py method of install is being deprecated, especially the imperative approach that WeeWX takes where many install methods are overwritten. Instead, the configuration will be specified declaratively using a pyproject.toml file. That necessarily requires that weewx.conf be treated as readonly. The actual install would be done by using pip.

-tk

[Vince Skahan]

This one scares me, as previous versions have definitely tweaked the contents along the way of the upgrade path from version to version.   Is this something for a future 5.x major version that might have breaking changes like dropping python2 support or the like ?

Has there been any thought to dumping configobj format completely and going to a more modern mainstream format (JSON, YAML, etc.) or alternately more of a unixy setup ala something like /etc/weewx/config.d/<whatever> type files that can be simply included much like webserver configs do it ?

[Joel Bion]

If there’s a new format, I’d much prefer TOML vs. JSON/YAML. TOML makes some improvements over ini files, without being an easy-for-humans-to-make-mistakes format like JSON.

But it sounds like the issue is more fundamental- that the issue has something to do with the new Python package installer not wanting to edit existing things like the .ini file at all, no matter what syntax it has? I really have no idea here as I have yet to begin to spend time learning the newer Python package install mechanisms. 

Sent from my iPhone

On Nov 6, 2022, at 11:03 AM, Vince Skahan <vince...@gmail.com> wrote:

This one scares me, as previous versions have definitely tweaked the contents along the way of the upgrade path from version to version.   Is this something for a future 5.x major version that might have breaking changes like dropping python2 support or the like ?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/7e89e9d5-81c2-47d0-b251-03a1a0e176cfn%40googlegroups.com.

[Tom Keffer]

Yes, this would be for V5. Most likely Python 2.7 would be dropped. 

The problem is not the configobj format, it's the future readonly nature of Python installs. The config file (and skins) are considered "data", and the only way to include "data'' is within a Python package. That package might be in a zipfile or Python egg, hence its readonly nature.

Furthermore, installs can only be done within "sys.prefix". You can't just roam around the file system and install things wherever you like. Hence, a more unixy setup could not be done with pip, even if we wanted to.

While I was at first put off by this decision, I can see their point. For one thing, it means sudo is no longer necessary. You can also install into a virtual environment as easily as into the system's python.

Regarding the format, I suppose we could move to a different format besides configobj, but it has served us well --- I don't see a compelling reason to change. For one thing, changing would break a lot of things.

See this thread. It's hard to follow without spending some time in Python packaging land, but it discusses these issues. 

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/7e89e9d5-81c2-47d0-b251-03a1a0e176cfn%40googlegroups.com.

[Vince Skahan]

Wow if you drill down through that thread it gets crazy confusing....but (this one) does help a little although (this other one) got me confused again.  Seems like untrodden ground.

That whole forum tagged with Packaging there can sure make your head spin !

On Sunday, November 6, 2022 at 11:55:38 AM UTC-8 Tom Keffer wrote:

installs can only be done within "sys.prefix". You can't just roam around the file system and install things wherever you like. Hence, a more unixy setup could not be done with pip, even if we wanted to.

Got it - but we're not installing a python 'module' here.  We're installing a pure python 'application' that is massively configurable by the user.  How do you do both ?

Would it be a multi-step installation ala:

• install unconfigured weewx as a library under /usr/lib/python3/dist-packages or the like (raspi example)
• and then have a utility therein that writes a customized config to /home/weewx like it does now ?
• and how would skins, services, extensions, etc. be installed/uninstalled/enabled/disabled/edited ?

Guess I'm lost here for how we'd mechanically do a future installation updating a current very tweaked setup.py installation....or even how would I build one up from scratch.

[Tom Keffer]

Doug: Yes: you'd either install using pip, or a package installer (apt, yum, etc.). I'd sure like to get out of the business of custom installs and, instead, rely on Python tools and all their goodness. 

Vince: The thinking is that the config file, skins, and database would all be initialized on first use, using the supplied readonly data as templates. The problem is that it's hard to get weewxd going without doing some customization, albeit via a series of prompts the way setup.py works now.

Alternatively, after the pip install, but before first use, the user would be expected to "initialize" or "customize" the install via a tool, which is when these resources would be copied into place. 

The advantage of either of these approaches is that it mirrors what the package installers do now, so it eliminates a lot of custom code.

The last possibility, albeit not the supported approach, is to do what I do most of the time: run right out of the git repository.

I hear you on the complexity of Python packaging. I've been using node for a few years now and they have done a much better job. Unfortunately, the Python gods have decided to create a backend standard (PEP 517), and then rely on a 3rd party ecosystem to do the actual packaging. 

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/c128da8b-3837-4feb-be89-04a8d3dbf35cn%40googlegroups.com.

[Joel Bion]

I've been wondering about the big changeover that will occur with Python 3.12 (the removal of distutils). I use three packages that are Python applications that need to configure things at a system level: fail2ban, ufw, weewx.

On the ufw git site, this page has a rather interesting conversation going. In it, they state that certain packages may want to move to non-Python build system (such as meson)... The page with this discussion is: https://bugs.launchpad.net/ufw/+bug/1962732

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEAKrYy72i4n%3DzvcCxjJ6St5MvSaaZYtqTVvpwdiTCHPug%40mail.gmail.com.

[Joel Bion]

..there's also this interesting page: https://whynothugo.nl/journal/2022/07/26/meson-for-python-applications/

[Tom Keffer]

Joel, thanks for that link. Meson looks interesting. I'll check it out.

[Vince Skahan]

On Sunday, November 6, 2022 at 2:07:07 PM UTC-8 Tom Keffer wrote:

The thinking is that the config file, skins, and database would all be initialized on first use, using the supplied readonly data as templates. The problem is that it's hard to get weewxd going without doing some customization, albeit via a series of prompts the way setup.py works now.

This goes down the wview path although Mark had a scary from a security standpoint perhaps admin gui for this.   Dunno if his approach for saving config info in a db might be worth thinking about or not but maybe....

 

Alternatively, after the pip install, but before first use, the user would be expected to "initialize" or "customize" the install via a tool, which is when these resources would be copied into place. 

I guess I lean toward installing a default that runs enough to throw a syslog error saying 'configure me' but without hacking up the db with simulated values...then having a utility therein that can do certain things (enable/disable/add/delete/tweak).

Unless you go with a config db it still seems like weewx.conf is seemingly going to have to be writable unless you componetize it ala additive .d files like profile.d or apache/nginx config files.  That means sudo.  Ugh.

Problem I've always had with weewx.conf is that it's not easy to script editing it as there is no guaranteed uniqueness.  You can't just edit "password = MYPASS" if every [[something_block]] can have a string with that keyword.  I know the current configobj can't effectively be edited inline with ansible for certain, given the current .conf structure.

Again this to me looks like a problem JSON can definitely easily solve.  I don't know anything about TOML at all so I can't speculate there.

Think about the config.db approach.  Have it supersede weewx.conf values.

Or is that crazy talk ????

 

[Tom Keffer]

One of the things I disliked about wview was its many configuration files and then, later, its database-oriented configuration. I wanted everything in one, easily readable, place.

To do edits of the config file it's best to read it in using configobj, do the edits in Python, then write it back out. Unless I'm missing something, I don't see why JSON would be any different.

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/91bd209c-01cf-4c91-b726-d5f9461fa8fbn%40googlegroups.com.

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> The problem is not the configobj format, it's the future readonly nature of
> Python installs. The config file (and skins) are considered "data", and the
> only way to include "data'' is within a Python package. That package might
> be in a zipfile or Python egg, hence its readonly nature.

I'm writing from the pkgsrc viewpoint, which is really a set of norms
adapted from coping with 10s of thousands of packages. (I think the
Debian situation is pretty similar, but I'm not entirely clear on the
details. Of course prefix = /usr in that case, which is normal in
GNU/Linux and not ok in BSD, where that's the base system only.)

packages have executables, libraries, and included data files, and all
of these are basically unchanging. If you wish them to be different,
you make a commit to the source repo for the package, and then make a
distfile and use that in a pkgsrc package. Or you apply at patch at
build time, but the point is you change the sources. Executables go
in $prefix/bin, normal libraries in $prefix/lib, and python modules in
$prefix/lib/python3.N/site-packages, and text files in $prefix/share.

packages have config files, and they go in $prefix/etc. But because
of packaging, they don't -- they are installed in someplace
$prefix/share/examples/$PKGNAME/$pkgname.conf and treated as read-only
there. They are declared as config files so a fresh install puts a
copy in $prefix/etc or $prefix/etc/$pkgname. On uninstall, if it
matches still, it's removed. If not, it's left. An install when the
file already exists doesn't change it.

packages when they run will store data, and that goes in
$prefix/var/$pkgname, or /var/$pkgname, depending on how the packaging
system (globally) is configured.

So I think regularizing weewx into this is a good step.

> Furthermore, installs can only be done within "sys.prefix

> <https://docs.python.org/3/library/sys.html#sys.prefix>". You can't just

> roam around the file system and install things wherever you like. Hence,
> a more unixy setup could not be done with pip, even if we wanted to.

I am not sure what you mean; pip can install executables to $prefix/bin,
and those executables can read config files in $prefix/etc. Staying
within --prefix seems unixy enough, but maybe you mean something even
more agile.

> While I was at first put off by this decision, I can see their point. For
> one thing, it means sudo is no longer necessary. You can also install into
> a virtual environment as easily as into the system's python.

And, you can meet packaging system norms which require staying within
the packaging system prefix.

I'll also note that you can use something like autoconf and have that
run setup.py or whatever as a sub-step, for a program that is not pure
python. Probably that does not make sense here.

And, my usual plea: there is a fairly small set of standard build
systems. Packaging systems (and individual humans) have to learn to
deal with each of them. It's best to do something a lot of other people
are doing.

> Regarding the format, I suppose we could move to a different format besides
> configobj, but it has served us well --- I don't see a compelling reason to

> change. For one thing, changing would break a *lot* of things.

I think it would be good to clean up the round-tripping issue, even if
weewx never writes. If we use configobj, then all files intended to be
read by it should be in its canonical form.

> See this thread <https://github.com/pypa/setuptools/discussions/2648>. It's

> hard to follow without spending some time in Python packaging land, but it
> discusses these issues.

Thanks; that helps me understand a little.

Regarding setup/wizards, I don't think there is anything wrong with
having a program that you run and asks questions or whatever and then
outputs a new weewx.conf in the current directory. The user can then
put it in $prefix/etc.

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> I hear you on the complexity of Python packaging. I've been using node for
> a few years now and they have done a much better job. Unfortunately, the
> Python gods have decided to create a backend standard (PEP 517

> <https://peps.python.org/pep-0517/>), and then rely on a 3rd party

> ecosystem to do the actual packaging.

Having worked on packaging, I think that's exactly right. Things like
node want the user to have to use their tool, and then there are 10 of
them, vs using a helper to build a native package that can then use used
uniformly.

[Vince Skahan]

On Sunday, November 6, 2022 at 3:39:54 PM UTC-8 Tom Keffer wrote:

One of the things I disliked about wview was its many configuration files and then, later, its database-oriented configuration. I wanted everything in one, easily readable, place.

To do edits of the config file it's best to read it in using configobj, do the edits in Python, then write it back out. Unless I'm missing something, I don't see why JSON would be any different.

Yes, I remember that.  I didn't mind the db, but I sure didn't want to need to use the gui to tweak the defaults.

What I'm looking for is a way to 100% totally script a very customized installation.  That is not possible using configobj as far as I've been able to tell in several years of trying to get ansible to do so, although I suspect some crazy complicated python might get me there.  

I *think* it's possible in ansible if it was in JSON if that matters any.  Really comes down to how abstract and flexible the scripting language tools are. Nobody writes tools around configobj is the basic problem.

I would like to be able to:

• add lines
• remove lines
  
• edit lines
• comment/uncomment lines
  
• add/remove/edit comments at the end of lines as documentation for myself
• 'hopefully' without writing complicated python code

The whole purpose of DevOps tools is to not need to write complicated python/perl/whatever code.

My weewx.conf is over 400 lines if I remove all commented and whitespace-only lines.  I'd be happy to pass an obfuscated one along if it'll help show what I'd love to be able to fully script.

But yes, ultimately it's a hierarchy of stuff there.  I would certainly agree that once you reach the point of writing a lot of custom code to parse/manipulate the file, it doesn't matter if it's configobj or JSON or whatever because you're writing a lot of custom complicated code no matter what...

[Vince Skahan]

On Sunday, November 6, 2022 at 3:52:07 PM UTC-8 Greg Troxel wrote:

Regarding setup/wizards, I don't think there is anything wrong with
having a program that you run and asks questions or whatever and then
outputs a new weewx.conf in the current directory. The user can then
put it in $prefix/etc.

As long as it can be 100% fully hands-off scripted, sure.

The way debian packages do it with DEBIAN_FRONTEND to me is both too painful and not nearly flexible enough to deal with the many (infinite) levels deep that you can go in a weewx.conf file.

[Joel Bion]

Everything I’ve wanted to do in weeex.conf and skin.conf could only be done via a text editor. The current ini format (or toml format) is much easier to edit than json or yaml.

For example: I’ve been writing an Alexa skill to use data that Weewx writes out - among other things - to announce weather related data and forecasts. Alexa uses JSON for many things. It’s not a more difficult a way to present the data but it’s full of little gotchas that are annoying. 

Sent from my iPhone

On Nov 6, 2022, at 5:15 PM, Vince Skahan <vince...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/b3ed6e89-cfd5-4c4d-9338-c8e42f16a697n%40googlegroups.com.

[Chris Richmond]

Maybe I've missed the whole point of these proposed changes, but I'll offer my experience on the offhand it might help.

For a *very* long time, when I was working, we had two very large sets of tools, the first in Perl, then a complete re-write

in Python.  In both cases, all the configs (a LOT of them) were coded in the native language and imported (use/require or include).

In both cases, it allowed simple syntax error checking (perl -wc, and the equivalent for python). The other advantage was zero

code for accessing the config data; you just used it.  Those configs were native data structures, so any variety of data types

the languages supported could be used.  The python versions even used lamda's (which I hated), because the idea was that the

configs were just data, no code.  Not necessarily relevant here. One other piece; the consumed configs could be written out using

DataDumper (more or less).  For another code set, I also used pickle files to store/retrieve state data.

As for weewx, I like having the code locally in a user's directory (setup.py type install).  Avoiding sudo isn't so hard; if I remember right

it's just a matter of pointing the PID file somewhere you have write permissions and getting added to the groups necessary to get at

the weather station data (dialout in my case).  I have a start-up script in perl that runs from crontab (as weewx, not root).  As for the

weewx.conf; it's usable, but seems awkward having to keep track of brackets, and does require a library to consume (as does all the

other mentioned formats), whereas native code is just included.

Is there a concise problem statement related to all this?  It's funny that over the years (decades) I think much better working from

a problem statement.

Chris

[Tom Keffer]

On Mon, Nov 7, 2022 at 2:20 PM Chris Richmond <cric...@pobox.com> wrote:

Is there a concise problem statement related to all this?  It's funny that over the years (decades) I think much better working from

a problem statement.

Good comments. As for the config problem statement, I don't know. You'd have to ask the others. I'm happy with configobj.

However, there very definitely is a problem with the present setup.py method. Basically, distutils will be going away, and the imperative setup.py approach is being replaced with a declarative approach driven by a "build file," pyproject.toml.

[Vince Skahan]

Perhaps review the original setup.py vintage weewx design principles and see if any have changed due to the distutils deprecation ?   

• be pure python for everything
• install everything to one tree
  • except os-required things (startup file and custom syslog configs)

And of course

• any particular bugs/issues you want to fix ?
• any particular enhancements you want to add ?
• any new setups you want to be able to support ?

Is that in the ballpark ?

[Joel Bion]

One other question: is it possible to ask the user, instead of running setup.py, to run a script located in the weewx 'top level directory' that does the following: (a) uses the new form Python install/upgrade (b) then runs separate code (even if it's in python) to upgrade the .ini file, put things where they need to be etc. In other words, by building a wrapper bash script, simply work around the limitations of the new Python install mechanism - still use it for the parts that are useful, but have separately called code that does the other stuff?

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/cc4c7bce-1c8b-40a3-80c4-3ea493b342c7n%40googlegroups.com.

[Greg Troxel]

Joel Bion <jpb...@gmail.com> writes:

> One other question: is it possible to ask the user, instead of running
> setup.py, to run a script located in the weewx 'top level directory' that
> does the following: (a) uses the new form Python install/upgrade (b) then
> runs separate code (even if it's in python) to upgrade the .ini file, put
> things where they need to be etc. In other words, by building a wrapper
> bash script, simply work around the limitations of the new Python install
> mechanism - still use it for the parts that are useful, but have separately
> called code that does the other stuff?

Working around packaging system norms by asking the use to run code
seems uncool to me. A script to produce a config file seems ok. But
moving things around means the package will fail checking if the
installed files are ok. The point of read only is just that, and I
think we have to deal, not resist -- even if that's no my nature either ;_)

[matthew wall]

we have a wiki page devoted to installation options.  it is a bit haphazard right now (especially the 'ctl' pattern), but it does capture some of the issues:

https://github.com/weewx/weewx/wiki/installer-options

please refrain from modifying the wiki page - tom and i are trying to curate it into a spec that we can use for v5.  but please comment here, or create additional threads in this group as needed so we can try to address the issues.

[Chuck Rhode]

On Sun, 6 Nov 2022 14:06:39 -0800
Tom Keffer <tke...@gmail.com> wrote:

> Alternatively, after the pip install, but before first use, the user
> would be expected to "initialize" or "customize" the install via a
> tool, which is when these resources would be copied into place.

Not grökking this thread in its entirety, I'm nevertheless moved to
chime in. Extension authors appreciate the convenience of the the
ExtensionEngine class in the bin/weecfg/extension.py module and its
interlineal expository comments.

(I've just poked through the WeeWX online docs, looking for something
that would remind me of how I decided to package my Phenology
Extension, and the extension.py code comments seem the most likely
source of inspiration.)

Such scripting for installing extension(s) is, of course,
user-initiated. Thus, it seems to fall into the category of
user-customization tools, which you allude to, above. It is a third
tier of adapting WeeWX to the needs of the user and comes after
successful WeeWX installation — when all the code and data storage
locations and execution permissions have been sorted out — and after
user initialization.

... so it would seem that there will always be a need for
customization whether or not the installation tools can handle it, at
least from the perspective of extension authors. It is most helpful
that you have paid some attention toward automating these
post-installation and post-initialization customization tasks,
particularly the additions to weewx.conf that an extension requires.

(If these insertions to weewx.conf destroy the preexisting pretty
formatting, well, that's a problem for the *configobj* maintainers;
isn't it?)

--
.. Be Seeing You,
.. Chuck Rhode, Sheboygan, WI, USA
.. Weather: http://LacusVeris.com/WX
.. 45° — Wind ESE 14 mph — Sky mostly clear.

[Doug Jenkins]

Tom:

Forgive my ignorance on this topic, but this approach solves which problem for weewx? Are you proposing in v5 that users install the software via pip? would custom installs be permanently deprecated?

I wonder if it would be an easier approach for future installs to do something like pi-hole, where we have a basic-install.sh file that users execute to configure the solution. The solution would pull down from github the latest release and prepare the current directory with the solution. This way you keep everything packaged and allow the local solution to compile any required dependencies (aka pull down pip packages) and walk them through the setup.

pi-hole project : https://github.com/pi-hole/pi-hole

Doug Jenkins

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEC_YK3Ugk%2B%3DgZQSMfukTvH7Nx6AOFFekA7cA58LtLOo1Q%40mail.gmail.com.