Configuration management with WeeWX

[Ben Cotton]

After far too long, I'm getting my server configs into Ansible. But looking at WeeWX, it doesn't seem like it would like that. Specifically the way it merges existing and new configs on an upgrade and the "Do not modify this. It is used when installing and updating weewx." in the config file. I found a playbook (https://github.com/digitalbear/ansible.weewx) that just clobbers the post-upgrade config file. Does anyone else have experience with this?

[mwall]

On Sunday, December 22, 2019 at 10:36:17 PM UTC-5, Ben Cotton wrote:

Does anyone else have experience with this?

first of all, be sure you distinguish between installation and upgrade.  i've used configuration tools (ansible, salt, etc) to do hundreds of system installations, but that is a different process/configuration than using them to upgrade existing installations.

as far as weewx is concerned, the skins are backward compatible - an existing skin should work without modification with a new weewx release (we go out of our way to ensure that changes to weewx will not break existing skins).  the weewx configuration file *may* need to be modified - an existing weewx configuration file *might* need to be modified to work with a new weewx release (a change from x.y.n to x.y.m should be ok, but a change from x.y.0 to x.z.0 might require modifications)

if you *do* use the configuration tool to do upgrades, there are two different approaches:

option 1: let weewx handle the upgrades.  with this approach, you would make the configuration management tool invoke setup.py or apt-get in order to modify the weewx configuration file.

option 2: make the configuration management tool handle the upgrades.  with this approach, you just put the new weewx configuration file into the configuration tool, and let it overwrite any existing configuration.

btw, this also applies to weewx extensions.  you can either invoke wee_extension or you can have the configuration tool drop the skins/.py files into the appropriate locations.

m

[vince]

On Sunday, December 22, 2019 at 7:36:17 PM UTC-8, Ben Cotton wrote:

After far too long, I'm getting my server configs into Ansible. But looking at WeeWX, it doesn't seem like it would like that. Specifically the way it merges existing and new configs on an upgrade and the "Do not modify this. It is used when installing and updating weewx." in the config file. I found a playbook (https://github.com/digitalbear/ansible.weewx) that just clobbers the post-upgrade config file. Does anyone else have experience with this?

Ben - I've asked for changes in the last few years (and Matthew+Tom have made many) along the lines of making it a bit easier to do weewx with DevOps mindset.   What in particular are you battling ?

[Ben Cotton]

On Mon, Dec 23, 2019 at 3:06 PM vince <vince...@gmail.com> wrote:
>
> Ben - I've asked for changes in the last few years (and Matthew+Tom have made many) along the lines of making it a bit easier to do weewx with DevOps mindset. What in particular are you battling ?
>

I may be battling my own tendency to over-engineer more than anything.
I'm really looking at a hypothetical problem than a real one. What
I've done for now is set up a playbook to install today's current
version. As infrequently as I reinstall this machine (before
yesterday, I last did an install probably 10-ish years ago), that's a
fine approach. I can always update the playbook with a post-upgrade
config file as needed.

But in my ideal scenario, I'd like to see something like the way
Apache httpd and other daemons parse configs. So there would be
/etc/weewx/weewx.conf that would have the default settings and then I
could overwrite things as needed in /etc/weewx/weewx.local and/or
/etc/weewx/conf.d/*.

Another approach would be, at least on RPM-based systems, for the RPM
to just write an .rpmnew file and leave the existing config in place.
I'm not sure how much effort that would take to keep weewx happy—I
haven't looked in detail to see what sort of things get changed in the
config file between releases. Along this line, I'd briefly looked at
packaging weewx in the Fedora repos (or at least a COPR build), but
the amount of shell script in the RPM spec file changed my mind. :-)

[Thomas Keffer]

Through the years, I've looked at lots of different configuration schemes, but haven't found any I like better. 

The big advantage of the present system is that all the state is held in a single place: weewx.conf. That makes support easy: just post the file. It also makes it easy to run more than one instance of weewx: just point each daemon to an appropriate config file. Upgrade merges are easy: just upgrade a single file, usually through a simple configobj merge.

By contrast, the weewx.d approach spreads state over multiple places.

I have also looked at "drop in" architectures, similar to Apache's "sites-available" approach. To enable a service, you would symlink its python file from "services-available" to "services-enabled". I can't remember exactly why I abandoned the idea, but it came down to python not liking symbolic links. It also makes remote debugging tough.

No perfect system, but this one has served us well.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAJox116eGQWipneTqwW5jQHamoqNvj-X%2Bk8u9kgJE0OHpZA6_Q%40mail.gmail.com.

[mwall]

On Friday, December 27, 2019 at 12:06:36 AM UTC-5, Ben Cotton wrote:

Another approach would be, at least on RPM-based systems, for the RPM
to just write an .rpmnew file and leave the existing config in place.

if you install/update weewx using the redhat (or suse) rpm package, then you get exactly this.

[Ben Cotton]

No you don't. Updates via the RPM package will overwrite the existing config:
https://github.com/weewx/weewx/blob/master/pkg/weewx.spec.in#L153

So while it writes the .rpmnew file, but it still makes updates to the
existing config file, which is not what I said.


--
Ben Cotton

[vince]

On Thursday, December 26, 2019 at 9:06:36 PM UTC-8, Ben Cotton wrote:

But in my ideal scenario, I'd like to see something like the way
Apache httpd and other daemons parse configs. So there would be
/etc/weewx/weewx.conf that would have the default settings and then I
could overwrite things as needed in /etc/weewx/weewx.local and/or
/etc/weewx/conf.d/*.

Yup - there have been multiple discussions along those lines over the years, but there were as many 'cons' as there were 'pros' and there was a many-thousand-sites pre-existing condition already out there in the field, so it was basically decided to stay the course.

Perhaps you might itemize exactly which tasks you are trying to automate specifically and maybe we can collectively get you there within the current implementation.

For my raspi (not my main site), I settled on:

• be able to get a working Simulator setup in one step via a bash script, with 'some' of my edits to weewx.conf where sed does the job easily.
• install all skins with the wee_extension utility
• install all extensions with the wee_extension utility
• keep all 'my' skins+extensions+scripts in GitHub since I'd never find's otherwise :-)

At that point, all I'd need to do would be to add in my edits to weewx.conf, which admittedly is still to-do:

• ansible lineinfile is very weak and doesn't work too well with weewx.conf's configobj format
• ansible blockinfile 'probably' could handle editing skin blocks (example - I have a zillion lines to tweak Belchertown) but I haven't tested it

And of course I'd need to switch from Simulator to my real Driver (could have been done in the first step above probably).

But it's not generally a big deal to automate enough.....not 100%....but good enough for my needs.

Some repos you might surf:

• https://github.com/vinceskahan/weewx-install-script/blob/master/install-weewx.sh - this is a one-step script to install via setup.py that lets you pick your version and git branch 'or' alternately use the dpkg versions.   Edit a couple variables at the top and run on a clean raspi or equivalent Debian(ish) system.
  
• I have lots of other repos there for Dockerizing, using Vagrant, etc.
• Many of them have snippets for using sed to edit in lat/lon and other variables many people edit that are unique enough that it's doable without breaking weewx.conf

Regardless - suggest you start with "I need to automate doing this" and ask away.  Somebody's likely already figured out how to do it.  Then you'd just need to connect the Legos into a whole that works for you.