Some wee_ctl thoughts from the draft wiki page

[Vince Skahan]

The idea of a wee_ctl utility is interesting, but a tough one perhaps given how very flexible weewx is and how creative folks have been over the years leveraging that flexibility.

I'll attach an obfuscated weewx.conf from my system.  Any new solution would have to be able to generate something that is functionally the same as this, at a very minimum.

A couple toplevel questions I guess:

• would we need to run wee_ctl potentially dozens and dozens of times to build up an aggregate .conf file (or equivalent) like my big one that I've attached ?
  
• could a wee_ctl command be able to run versus a TBD format data file of its own telling it what to do ?
• could a wee_ctl command be made idempotent so that a one-liner change to the (theoretical) data file would only do one thing differently

I'm thinking a little notionally about how you do "terraform plan" before "terraform apply" but haven't spent much time mentally drilling down on that notion, but I wanted to mention it.

And lastly:

• in the new vision's future, would we have one weewx.conf or a bunch of them that are either additive or with some superseding rules.  Again, thinking about how puppet and splunk do things, which can get really confusing in figuring out why it did what it did.

[matthew wall]

On Tuesday, November 8, 2022 at 12:10:22 AM UTC-5 vince...@gmail.com wrote:

I'll attach an obfuscated weewx.conf from my system.  Any new solution would have to be able to generate something that is functionally the same as this, at a very minimum.

on a first install, the install mechanism (either pip, venv, apt-pre, yum-pre, etc) would install the python code in the correct place.  part of that 'code' is the reference weewx.conf and reference skins - those would live with the python code and would be read-only.

then you would be prompted (as you are now) for the minimal set of information required to create a station instance.  this would probably be done by wee_ctl, for example as part of 'wee_ctl create-station'.  it would use the reference weewx.conf then modify with whatever you prompted it with.

might have to add an option to the driver prompt so you could download a driver during the initial install.

 

A couple toplevel questions I guess:

• would we need to run wee_ctl potentially dozens and dozens of times to build up an aggregate .conf file (or equivalent) like my big one that I've attached ?
  

the intent is not to create a config file editor.  you just edit your config file manually as you do now.

the wee_config tool would provide config-specific options, such as validate ("is there anything wrong with my config?") and update ("modify my config to match whatever weewx version is installed").  weewx will be backward compatible (as much as possible), but if there are breaking changes along the way, keeping that check/update function in wee_config will prevent unexpected behaviour.  installation/updating would not modify any configs - weewxd would just die after an upgrade, with messages about "this feature is no longer compatible", leaving you to modify the config file (directly and/or with wee_config to help).

• could a wee_ctl command be able to run versus a TBD format data file of its own telling it what to do ?

not sure what you mean by this

 

• could a wee_ctl command be made idempotent so that a one-liner change to the (theoretical) data file would only do one thing differently

i think that idempotence is desirable for every utility.  in the case of conf modifications, any tool that modifies a config must leave the previous state next to the current state, using a naming convention that makes the process infinitely repeatable without losing initial state.

 

 And lastly:

• in the new vision's future, would we have one weewx.conf or a bunch of them that are either additive or with some superseding rules.  Again, thinking about how puppet and splunk do things, which can get really confusing in figuring out why it did what it did.

one config file for each station instance.  for most weewx installations, that means a single config file.

if you want to maintain multiple weewx versions concurrently (e.g., for regression testing), or if you have multiple station instances (e.g., the weewx-multi use case), then managing the multiple conf files is up to you.  it must be possible (and i would argue easy, at least if we do this right), but it is not the primary use case.

where the conf lives is not yet clear.  for apt/yum it is clear - they live in /etc/weewx.  for a python install, we are considering ~/weewx of the user who created the station instance (not necessarily the same as the user who installed weewx).  from a support point of view, it would be nice if the conf file (and skins and database) were always in the same place.  i'm not sure if that is possible, since the developer use case wants these in a single directory in the user's home directory space, whereas that is non-standard for an apt/yum install.

in the case of a docker (or salt or ansible) installation, i would create the conf file *before* the installation, then just drop that config file into the 'create-station' part.  i do that right now on systems i manage with salt where i run weewx.  i use a bare-bones config file (with no comments, and only the sections that *must* have parameters specified (in older weewx, not specifying values would cause weewxd to die, even if you just wanted defaults).  so effectively i 'create-station' myself.

not sure if wee_ctl is going to help much for the salt/ansible/docker use case - so far it has been (1) to make weewx more pythonic, and (2) make it easier for users to do non-python things (e.g., install the appropriate init file(s)), and (3) move the post install stuff into a unified code for deb/rpm (which has added benefit of making bsd installs use the same pattern and code).

m

[Greg Troxel]

Vince Skahan <vince...@gmail.com> writes:

> - in the new vision's future, would we have one weewx.conf or a bunch of

> them that are either additive or with some superseding rules. Again,
> thinking about how puppet and splunk do things, which can get really
> confusing in figuring out why it did what it did.

Sort of related, I have encountered multiple styles of reading config
files and multiple cultures of config file contents over the years, and
strongly prefer configuring only non-default behavior without massive
commented-out content.

1) Ignoring comments and commented-out config, some programs seem to need
large config files where many things are set. Others have defaults for
most configurable things, and one adds config lines only when setting
something to a non-default value. This results in smaller config
files that are easier to understand, and it results in easier upgrades
because there is no need to merge changes to defaults and to comments
that explain them, because those things are in the documentation and not
in the user's config file.

2) Related, even for programs that only need non-default config, there are
(oversimplifying) two styles of config file templates for users:

A) a file that is a full example of everything that has, for each thing,
and explanatory comment and a commented-out line that configures the
default.

B) a file that has, not commented-out, and not trying to be
documentation, config lines for things that must be set

I think that type-A examples are useful, but that they should truly be
examples and no one should encourage people to copy them to weewx.conf
and then edit them for use.


3) Thoughts about defaults: Looking over the diff from my weewx.conf to
the sources, I find:

log_success set to False

location/lat/lon/altitude

station_type (Vantage)
A vast amount of stuFf that I copied at some point and should not
have. It needs:
serial port path and baud rate
model_type (belongs near 1st, I think)

It does not need
tcp port, tcp send delay (but others would, *instead* of port/baud)

It has, but could/should be default for most
loop_request (LOOP1)
iss #1
timeout 4
wait_before_retry
max_tries

It has, oddly
driver=
and i say odd because that's at end, and it points out that the
section has a name, but really this driver= is the actual config and
the things that came before are data fortthat config engine.

mqtt section
[an extension, but other than retain=False and
aggregation=aggregate, my lines are mostly real config that can't be
default, setting the broker URL and topic, setting pressures to hPa,
and putting %.1f on dewpoint.]

skin config
lots of stuff, that I changed. Makes we wonder if main config and
skin config should be separate files, but probably doesn't matter

setting group_pressure to hPa.

getting rid of the vast restful_services and setting to just mqtt. I
don't like all these random services being enabled even if they decide
they aren't configured and don't do anything. default should be none.


Overall, the stock config is 595 lines. "diff -U0" of mine is 162, and
the + lines (lines in mine not in stock, so ignoring removed lines and
ignoring diff metadata) is 134.

Having analzyed I find that weewx is not in the massive-commented-out
camp, but there's still a fair bit.

In comparison, to pick an example, /usr/pkg/etc/sudo_logsrvd.conf
has 251 lines and configures exactly nothing. Maybe that's a packaging
bug to install it, maybe it's an upstream culture bug.

clamav is even bigger. 786 lines of clamd.conf, yet:

$ egrep -v '^$' /usr/pkg/etc/clamd.conf | egrep -v '^#'
PidFile /var/clamav/clamd.pid
LocalSocket /var/clamav/clamd.sock

and even those ought to be built-in from the var path at configure time,
and probably are.

[Greg Troxel]

matthew wall <mwall...@gmail.com> writes:

> where the conf lives is not yet clear. for apt/yum it is clear - they live
> in /etc/weewx. for a python install, we are considering ~/weewx of the
> user who created the station instance (not necessarily the same as the user
> who installed weewx). from a support point of view, it would be nice if
> the conf file (and skins and database) were always in the same place. i'm
> not sure if that is possible, since the developer use case wants these in a
> single directory in the user's home directory space, whereas that is
> non-standard for an apt/yum install.

That's a really interesting point, between weewx as a system daemon with
config in the standard place, and alternate config.

I would suggest that by default the standard config is read and there is
a "-C /path/to/the/weewx.conf/I/want/to/use" and also have that file be
able to set paths to database, etc.

Also for development one could just do the normal install with
--prefix=$HOME/weewx-devel or whatever, I think.

[Doug Jenkins]

First off, Thanks Matt and Tom for the great wiki article! It is a breath of fresh air to see the level of thought and purpose put into this piece of software.

I read the wiki and I do like the idea of the CTL pattern. This will be a good start to get the "code/data" separation going. I do think it should be considered to install the configuration files in the user's home directory as that would be a secure way to manage the tool and would be a lot easier for new users to perform backups (eg backup weewx-data/ directory). 

Also a big benefit of the CTL tool/class will be for extension writers. They can invoke the tool to add/edit config items directly, kind of in the same way as the ExtensionInstaller Class works now.

I have been noodling on a solution to help users who are not as strong technically get WeeWX up and to manage it. I have been designing a WeeWX Admin interface as a companion product to help users manage their WeeWX instance as a software appliance. I will publish a separate thread on this to get feedback from the dev community later today.

Just a FYI , I added the link to the article to this thread for others to review and opine :  https://github.com/weewx/weewx/wiki/installer-options

Thanks,

Doug Jenkins
largoweather.com

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/rmiv8npk4gg.fsf%40s1.lexort.com.

[Joel Bion]

On Nov 8, 2022, at 5:48 AM, Doug Jenkins <do...@dougjenkins.com> wrote:
>
>  I do think it should be considered to install the configuration files in the user's home directory as that would be a secure way to manage the tool and would be a lot easier for new users to perform backups (eg backup weewx-data/ directory).

If by user you mean interactive user where shell access and remote login is possible,I disagree with the idea of the config files being in an interactive-access user’s home directory (unless the user is “weewx” - an account dedicated for running weewx) for the following reasons:

1) Weewx is typically run on systems which do not have multiple users. So the benefit of per interactive user is what?

2) to me, anything that is run on system boot up, started by a system init function (sysv) or systemd or whatever and stays running the entire time the system is running is a system-level application which should run from a safeguarded and isolated system account - not a user account with interactive login capability.

3) storing weewx application data ( even configuration data) remotely on the file system from the Weewx installation area would be a hassle to me. I like that my installation of Weewx is portable - fully contained in the home/Weewx tree and the only external bits are the system startup script and the pointer from my web home page to /home/Weewx/public_data.

Again, if by this you mean Weewx config data is stored in home/Weewx, I’m entirely fine with it.

[Vince Skahan]

(apologies if a previous draft reply made it out - Safari got me again in the post message gui)

On Tuesday, November 8, 2022 at 4:15:02 AM UTC-8 matthew wall wrote:

On Tuesday, November 8, 2022 at 12:10:22 AM UTC-5 vince...@gmail.com wrote:

• could a wee_ctl command be able to run versus a TBD format data file of its own telling it what to do ?

not sure what you mean by this

 

In days of old most software used autoconfig/metaconfig and you ran "sh Configure.sh" to generate a datafile with all the keyword=value pairs to configure the software during a make by writing an intermediate data file out.   This essentially was nice and idempotent, as subsequent configuring used that to preseed the answers it would ask you again.

 

in the case of a docker (or salt or ansible) installation, i would create the conf file *before* the installation, then just drop that config file into the 'create-station' part.  i do that right now on systems i manage with salt where i run weewx.  i use a bare-bones config file (with no comments, and only the sections that *must* have parameters specified (in older weewx, not specifying values would cause weewxd to die, even if you just wanted defaults).  so effectively i 'create-station' myself.

yup - a bit of a chicken-and-egg problem for updates where weewx.conf content changed (all updates do this).  You have to know experimentally get a file before knowing what to do. I tend to build a throwaway vagrant VM and diff the weewx.conf files to see what changed.  A little painful.  But yes.  Agree.

I did have a possibly crazy idea though - what if you built weewx.conf at 'runtime' based on multiple mostly-optional config definitions....

Here's an example that illustrates things using the filesystem as an example.   Anything with [ ] is a directory, [[ ]] is a subdirectory therein, [[[ ]]] one more level deep, and so on.   Anything with an = sign would have the keyword be the filename and the value be the content of the file.

  [Station]

      latitude = 12.345

  would be a file ./Station/latitude with content 12.345

Similarly:

 

  [StdRESTful]

      [[StationRegistry]]

          register_this_station = true

   would be a file ./StdRESTful/StationRegistry/register_this_station with content true

And:

  [StdReport]

       [[Belchertown]]

            [[[Labels]]]

                  [[[[Generic]]]]

                      home_page_header = Current Weather Conditions

   would be a file ./StdReport/Belchertown/Labels/Generic/home_page_header with content 'Current Weather Conditions'

So to define a keyword=value pair, you'd 'mkdir -p' the tree and create the file with content being the value on the right side of the = sign. To disable something, delete the file. If you have items with values that are comma-delimited, split them up one line per item in the element's data file's content perhaps.

(disclaimer - this breaks if you have a keyword with a / in it such as a MQTT topic like vp2/loop, but lets ignore that tidbit for now)

At runtime, assemble an aggregate weewx.conf you'd walk the tree, put the right number of [ ] around the directory names, create the variable names in the right places based on their file's depth in the tree, and put the values on the right side of the = based on the content of those files.

In short, build an aggregate weewx.conf at 'runtime' based on pieces of the puzzle that are easy to add/delete/change either manually or via tooling.

That would get you a drop-in kind of scenario where you'd have a lot of very simple one-line files in a hierarchy.   The simplest possible setup would have a data file tree that would be pretty simple based on the questions asked today - location/name/lat/lon/driver and a few other things.  Complicated setups would be pretty much unbounded.

Guess I really like drop-in componentization, being a somewhat lazy coder as well...

[Vince Skahan]

On Tuesday, November 8, 2022 at 9:48:43 AM UTC-8 Vince Skahan wrote:

what if you built weewx.conf at 'runtime' based on multiple mostly-optional config definitions....

Or if you didn't like doing it at weewx startup, hook it into './setup.py build' or whatever that evolves to be.  Same idea.  Assemble weewx.conf from pieces of the puzzle that are easier to add/delete/edit.  Service/extension authors might even be able to add asking questions to their installation steps to preseed answers.

 

[Tom Keffer]

Sorry, Vince, but I don't like your one file = one option approach. There would be dozens of files --- impossible to support.

Also, we cannot use setup.py() in an imperative approach any longer. No modifications of weewx.conf, or any other file. All you can do with the new setuptools is move things into place. Everything is declarative. No running of code. 

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/50d51375-8cf6-41cd-a8ff-d0038f586344n%40googlegroups.com.

[Vince Skahan]

On Tuesday, November 8, 2022 at 10:02:26 AM UTC-8 Tom Keffer wrote:

Sorry, Vince, but I don't like your one file = one option approach. There would be dozens of files --- impossible to support.

Also, we cannot use setup.py() in an imperative approach any longer. No modifications of weewx.conf, or any other file. All you can do with the new setuptools is move things into place. Everything is declarative. No running of code. 

Sure - just throwing it out there if an 'assemble from pieces' kind of thing is worth anything to think about investigating.  Perhaps that is sidetracking things a bit anyway.  Sorry.

Could you explain what your second paragraph means in layman's terms because I'm obviously not getting it.

• if setup.py can only move code into place as step one
• then why can't a new second step's tool 'not' using setup.py do whatever setup.py will not let you do  ?
• and then start up like we do now

So a 'build install startup' scenario becomes a 'build install configure startup' future.  One more step.

Is it really more complicated than that ?

[Tom Keffer]

Putting aside the wisdom of tacking against the prevailing Python winds, a key option that we depend on, "data_files", is going away. It allows you to list any data files, whether in a package or not, and have access to it from within the setup.py() machinery. Without that, you're left with finding the files on your own. See the link I posted earlier: https://github.com/pypa/setuptools/discussions/2648

Our new "second step" tool would use importlib.resources to find package data, which is still supported (indeed, encouraged), and then copy it into place. 

Yup. One more step. 

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/d123f83e-5f26-4ea0-be12-cb62151bf9bdn%40googlegroups.com.

[Vince Skahan]

On Tuesday, November 8, 2022 at 10:34:19 AM UTC-8 Tom Keffer wrote:

Our new "second step" tool would use importlib.resources to find package data, which is still supported (indeed, encouraged), and then copy it into place. 

Yup. One more step. 

Assuming this means weewx will now install under /usr/lib/python3/dist-packages (pi example) like any other python library, will the new tool also drop the core skins like Seasons into place in /home/weewx/skins ?

In other words anything writable/editable or user-added remains as-is under /home/weewx ?

[Greg Troxel]

Vince Skahan <vince...@gmail.com> writes:

> On Tuesday, November 8, 2022 at 10:34:19 AM UTC-8 Tom Keffer wrote:
>
>> Our new "second step" tool would use importlib.resources

>> <https://docs.python.org/3/library/importlib.resources.html> to find
>> package data, which is still supported (indeed, *encouraged*), and then

>> copy it into place.
>>
>> Yup. One more step.
>>
>
> Assuming this means weewx will now install under
> /usr/lib/python3/dist-packages (pi example) like any other python library,
> will the new tool also drop the core skins like Seasons into place in
> /home/weewx/skins ?
>
> In other words anything writable/editable or user-added remains as-is under
> /home/weewx ?

Why don't they belong in $prefix/etc/weewx? That's where config files
(that should be edited) belong in the normal scheme.

[Tom Keffer]

Vince: pip can install to many places. Yes, it can install to the system's dist-packages, although I don't think that's a great idea. It requires sudo and can break existing applications. Better idea is to use "--user" and install to ~/.local, or install to a virtual environment, or use pipenv. 

The new tool would then install to either /home/weewx or ~/.weewx, or something similar.

Greg: The package installers would continue to install to /etc/weewx. For those who want something that "just works" that continues to be a good idea. However, there is a disadvantage: it requires sudo to install and edit. Many of us also prefer to have everything under one subdirectory (such as /home/weewx or ~/.weewx, or what have you).

While many apps install to $prefix/etc, not all do. Many depend on configuration settings under ~/.local or their own subdirectory. It's by no means universal. In fact, it feels to me that the trend has been towards non-privileged installs. Npm, nvm, node, chrome, pyenv, JetBrains products, slack (via snap), docker-desktop, virtualbox --- none of them require root privileges and all of them put config information under the user directory.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/fadf79ab-49aa-4ab8-95bb-dd4b8ca7a906n%40googlegroups.com.

[Tom Keffer]

Misspoke about virtualbox. It does not require root to install a vm, but it does to install the host. Config information is held in ~/.config

[Vince Skahan]

On Tuesday, November 8, 2022 at 11:39:33 AM UTC-8 Tom Keffer wrote:

Vince: pip can install to many places. Yes, it can install to the system's dist-packages, although I don't think that's a great idea. It requires sudo and can break existing applications. Better idea is to use "--user" and install to ~/.local, or install to a virtual environment, or use pipenv. 

The new tool would then install to either /home/weewx or ~/.weewx, or something similar.

Again, totally lost.  First we 'may not' install outside system locations.  Now we will do just that.

I'm going to stop reading I think at this point.

Build something and I'm sure lots of people will try it out...please pick 'one' way to do it, whatever that way is.

 

[Tom Keffer]

Think of python as a system of jails, whose location is determined by the version of python that invokes the install. If you use /usr/bin/python, then your jail is something like /usr/lib/python3.10/dist-packages. If your version of python is invoked from pyenv, then it's ~/.pyenv/versions/3.10.5/lib/python3.10/site-packages. If you use the --user flag, then your jail is slightly different: ~/.local/lib/python3.10/site-packages, but you get visiting rights to the pyenv packages. If you install with pipenv, then your jail will be located at something like ~/.local/share/virtualenvs/tkeffer-g5o4x4y2/bin/distro. For virtual environments created using venv, you get to specify the location of the jail.

The point being that each instance of the python interpreter comes with a well-defined "jail" and you're not allowed to roam around free into the other jails, nor the rest of the file system. It's kind of like sandboxes in web browsers. You're stuck in your own sandbox.

Only one of them, the first example, requires root. 

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/01d0f44f-8d69-4799-a55a-69f6822ad046n%40googlegroups.com.

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> Greg: The package installers would continue to install to /etc/weewx. For
> those who want something that "just works" that continues to be a good
> idea. However, there is a disadvantage: it requires sudo to install and
> edit. Many of us also prefer to have everything under one subdirectory
> (such as /home/weewx or ~/.weewx, or what have you).

Sure, I realize some people prefer a root for all of weewx, and some
want it to be like one of the other 200 python programs.

> While many apps install to $prefix/etc, not all do. Many depend on
> configuration settings under ~/.local or their own subdirectory. It's by no
> means universal.

Agreed, but I think there is a difference in semantics. Some things
are programs that are installed in the system and configured, as a
single instance, systemwide. I see weewx in the normal case as like
that, similar to nginx, sshd, pgsql, and many others.

Things that look in the user's homedir I see more like gimp, where the
program is installed so that it is available, but there is no singleton
system instance of it. Each user runs the code on their own data.

> In fact, it feels to me that the trend has been towards
> non-privileged installs. Npm, nvm, node, chrome, pyenv, JetBrains products,
> slack (via snap), docker-desktop, virtualbox --- none of them require root
> privileges and all of them put config information under the user directory.

Yes, but privilege and system vs user are not entirely the same thing.
And I would argue that the trend to containerization in various forms is
driven by a lack of discipline in software engineering in terms of
stable APIs, leading to needing pinned dependencies far more than you
would have 15 years ago. So I don't think it's really a good thing; it
just is how it is. But that's off topic really, and I think the path
you are heading down is going to be fine.

[Joel Bion]

Greg’s note conveys what I was trying to say about system applications better than I could. The truth is, if Weewx decides to go the “stuff in installing user’s home directory” route - I’ll just make it installed by the Weewx user, which gets much (not all) of what I want. I do believe that the normal Python installers are optimized for a use case that isn’t Weewx, or any , and programs like ufw and fail2ban that are - as Greg points out - single instance system installs vs something any user can run. They are going to struggle to fit into the python install shoehorn even more than Weewx, which doesn’t require root access to run, like these two do.

Sent from my iPhone

> On Nov 8, 2022, at 3:36 PM, Greg Troxel <g...@lexort.com> wrote:
>
> 

> --
> You received this message because you are subscribed to the Google Groups "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/rmitu39f280.fsf%40s1.lexort.com.

[Tom Keffer]

For those who are still interested, finally something concrete!

1. The document outlining the goals and strategies of V5 has been updated. https://github.com/weewx/weewx/wiki/Version-5

2. New document on strategies for installing WeeWX using pip: https://github.com/weewx/weewx/wiki/pip-install-strategies 

I'd really like some feedback on this one. Did I miss any viable approaches? I'm leaning towards using pipx to do the install, although I could be talked into other approaches. 

3. Best of all, there is now an alpha version of WeeWX, which can be installed entirely using pip!

Try this to install into a virtual environment (which you can then delete when you're done playing):

cd ~

# Create a virtual environment in weewx-venv

python3 -m venv weewx-venv

# Activate it

source weewx-venv/bin/activate

# Then install WeeWX into it using pip

python3 -m pip install weewx 

To use it, create a new station with

weectl station create

This will create a directory ~/weewx-data, which will hold the configuration file, skins, and (after startup) the database, and HTML files.

Then run:

weewxd

-tk

[Vince Skahan]

Cool - some quick testing things I noticed in a ubuntu2204 vagrant vm...

The vagrant base ubuntu2204 VM is very minimal so I needed to add a couple packages:

• sudo apt-get update
• sudo apt-get install -y python3-venv  # for venv itself
• sudo apt-get install -y gcc           # to compile pyephem within the venv

I also did the station create explicitly just to try that too (which worked fine):

• weectl station create --driver weewx.drivers.simulator --no-prompt

Also tested calling weewxd from outside of the venv via a full path ala ~/weewx_venv/bin/weewxd which I suppose is where a future systemd startup file use if an absolute path was specified (???)

Re: your installer mechanisms options, I lean toward this way since it relies on just core python and not a third party wrapper tool that might theoretically go away someday or might not work on all the unusual/creative places folks run weewx on like NAS or under WSL and the like.

For the future, I guess I'd like to also see docs re: how to upgrade weewx and/or switch to a specified version of weewx.  Sometimes folks need/want to freeze to a particular version.

Very cool though as-is !

[Tom Keffer]

Thanks for the quick feedback.

Also tested calling weewxd from outside of the venv via a full path ala ~/weewx_venv/bin/weewxd which I suppose is where a future systemd startup file use if an absolute path was specified (???)

Yes, exactly so. You don't need to 'activate' the virtual environment --- all that does is put it in your PATH. If you already have an explicit, absolute path to weewxd, that's good enough

Re: your installer mechanisms options, I lean toward this way since it relies on just core python and not a third party wrapper tool that might theoretically go away someday or might not work on all the unusual/creative places folks run weewx on like NAS or under WSL and the like.

I just worry about the multiple steps involved and a newbie's ability to do them. In any case, any of the ways I outlined in the "pip" document will work, it's just that I'd like to settle on one "official" way that we document and encourage users to use.

For the future, I guess I'd like to also see docs re: how to upgrade weewx and/or switch to a specified version of weewx.  Sometimes folks need/want to freeze to a particular version.

One nice feature about pip is that you can install a specific version. For example

pip install weewx==5.1.3

So many docs, so little time...

-tk

[Vince Skahan]

Hundreds of newbies seem to be ok with the pihole approach...

• https://docs.pi-hole.net/main/basic-install/  <= one-step automated install
  
• https://docs.pi-hole.net/main/post-install/   <= what to tweak
  
• https://docs.pi-hole.net/main/update/         <= how to update
  
• https://docs.pi-hole.net/main/uninstall/      <= how to uninstall
  

(and their docs are soooooo nice looking now too....)

For weewx a similar bash script approach could handle os detection (apt/yum) and installing a webserver (optionally?) and hooking that in.  Just go with nginx there since it's light+fast and relatively simple.

Personally I love the "pihole -up" to upgrade to current.  You can't get much simpler than that.   Maybe a future "wee_ctl" feature (?)

Folks who want to install the long (ten line) way can of course go appropriately wild.  Just as pihole does, continue to give'em the instructions or they can install the pre-packaged variant for their os.

Re: docs - it would be interesting to see how pihole's are so blasted good looking.....

[Tom Keffer]

I'm seeing that style of documentation everywhere now. For example, the pip User Guide does something similar. I wonder what tool produces it?

I do like using git to distribute and update. More and more tools seem to be going that direction, too.

In any case, those are larger projects with staff that are probably getting paid. I'm just one senior citizen, with a life, plugging away...

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/9b6757d9-7baf-429a-ba97-84835066abebn%40googlegroups.com.

[Doug Jenkins]

pi-hole uses mkdocs, which is a website generator that translates markdown to a nice website solution. it is simple to setup.

pi-hole/docs: The official Pi-hole documentation (github.com)

deployment can be done with github actions where you approve a pull request to main, GitHub can run the builder and send the html to a target directory.

now writing the documentation, well that is a different story.

DDJ

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEDpcvx%3D0_yiEcyuW3PRyHF-F%3DqL2ySXfRB%3DyfvZJ-9zBg%40mail.gmail.com.

[Vince Skahan]

Thanks Doug.   It turns out I used mkdocs at my last $job for our local docs but those brain cells apparently retired with me it seems 18 months ago.  Thanks for refreshing my memory.

I also asked on reddit and got pointed to https://github.com/pi-hole/docs which has the howto for how they build those docs.  If I get a chance I'll try to run the weewx docs through and see what they look like with that styling.

And github actions is a great idea for automating !

[Tom Keffer]

Wish this had all existed when I first started writing the docs 10+ years ago. I mean, which would you prefer?

<li>Run <span class=code">weewxd</span></li>

or

- Run `weewxd`

But, I'm not about to rewrite them just to switch technologies. 

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/caa275f3-898f-4555-a7c3-391b2846b320n%40googlegroups.com.

[Doug Jenkins]

I have used Python Sphinx (https://www.sphinx-doc.org/en/master/) for a work project a couple of years ago. It has the ability to "autodoc" your python code if you follow their commenting template for each function and class. The tool was simple to setup and I had a build process that simply created the html files and pushed them to our docs webserver location each time we did a release. 

Looking back, I would have used mkdocs as markdown is more accepted as a documenting language than RST (ReStructuredText).

One benefit of using mkdocs that should be considered is that we can have the documentation in multiple languages. Since the Seasons skin was internationalized and we have about the same amount of weather stations inside the US as well as outside the US, having a platform like this allows for the documentation to be translated and easily accessible.

Given the large changes to install and manage WeeWX in v5, it begs us to consider a new documentation platform as refactoring all that html code with the changes will take as much time as migrating the raw text to markdown. 

I also to hate to say this, but this could spillover for a large website update for WeeWX in general as the documentation and the station registry is about 90% of the website today.

All of this will take a lot of time. Maybe we should run a poll with the existing users to see what they want to see in the documentation and website before looking at moving tools and such.

DDJ

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/caa275f3-898f-4555-a7c3-391b2846b320n%40googlegroups.com.

[Vince Skahan]

I did a 'quick' pass through running the wiki through mkdocs just as an example and it looks pretty nice to my eyes FWIW....

Navbar to the left has to be made manually in mkdocs.yml and the table of contents to the right is auto-generated based on the tagging of the markdown.

Note - this is just the minimal mkdocs look'n'feel.  I didn't apply the nicer eye candy the pihole guys did on their docs since this was just a quick PoC.

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> 1. The document outlining the goals and strategies of V5 has been updated.
> https://github.com/weewx/weewx/wiki/Version-5

IMHO python >= 3.8 is good enough for a few months from now but I don't
use Laggard Term Stable :-) dropping <= 3.6 and 2.7 is good.

> 2. New document on strategies for installing WeeWX using pip:
> https://github.com/weewx/weewx/wiki/pip-install-strategies

It seems obvious that extensions should use the same pip strategy and
depend on weewx. Yes, need to cope until they are updated, but need to
push them to align.

> I'd really like some feedback on this one. Did I miss any viable
> approaches? I'm leaning towards using pipx to do the install, although I
> could be talked into other approaches.

It should be easy to choose a different place for what is "etcdir" and
"vardir" where the config and db and html output live. They should be
able to be separate. skin/conf should be able to be on a ro fs.

I think 'weewx-data' is an artifact of the previous install and would be
separate config and state were we starting fresh.

[Greg Troxel]

Greg Troxel <g...@lexort.com> writes:

> It seems obvious that extensions should use the same pip strategy and
> depend on weewx. Yes, need to cope until they are updated, but need to
> push them to align.

my opinion about a basic rule: in a pip venv, everything is managed by pip.

[Tom Keffer]

It seems obvious that extensions should use the same pip strategy and
depend on weewx.   Yes, need to cope until they are updated, but need to
push them to align.

I'd like to install extensions using pip, but they would have the same problem that pip cannot be used to install or manage data, such as the settings an extension might use. Any ideas on how to do that would be welcome.

It should be easy to choose a different place for what is "etcdir" and
"vardir" where the config and db and html output live.  They should be
able to be separate.  skin/conf should be able to be on a ro fs.

As always, you can specify different locations for the skins (SKIN_ROOT), html files (HTML_ROOT), database (SQLITE_ROOT), etc

I think 'weewx-data' is an artifact of the previous install and would be
separate config and state were we starting fresh.

I'm having a hard time decoding what you're trying to say. Can you be more specific?

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

>> I think 'weewx-data' is an artifact of the previous install and would be
>> separate config and state were we starting fresh.
>
> I'm having a hard time decoding what you're trying to say. Can you be more
> specific?

In the context of packaging, a package has code and other RO resources,
and then there are config files that according to hier(7) and ancient
tradition from BSD belong in etc. And changeable files that belong in
/var/db or /var/log, which is usually spelled $prefix/var/PKGNAME within
packaging. So the place the skins and weewx.conf goes is etc, which is
different from where the db belongs which is varlike

pip has its own opinions but that's basically a bug to work around as
the world is not python; all per-language helpers need to play nice with
the larger packaging ecosystem.

[Tom Keffer]

OK, now I get you.

It may be tradition, but the trend has been towards avoiding root installs, which by necessity require installation in user directories.

In any case, if you want the more traditional approach, there's always an apt install.

[Tom Keffer]

Vince, that look at mkdocs is compelling. It would be so nice to switch from HTML to GFMD.

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/5092796d-ac71-48af-867a-254c95b7e702n%40googlegroups.com.

[Vince Skahan]

Let me get the rest of the wiki PoC done and I'll upload a snapshot of the results someplace for you to take a look at.

Not today - it stopped raining momentarily :-)

I'll do one of the docs html=>md and process with mkdocs next.

Don't sweat editing the official docs in the interim, it's not going to affect anything.  The wiki being already markdown is particularly straightforward to catch up on (for me here, git pull of the wiki github repo, then "mkdocs build" to generate the html results suitable for rsyncing up to someplace)

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> OK, now I get you.
>
> It may be tradition, but the trend has been towards avoiding root installs,
> which by necessity require installation in user directories.
>
> In any case, if you want the more traditional approach, there's always an
> apt install.

Root vs not is about system prefix. Installing to ~/weewx can still
respect the hierarchy and not need privs. pkgsrc actually can be
bootstrapped without root to a prefix (that the user running it can
write of course).

my comment was really about pushing the config vs db separation into
weewx proper as a concept, and partly because the apt, pkgsrc, whatever
packagers need to take the release and munge it into the various areas,
so the more it's "pass args" and the less "patch" that's better.

[Tom Keffer]

Perhaps the usage output of "weectl station create" will help:

% python3 -m weectl station create --help

usage: weectl station create [--config=CONFIG-PATH] \

                             [--driver=DRIVER] \

                             [--location=LOCATION] \

                             [--altitude=ALTITUDE,{foot|meter}] \

                             [--latitude=LATITUDE] [--longitude=LONGITUDE] \

                             [--register={y,n} [--station-url=STATION_URL]] \

                             [--units={us,metricwx,metric}] \

                             [--skin-root=SKIN_ROOT] \

                             [--sqlite-root=SQLITE_ROOT] \

                             [--html-root=HTML_ROOT] \

                             [--docs-root=DOCS_ROOT] \

                             [--examples-root=EXAMPLES_ROOT] \

                             [--no-prompt]

In what follows, WEEWX_ROOT is the directory that contains the

configuration file. For example, if "--config=/Users/tkeffer/weewx-

data/weewx.conf", then WEEWX_ROOT will be "/Users/tkeffer/weewx-data".

optional arguments:

  -h, --help            show this help message and exit

  --config CONFIG-PATH  Path to configuration file. It must not already exist.

                        Default is "/Users/tkeffer/weewx-data/weewx.conf".

  --driver DRIVER       Driver to use. Default is "weewx.drivers.simulator".

  --location LOCATION   A description of the station. This will be used for

                        report titles. Default is "WeeWX".

  --altitude ALTITUDE,{foot|meter}

                        The station altitude in either feet or meters. For

                        example, "750,foot" or "320,meter". Default is "0,

                        foot".

  --latitude LATITUDE   The station latitude in decimal degrees. Default is

                        0.00.

  --longitude LONGITUDE

                        The station longitude in decimal degrees. Default is

                        0.00.

  --register {y,n}      Register this station in the weewx registry? Default

                        is "n" (do not register).

  --station-url STATION_URL

                        Unique URL to be used if registering the station.

                        Required if the station is to be registered.

  --units {us,metricwx,metric}

                        Set display units to us, metricwx, or metric. Default

                        is "us".

  --skin-root SKIN_ROOT

                        Where the skins will be located, relatve to

                        WEEWX_ROOT.

  --sqlite-root SQLITE_ROOT

                        Set the location of the sqlite directory, relative to

                        WEEWX_ROOT.

  --html-root HTML_ROOT

                        Where generated HTML and images will go, relative to

                        WEEWX_ROOT.

  --docs-root DOCS_ROOT

                        Where to put the documentation. Default is

                        "$WEEWX_ROOT/docs".

  --examples-root EXAMPLES_ROOT

                        Where to put the examples. Default is

                        "$WEEWX_ROOT/examples".

  --no-prompt           If set, do not prompt. Use default values.

[Greg Troxel]

That looks flexible enough. I guess the package will just have the pip
and then someone will create a station, perhaps multiple someones on the
same box with different serial/etc. ports (unlikely I know). Thanks for
the discussion.

[Greg Troxel]

Not really weewx, but beware private pip repos:

https://pytorch.org/blog/compromised-nightly-dependency/

[G Hammer]

I do like the idea of a git install. pihole and netdata do this. netdata is a fairly involved install from their side of it. For the user, as long as you have curl installed, you just copy/paste one line and wait.

As an aside, let's say that you just don't want things in your home directory that aren't, well, your data.

If you created a user, say weewx, logged in as weewx and ran the install, will any user be able to run/edit/add to that install?

[Tom Keffer]

By default, weectl will create the data in your home directory at ~/weewx-data, but you can place it anywhere you want (provided, of course, that you have write privileges). 

Whether someone else can write to it depends on OS privileges, just like any other directory or file.

You can run weewx out of a git repository, which is what I typically do. However, what you probably do not want to do is place site-specific state in the repository. such as the database or weewx.conf,. You'll want to run "weectl station create" to place those someplace else.

For V5, it would look something like:

pip install CT3 pyusb pyserial configobj mysqldb pillow

cd ~/git

git clone https://github.com/weewx/weewx.git

cd weewx/bin

python3 -m weectl station create

... weectl prompts for station information...

python3 -m weewxd

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/01b3b157-10d9-42cb-9a77-637837e216b8n%40googlegroups.com.