Driver Installation in Version 5

[Karen K]

Today I started testing Version 5. I chose the Debian installer. I know, you would prefer pip, but I do not want to maintain 2 parallel update paths.

There were no issues during installation. The simulator driver is working well.

No I tried to install the MQTTSubscribe driver. First I installed the package. No error occurred. The module was saved to /etc/weewx/bin/user, and weewx.conf got updated.

Now I ran:

sudo weectl station reconfigure --driver=user.MQTTSubscribe

I was asked some questions, and finally got an error message:

Traceback (most recent call last):

  File "/usr/share/weewx/weectl.py", line 75, in <module>

    main()

  File "/usr/share/weewx/weectl.py", line 67, in main

    namespace.func(namespace)

  File "/usr/share/weewx/weectllib/station_cmd.py", line 222, in reconfigure_station

    weecfg.station_config.station_reconfigure(config_path=namespace.config,

  File "/usr/share/weewx/weecfg/station_config.py", line 102, in station_reconfigure

    config_config(config_path, config_dict, dry_run=dry_run, *args, **kwargs)

  File "/usr/share/weewx/weecfg/station_config.py", line 134, in config_config

    config_driver(config_dict, driver=driver, no_prompt=no_prompt)

  File "/usr/share/weewx/weecfg/station_config.py", line 350, in config_driver

    driver_editor, driver_name, driver_version = weecfg.load_driver_editor(final_driver)

  File "/usr/share/weewx/weecfg/__init__.py", line 423, in load_driver_editor

    driver_module = importlib.import_module(driver_module_name)

  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'user.MQTTSubscribe'

But the file is there:

user@weewx:~$ ls /etc/weewx/bin/user/

installer  MQTTSubscribe.py

user@weewx:~$ 

If I insert print(sys.path) into /usr/share/weewx/weecfg/__init__.py I get:

['/usr/share/weewx', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/etc/weewx/bin']

I have no idea where to look now. Could someone help?

[Vince Skahan]

On Saturday, November 25, 2023 at 8:30:15 AM UTC-8 Karen K wrote:

If I insert print(sys.path) into /usr/share/weewx/weecfg/__init__.py I get:

['/usr/share/weewx', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/etc/weewx/bin']

The last item in the path tells me it's looking for /etc/weewx/bin/user/MQTTSubscribe.py

I seem to recall there was a bug at one point in time for the betas related to where weewx looked for bin/user things and the fix was to copy the file into the expected location.   Find where your installation put MQTTSubscribe.py and copy it to /etc/weewx/bin/user and see if that works for you.

 

[Karen K]

Vince Skahan schrieb am Samstag, 25. November 2023 um 19:02:15 UTC+1:

On Saturday, November 25, 2023 at 8:30:15 AM UTC-8 Karen K wrote:

If I insert print(sys.path) into /usr/share/weewx/weecfg/__init__.py I get:

['/usr/share/weewx', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/etc/weewx/bin']

The last item in the path tells me it's looking for /etc/weewx/bin/user/MQTTSubscribe.py

Yes. And there the file is indeed located.

 

I seem to recall there was a bug at one point in time for the betas related to where weewx looked for bin/user things and the fix was to copy the file into the expected location.   Find where your installation put MQTTSubscribe.py and copy it to /etc/weewx/bin/user and see if that works for you.

The file is there and was always there:

user@weewx:~$ ls /etc/weewx/bin/user/

installer  MQTTSubscribe.py

user@weewx:~$ 

But there is the error.

[Vince Skahan]

Sorry, I don't know.  Perhaps copy it to the same place the Vantage driver is in and just reference it similarly as a workaround.

[Karen K]

Finally, I found out. It refers to the deb packet of version 5.0.0b10.

There are 2 directories called "user": /etc/weewx/bin/user and /usr/share/weewx/user. Both of them are included in Python's search path, where it looks for modules. Python decides which one it uses to really load modules. And it uses the other one.

As a workaround I did:

cd /etc/weewx/bin

sudo rm user

sudo ln -s /usr/share/weewx/user user

After the installation of WeeWX "extensions.py" resided in /usr/share/weewx/user. That is the reason I set the link the way I did.

This may be an issue for the development team, @Tom, except it is already known.

[Tom Keffer]

Sorry. I should have followed this thread more closely. 

This issue is already known (always read the Upgrade Guide!). See https://weewx.com/docs/5.0/upgrade/#new-location-for-user-directory

Our plan for upgrades using a package installer is to copy the user directory over to /etc/weewx/bin/user, then timestamp the old directory /usr/share/weewx/user.YYYY.MM.dd.hh, or something similar. However, this has not been implemented yet.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/4c3d068d-ec47-4935-a7fb-26879811f699n%40googlegroups.com.

[Karen K]

Tom Keffer schrieb am Sonntag, 26. November 2023 um 13:50:23 UTC+1:

This issue is already known (always read the Upgrade Guide!). See https://weewx.com/docs/5.0/upgrade/#new-location-for-user-directory

This is a slightly other problem here. I did a new clean install, no upgrade.

My problem was, that the old directory /usr/share/weewx/user still exists and Python looks for modules there only. 

So the extensions were installed to /etc/weewx/bin/user, but Python did not look for them there. 

[Tom Keffer]

/etc/weewx/bin is added to the Python path dynamically when weewxd is started up. However, because it is added to the end of the path, the interpreter will find the version in /usr/share/weewx first.

So, why not add it to the beginning of the Python path? Because that causes problems for legacy setup.py installs. In this case, the interpreter will find the code in /home/weewx/bin first, which is the old V4.10 code. The V5 code is under ~/weewx-venv.

In summary, there's no good solution except to warn users.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/35032165-08d5-4415-8e9d-d2996ce8ca84n%40googlegroups.com.

[Joel Bion]

I seriously think there is a formal case study that needs to be written about the challenges of moving WeeWX to the current Python installation model. 

There’s a lesson here for not just the Python developers but others about the difficulty of this transition, because the way WeeWX did things (all things under /home/weewx) was not unreasonable but what has been required to migrate, and the need to support custom package managers, etc…. I just think that this information is useful.

Sent from my iPhone

On Nov 26, 2023, at 11:33 AM, Tom Keffer <tke...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEAtRxWk2JG7OYC_ontEnv-fFLLdXGUr2KDzf_nyv24XaQ%40mail.gmail.com.

[Tom Keffer]

Not sure what you're asking for. There is quite a bit of information in the Upgrade Guide. What's missing?

[Vince Skahan]

[...long answer follows...]

v5 beta pip user who came from setup.py installation here so let me pass along my experience and what I did....

This is my personal method and definitely differs from the official steps, but maybe it will help identify one way to approach setting up a test setup before throwing the switch and going live with it....

I might add that I'm a Vantage VP2 with datalogger user here, so it's easy to stop v4, run v5, and then even go back to v4 if needed (I never needed to do so) and the datalogger will handle catching up an old v4 installation.  This is a feature that not all stations have, but it was a nice security blanket here.

Anyway - apologies for this being verbose.  Writing it down just in case somebody needs to search for it.

1.  Set up a v5 pip virtual environment and the station's weewx-data directory

• created a simulator station using the pip quickstart steps, but did a couple things differently:
  • I added the --no-prompt option to 'weectl station create' so it didn't prompt me
  • I didn't put the weewx.service file into place (yet)
• installed all the skins and extensions I have in my v4 setup.py config into the new v5 pip weewx-data using 'weectl extension install'
• installed any python modules my skins/extensions require (requests and paho-mqtt, to name two) into the python venv
  • at this point I could activate the python venv and run weewxd and see simulated data returned
  • and after 5 minutes it should save to the simulator weewx.sdb and run the reports
  • if anything goes boom, you missed adding something to your new pip based installation
• do not (yet) edit weewx.conf to do anything that acts like an uploader (rsync, mqtt, pws, wu, cwop, and so on)
• do not (yet) do anything to hook into systemd
• if you want to hook the v5 pip public_html into your web at a test path, perhaps http://xyz/weewx-v5, you can do that now

2. Copied my NOAA files into place.  The historical ones take too long to regenerate if you have many years.  When we get to it later the current month/year will be overwritten correctly.

3. Copied my archive db into place.  Don't sweat it getting overwritten with simulator data.  We'll copy the real db (again) when putting v5 pip into production

4. Run weewxd interactively again and let it run the reports a couple times.  Verify nothing went boom.  You should see full graphs etc. and NOAA reports.  Don't sweat the contents.  Just ensure it generated them.

5. At this point you have a pretty close to done config.  Go ahead and edit weewx.conf and put in your normal edits there, but be sure to 'enable = false' all your uploaders etc. still at this point.  For me it was anything that uploaded using MQTT and Belchertown as well as RSYNC to my internet site and things like PWS.  Leave them off for now.

6. Now test the weewx.service file.  Save your old one.  Let me repeat that. Save your existing one.  Really.

• Be a little careful here if you previously had a /etc/init.d file for v4 weewx.  If so, move that aside, saving the old one.
• copy the service file into place. daemon-reload.  systemctl start.  Verify weewx started ok with "ps -elf" and watch your logs

At this point you should have a v5 pip installation that matches your v4 setup.py installation, other than having the uploaders disabled and having bogus data consisting of historical real data and some appended simulator cruft data.  If you're happy and want to jump to v5 pip you would then:

• stop v5 weewx
• once again copy the v4 archive over into the v5 data tree
• this time you can enable your uploaders 
• go for smoke and run v5 now
• tweak your webserver setup to point to the v5 public_html directory
• verify you can surf to your v5 web
• Check your logs.  If good, go for the victory pizza.

[Tom Keffer]

If that's the information Joel is seeking, I honestly don't think it's that complicated. Instead of setting up a new station data area by using "weectl station create," just use your old one: /home/weewx.

See the detailed instructions Migrating setup.py installs to Version 5.0 in the Wiki. There's also a link in the Upgrade Guide.

TL;DR? Read the Guide!

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/6062f16b-2fbb-452d-91e4-42037cc3ebc0n%40googlegroups.com.

[Vince Skahan]

Sure.  I did my switchover before that guide existed and also used v5 pip as an opportunity to do a lot of cleanup on my pi.  Now I have everything in /home/pi and there are only two directories in there, weewx-venv and weewx-data, as well as a bin directory with a couple things I call from cron.  Nice and clean.  Previously my /home/weewx tree had become a bit of a mess so I wanted to leave that self-induced mess long long behind :-)

Joel - Tom pointed me to the second warning box at the top of the Upgrade Guide for where the pointer to the setup=>pip wiki page is.  I visually blew right past that in trying to find that link down below in the Upgrade Guide. 

[Joel Bion]

Hi -

I upgraded my setup from 4 to 5 around the time beta 15 came out. The work doing that wasn’t hard at all.

The case study I am referring to is _not_ a case study for telling how users upgrade to v5. Rather, it’s the case study of a standalone software system (weewx) migrating from the old Python setup/install/distribution paradigm (distutils? The use of /home/python) to the one Python recommends you use from 3.12 forward… with the use of venv, the split between the location of program data vs program, etc… 

Looking from the sidelines at the development of weewx 5 - it seems there were lots of steps for Tom and co. to follow, lots of work to make various user upgrade paths work, lots of details.

I am suggesting that the things you had to do might be interesting to others. Written up as a case study like a tough case in in a medical journal, it might make interesting reading. It could touch upon the adequacy of online information to inform you on how you migrated weewx to v5, on the applicability of documented conversation steps to your particular case, etc. little “gotchas” that you had to solve, etc.

Sent from my iPhone

On Nov 26, 2023, at 1:26 PM, Tom Keffer <tke...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEAVnxh7Un%3DSzbK9s0DkJpbhXWRsinEW1gXG75vm%3D3JzKA%40mail.gmail.com.

[Joel Bion]

Typo in my note… /home/python —> /home/weewx 

Sent from my iPhone

On Nov 26, 2023, at 2:48 PM, Joel Bion <jpb...@gmail.com> wrote:

Hi -

[Tom Keffer]

As we said in my old company for doomed projects, "We'll put a man on it."

Seriously though, there is the "Version 5" design document.

[Vince Skahan]

Oh easy.   The python guys upstream deprecated some things weewx (and many other consumers) have been using for over a decade and will EOL more stuff in their next version, even though many people objected to those decisions all through their equivalent of the RFC process.  The debian folks then rigidly and as-always very opinionated(ly) blocked any workarounds.  The raspi people just leverage whatever debian decided so 'their' users get hosed too. Weewx had to react to keep viable moving forward.  Being last in the food chain in such instances hurts the last guy.

And weewx 'did' want to implement a lot of changes as well for a variety of feature, architecture, and bug fix reasons.  It's a major version for sure as a result, with legacy multi-platform support making getting from A-to-Z among all the combinations of paths joyous as well.   There are a lot of pretty creative users out there.  They do all kinds of creative things leading to lots of end cases.   Upgrades up the difficulty ante.