Python version in a venv

[Dan Hinckley]

Starting weewx throws this error:

Oct 20 16:50:47 kali-linux-2024-2 systemd[1]: Started weewx.service - WeeWX weather system.
Oct 20 16:50:48 kali-linux-2024-2 python3[2820]: /home/house/weewx-venv/bin/python3: can't open file '/home/house/weewx-venv/lib/python3.12/site-packages/weewxd.py': [Errno 2] No such file or directory

Which is accurate. That venv has python 3.13

How do I cure this?

[vince]

Which 'what' is accurate ?

It sounds like your os has a different system-level python version than your pre-existing venv, perhaps from an os upgrade.  It's happened to others.

See a similar thread at https://groups.google.com/g/weewx-user/c/04WSgv2Ze7g/m/9nlZklaPAgAJ - there are multiple ways to work this one, but be sure to make certain your systemd service file (or equivalent) matches what is in the venv.  I don't know kali well enough to know how it handles startup files.

Look at your weewx-venv/bin symlinks and you will likely see that python3 resolves to whatever your os python version is, and 'not' a specific version that was there when you set up the venv.  You might be able to just fix the symlink therein to point to python3.12 (to match the venv) if you have multiple python versions on the os.  If that doesn't work you'll need to reinstall your venv contents and fix up your startup file to match.

This is not a weewx bug.  It is a (massive) pain in the neck non-feature from the python team.

[Jobu]

A bit of a tangent, but is it customary to run `activate` from a venv before using it in systematic use cases like this?

[vince]

Yes.

[Tom Keffer]

It is customary when using a venv interactively, but it's not actually necessary. As an alternative, you can use an absolute path to the Python interpreter inside the venv. That's enough for it to find its packages.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/2d424f13-0657-4366-bb69-167822a8f926n%40googlegroups.com.

[gary....@gmail.com]

I'd just like to chime in here.

I certainly do not like the pip install.

It works fine for me.

Until an OS upgrade. Then it seems the choices are either rebuild the machine and start from scratch so to speak.

Fiddle with finding every instance of the path to the missing python and edit it.

Or hard link.

Or symlink.

Or...

Yeah, I know. Don't upgrade. Not the best answer to me.

I may revert to a deb install when I rebuild the weewx machine to solve these errors after the upgrade to Trixie.

Shoot, for me, in the venv there isn't a pip and it was not possible to reinstall pip for reasons known only to the gods.

Luckily, I run on proxmox and rolled back to my previous state.

My 10 cents

[Pablo Sanchez]

Hmm, I do not have these issues with pip.

I have the weather installed on a rolling distribution. I am constantly upgrading the OS. Although as I am currently doing WeeWx development, it has been 45 days since any OS changes. I use tmux to maintain my state so I do not want to lose all these windows. :)

I recall early on,  after installing a new package via pip, I had to recreate the virtual environment. I followed the WeeWx doc.

Cheers!

---
pablo

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/31f348fa-d630-4caf-99d1-877ac006b2a3n%40googlegroups.com.

[Tom Keffer]

Yes, pip installs can be vulnerable to operating system upgrades. Then again, so can package installs --- see the recent thread on the disappearance of the function locale.format() when the system Python was upgraded to V3.12.

Still, the pip install has advantages. Because of the clean separation of code from user state, the old virtual environment can always be deleted, then recreated by following the pip install instructions.

Alternatively, one could create the virtual environment using the --copies flag:

python -m venv --copies weewx-venv

This will install copies of the Python interpreter instead of symlinking. Of course, expect it to take up a lot more space.

Still another alternative is to use Docker.

-tk

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/31f348fa-d630-4caf-99d1-877ac006b2a3n%40googlegroups.com.

[vince]

If pip bothers you, don't use pip.  Use the packaged versions.  That's why they're there...

Long TL;DR; follows.....

Some thoughts on risk mitigation from a too many decades to admit retired sysadmin and custom linux os developer at $work....

1. os upgrades can and do break things on every os I've ever seen

• 'you' control whether your os updates things
• you also control 'which' things you will permit to be updatable
  
  • you can pin packages to 'not' update if you are so inclined
    • for debian-like (apt-mark)
    • for redhat-like (there are several ways to do this)

2. if you understand your risks, you can choose to simply not update your os

• yes I know others will differ in opinion here, everybody has a different risk decision to make
• FWIW -  I never update 'anything' on my home LAN-only weewx system.  Never.
  • why ? Because I only upload outbound to Internet sites
  • and my gateway blocks all incoming traffic from Internet

• just mentioning it above, although others can/will differ in opinion, so I don't want to start a flame war on that one
  • ultimately, the os upgrade setup is the admin's choice

3. for me, I run pip for a couple reasons

• packaged installs tend, for me, to be less stable/predictable in what they will upgrade, sometimes os by os
  • packaging is hard.  Real hard.  And real painful for the developer.  (many thanks to matthew there)

• for pip I know exactly what it'll do, and it has been 100% rock-solid for me in literally dozens of installs

4. creating your venv slightly differently to select a specific python version can help a little

• "python3 -m venv" expects the os default python version to never change
• but "python3.11 -m venv" will use a specific python version available on the system

• so in the latter case, unless the python3.11 packages get deleted, your venv will be fine
• and if an os upgrade (bookworm=>trixie) 'did' delete that version, just add it back in ('apt install python3.11')

5. reinstalling via pip is not 'that' painful really if you do a 'little' up front work even if you've added a lot of python libraries

• on a working system save your venv python setup to a file
  • activate your venv
    • source ~/weewx-venv/bin/activate
  • save a listing of the libs and their versions to a file in a persistent location (weew-data)
    • pip3 freeze > ~/weewx-data/requirements.txt

• if you ever have to rebuild your venv from scratch
  • create a new venv
    • python3 -m venv weewx-venv
  • activate your new venv
    • source weewx-data/bin/activate
  • use the saved requirements.txt file to reinstall your libs
    • pip3 install -r ~/weewx-data/requirements.txt
  • and you'll have a new complete weewx-venv with all the libs you had before in the old version

Personally - I kinda like item (5) above a lot.  

• stash a requirements.txt file that itemizes your setup
• use it if you ever 'do' need to recreate a previously working setup
• all without needing to remember what you added months or years ago
• quick test here in debian took under 20 seconds end-to-end to rebuild.  Really.

[cric...@pobox.com]

As an alternative, how about the python gods quit this ongoing assault on backwards compatibility.

Just saying...

The whole venv seems to be a work-around to the disconnect between OS-based python versions not advancing

and Python having this obsession with breaking things over minor revisions.

Wrong place, wrong time.  Sorry.

Chris 

[Tom Keffer]

I've already offered my opinion on the matter: https://groups.google.com/g/weewx-user/c/XmKEPTN1L_g/m/-umQKZCWCQAJ

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/bba47d65-5c93-4ad0-bbab-db16489fafdan%40googlegroups.com.

[vince]

On Tuesday, October 21, 2025 at 7:29:57 AM UTC-7 Tom Keffer wrote:

Alternatively, one could create the virtual environment using the --copies flag:

python -m venv --copies weewx-venv

This will install copies of the Python interpreter instead of symlinking. Of course, expect it to take up a lot more space.

Just a data point FWIW - this adds about 20MB to the size of the venv, which isn't bad at all.

 

[gary....@gmail.com]

Just to complete my thoughts.

Yep, I understand that OS upgrades are under my control. I want to upgrade, why wouldn't I want the enhancements and security of a new release?

So, the "Don't upgrade" answer isn't valid.

Pin packages to a specific version.

Again, are these newer releases done just to give the devs something to do? Or are there various fixes and enhancements? When would I wish to unpin a package?

Creating a venv using a specific python version sounds like an additional step to get where I am now. The OS version changed. So, then I'd try to install an additional version of python using apt.

No, thanks, I'm not looking for additional things to dink with.

Having a list of the modules in use is great and will save time going forward.

I kinda like the idea of --copies

I don't mind the extra space needed if I avoid the mess I had trying to upgrade to Trixie.

I'm going to create a new install and keep weewx separate from my other website. Until now, I ran them on the same machine for laziness/simplicity sake. SSL certs, mosquitto, DDNS, proxy. I believe weewx now warrants it's own virtual machine.

[vince]

On Wednesday, October 22, 2025 at 10:42:03 AM UTC-7 gary....@gmail.com wrote:

Yep, I understand that OS upgrades are under my control. I want to upgrade, why wouldn't I want the enhancements and security of a new release?

So, the "Don't upgrade" answer isn't valid.

Sure it is.  If a new os release has enhancements you don't need and/or fixes for bugs you are not experiencing, you are just making work for yourself touching it.  If it's working, why mess with it at all ?  I treat my weewx setup basically as an appliance like a toaster.  Turn it on and let it be.

 

Pin packages to a specific version.

Again, are these newer releases done just to give the devs something to do? Or are there various fixes and enhancements? When would I wish to unpin a package?

You would unpin when a new version has a security fix or bug fix or feature addition you need.  This is actually rather rare.

 

Creating a venv using a specific python version sounds like an additional step to get where I am now. The OS version changed. So, then I'd try to install an additional version of python using apt.

No, thanks, I'm not looking for additional things to dink with.

You're not 'dinking with' anything.  You're simply adding back in one older version that you relied on before.  You never touch it afterward. "apt install python3.11" is one command to run to fix the os deleting it on you.  Doesn't get easier than that.

 

Having a list of the modules in use is great and will save time going forward.

I kinda like the idea of --copies

I don't mind the extra space needed if I avoid the mess I had trying to upgrade to Trixie.

You're not 1000% safe from upgrade misadventures but you're probably as close as you can get.

 

I'm going to create a new install and keep weewx separate from my other website. Until now, I ran them on the same machine for laziness/simplicity sake. SSL certs, mosquitto, DDNS, proxy. I believe weewx now warrants it's own virtual machine.

Sure - but I have to ask.  A virtual machine is just another way of doing a full os.  The same questions exist.  Are you going to set automatic updates on for the virtual machine ?  You have the same decisions to make for pain-vs-gain and risk-vs-benefit.  Same thing if you go docker.  The questions don't go away nor does your individual measure of worth it or not.  You're just making a pretty simple thing potentially more complicated unless you have other reasons to go VM or docker.  Everybody's worth-it vs not measure is different.  There are no wrong answers.

FWIW - I spend the $4/month to have a tiny AWS lightsail VM to rsync out to, so I don't need to sweat proxying, certs, firewalls, and all that complexity.  That minimal VM is nginx only.  Yes since it's on Internet it is set to auto-update just in case but it's so minimal and locked down that's likely not needed either for me

The way I look at it is 'what is the value of your time+labor'.  For me it's $50/year well spent to not need to worry about my LAN linux gear at all.  But there are no wrong answers either way.

 

[gary....@gmail.com]

 I guess we'll disagree on the OS update question. I'm firmly in the Upgrade camp.

On python/venv have you seen/used/thought about this?

https://github.com/astral-sh/uv

[gary....@gmail.com]

 

I'm going to create a new install and keep weewx separate from my other website. Until now, I ran them on the same machine for laziness/simplicity sake. SSL certs, mosquitto, DDNS, proxy. I believe weewx now warrants it's own virtual machine.

Sure - but I have to ask.  A virtual machine is just another way of doing a full os.  The same questions exist.  Are you going to set automatic updates on for the virtual machine ?  You have the same decisions to make for pain-vs-gain and risk-vs-benefit.  Same thing if you go docker.  The questions don't go away nor does your individual measure of worth it or not.  You're just making a pretty simple thing potentially more complicated unless you have other reasons to go VM or docker.  Everybody's worth-it vs not measure is different.  There are no wrong answers.

The benefit of this is that it takes under a minute to rollback to the previous install on proxmox. Backup, do whatever, fail, restore.