V5.0 release candidate available

[Tom Keffer]

Celebrate the solstice (coming up in 4 hours) and the start of Winter with a WeeWX release candidate!

Changes since b17

We gave up on a standalone logger and have gone back to whatever your system uses. This is more robust to permission problems, but may make it harder to get the logs back out. See the wiki article View logs for tips on how to convince your system to let you have a look.

As always, you can add a [Logging] section to weewx.conf and set up your own rotating log system. See the article WeeWX V4 and logging for how to do this.

Pip

For pip installs, please delete your old virtual environment, then install from scratch by following the pip install instructions. While upgrading should work, we are particularly interested in the experience of a new install, including setting up a daemon and udev files. Make sure to follow the new instructions that use a daemon setup script.

Debian

For Debian package installs, modify /etc/apt/sources.list as follows:

echo "deb [arch=all] https://weewx.com/apt-test/python3 buster main" | sudo tee /etc/apt/sources.list.d/weewx.list

Note the "apt-test". This tells apt to look there for the beta release, instead of the normal repository. You may want to change it back when you're done.

Red Hat

For Red Hat package installs, put this in /etc/yum.repos.d/weewx.repo

[weewx]
name=weewx
baseurl=http://weewx.com/yum-test/weewx/el9
enabled=1

Note the "yum-test". This tells yum to look there for the beta release, instead of the normal repository. Again, you may want to change things back when you're done.

What we're looking for

1. We are interested in your upgrade experience. Did the installers get your configuration file right? Skins? Is the daemon configured correctly?

2. With this release, we have included udev rules to set the correct permissions for devices. Did it work? If not, did unplugging then replugging the device work?

3. How about logging? Is it going to your system logger? Do the labels look reasonable?

Thanks!

-Tom & Matt

[Al Barnes]

First off, many thanks to all involved in developing weewx V5! After
reading the git logs for the past few months I can see that this has
taken a ton of time and effort.

I've finished a pip install and here are my findings.

I'm running Raspberry Pi OS (Debian Bookworm) on a pi 3b that has been
used for testing of the alpha versions of weewx. The station is a
Vantage Pro2 with the official Davis logger.

After deleting weewx-data and weewx-venv and a weewx.service file in
/etc I followed the instructions at
https://weewx.com/docs/5.0/quickstarts/pip/ and everything went as
expected. I created the station with 'weectl station create'. The
command 'sudo sh ~/weewx-data/scripts/setup-daemon.systemd' set up
systemd logging.

After this point I made no changes whatsoever to weewx.conf as I wanted
to see how it would run with the default settings. Before starting
weewx, I ran 'journalctl -f -u weewx in another terminal and first
received a message that I've never seen before:

/etc/systemd/system/weewx.service:15: Standard output type
syslog+console is obsolete, automatically updating to journal+console.
Please update your unit file, and consider removing the setting altogether.

After starting weewx the logging was normal.

As I had not copied over my old database, weewx faithfully starting
downloading records from the Vantage Pro2 starting with 2023-12-12
22:35:00 PST.

It can take quite a while to download almost 9 days of data from the
logger, and here is where I ran in to a serious problem. The downloading
of records stopped at 2023-12-21 19:50:00 when the actual time was
20:38:44. At this point the next line in the logs was 'Starting main
packet loop.'
Shortly thereafter it added records at 20:38:00 and 20:40:00, so we are
now missing all records from 19:50 through 20:35.
A minute later I received the dreaded 'Expected to read 99 chars; got 0
instead' message, followed by 'Main loop exiting. Shutting engine down'.

Weewx restarted on it's own and seems to be downloading a new archive
record every five minutes, but of course we still have the problem of
the missing records. I had the same problem in early version 5 alpha
releases when weewx was downloading many records and couldn't complete
everything within the five minute cycle. Not a problem if you are
copying over your old database as you are most likely only downloading
an hour or two of records from the logger, but this could bite someone
who's station has been down for a long time due to a power outage or
whatever. I've attached the log file so that you can see the details.

Anyhow, hopefully this is helpful and thanks again for all of the hard
work that has been done to come up with a new version of weewx!

Al

[Hartmut Schweidler]

Guten Morgen,

Zunächst einmal vielen Dank an alle, die an der Entwicklung von weewx V5 beteiligt waren!

Mein Installation erfolgte per "apt upgrade". auf einem Banana Pi + 1TB HDD

Die folgenden Pakete werden aktualisiert (Upgrade):
  weewx
1 aktualisiert, 0 neu installiert, 0 zu entfernen und 2 nicht aktualisiert.
Es müssen 2.278 kB an Archiven heruntergeladen werden.
Nach dieser Operation werden 2.048 B Plattenplatz zusätzlich benutzt.
Holen:1 https://weewx.com/apt-test/python3 buster/main all weewx all 5.0.0rc1-1 [2.278 kB]
Es wurden 2.278 kB in 2 s geholt (948 kB/s).

Preconfiguring packages ...
(Lese Datenbank ... 75361 Dateien und Verzeichnisse sind derzeit installiert.)
Vorbereitung zum Entpacken von .../weewx_5.0.0rc1-1_all.deb ...
Entpacken von weewx (5.0.0rc1-1) über (5.0.0b17-4) ...
weewx (5.0.0rc1-1) wird eingerichtet ...
Neue Version der Konfigurationsdatei /etc/weewx/skins/Ftp/skin.conf wird installiert ...
Neue Version der Konfigurationsdatei /etc/weewx/skins/Mobile/skin.conf wird installiert ...
Neue Version der Konfigurationsdatei /etc/weewx/skins/Rsync/skin.conf wird installiert ...

Konfigurationsdatei »/etc/weewx/skins/Seasons/skin.conf«
 ==> Geändert (von Ihnen oder von einem Skript) seit der Installation.
 ==> Paketverteiler hat eine aktualisierte Version herausgegeben.
   Wie möchten Sie vorgehen? Ihre Wahlmöglichkeiten sind:
    Y oder I : Die Version des Paket-Betreuers installieren
    N oder O : Die momentan installierte Version beibehalten
       D     : Die Unterschiede zwischen den Versionen anzeigen
       Z     : Eine Shell starten, um die Situation zu begutachten
 Der Standardweg ist das Beibehalten der momentanen Version.
*** skin.conf (Y/I/N/O/D/Z) [Vorgabe=N] ? n

...

Neue Version der Konfigurationsdatei /etc/weewx/weewx.conf.dist wird installiert ...
Copying previous config file to /etc/weewx/weewx.conf-5.0.0b18-5.0.0rc1
Saving distribution config file as /etc/weewx/weewx.conf-5.0.0rc1
Creating maintainer config as /etc/weewx/weewx.conf-5.0.0b18-5.0.0rc1
Using configuration file /etc/weewx/weewx.conf-5.0.0b18-5.0.0rc1
Finished upgrading configuration file /etc/weewx/weewx.conf-5.0.0b18-5.0.0rc1
Saving configuration file /etc/weewx/weewx.conf-5.0.0b18-5.0.0rc1
Created symlink /etc/systemd/system/multi-user.target.wants/weewx.service → /lib/systemd/system/weewx.service.

root@ba002:/etc/weewx# systemctl restart weewx@weewx
Warning: The unit file, source configuration file or drop-ins of we...@weewx.service changed on disk. Run 'systemctl daemon-reload' to reload units.
root@ba002:/etc/weewx# systemctl daemon-reload
root@ba002:/etc/weewx# systemctl restart weewx@weewx
root@ba002:/etc/weewx# systemctl restart weewx@weeusb

Meine Anpassungen:

Anpassungen nur in weewx@.service

"
# systemd service template file for running multiple instances of weewxd
#
# Each instance XXX must have its own config, database, and HTML_ROOT:
#
#  item            name                           where to specify
#  --------        -----------------------------  ----------------------------
#  config          /etc/weewx/XXX.conf            configuration directory
#  database_name   /var/lib/weewx/XXX.sdb         specified in XXX.conf
#  HTML_ROOT       /var/www/html/XXX              specified in XXX.conf

[Unit]
Description=WeeWX %i
Documentation=https://weewx.com/docs
Requires=time-sync.target
After=time-sync.target
PartOf=weewx.service

[Service]
ExecStart=weewxd --log-label weewx5-%i /etc/weewx/%i.conf
StandardOutput=null
StandardError=journal+console
#User=weewx
#Group=weewx

[Install]
WantedBy=multi-user.target

"

Es läuft hervorragend.

2023-12-22T07:00:24.155271+01:00 ba002 weewx5-weewx[5037]: INFO weewx.manager: Added record 2023-12-22 07:00:24 CET (1703224824) to database 'weewxDavis'
2023-12-22T07:00:24.393038+01:00 ba002 weewx5-weewx[5037]: INFO weewx.manager: Added record 2023-12-22 07:00:24 CET (1703224824) to daily summary in 'weewxDavis'
2023-12-22T07:00:25.032855+01:00 ba002 weewx5-weewx[5037]: INFO weewx.manager: Added record 2023-12-22 07:00:00 CET (1703224800) to database 'weewxGW2000'
2023-12-22T07:00:25.642882+01:00 ba002 weewx5-weewx[5037]: INFO weewx.manager: Added record 2023-12-22 07:00:00 CET (1703224800) to daily summary in 'weewxGW2000'
2023-12-22T07:00:34.314116+01:00 ba002 weewx5-weewx[5037]: INFO weewx.cheetahgenerator: Generated 9 files for report SeasonsReport in 8.51 seconds
2023-12-22T07:00:45.239521+01:00 ba002 weewx5-weewx[5037]: INFO weewx.imagegenerator: Generated 24 images for report SeasonsReport in 10.92 seconds
2023-12-22T07:00:45.241651+01:00 ba002 weewx5-weewx[5037]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx
2023-12-22T07:00:48.646048+01:00 ba002 weewx5-weeusb[4799]: INFO weewx.drivers.fousb: synchronising to the weather station (quality=1)
2023-12-22T07:01:34.532773+01:00 ba002 weewx5-weeusb[4799]: INFO weewx.manager: Added record 2023-12-22 06:57:00 CET (1703224620) to database 'weewx.sdb'
2023-12-22T07:01:34.687560+01:00 ba002 weewx5-weeusb[4799]: INFO weewx.manager: Added record 2023-12-22 06:57:00 CET (1703224620) to daily summary in 'weewx.sdb'
2023-12-22T07:01:41.011145+01:00 ba002 weewx5-weeusb[4799]: INFO weewx.cheetahgenerator: Generated 8 files for report StandardReport in 5.96 seconds
2023-12-22T07:01:43.778354+01:00 ba002 weewx5-weeusb[4799]: INFO weewx.imagegenerator: Generated 13 images for report StandardReport in 2.76 seconds
2023-12-22T07:01:43.784111+01:00 ba002 weewx5-weeusb[4799]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/wx2013

Noch einmal Danke und ein gesundes Weihnachtsfest, verbunden mit einem Guten Rutsch in Jahr 2024

Hartmut

[Greg]

I installed weewx using the pip method and followed the instructions.

I did a pip list before installing to get the list of what other dependencies I am using.

 When I installed paho-mqtt and pyephem I got these errors:

Installing collected packages: paho-mqtt
  DEPRECATION: paho-mqtt is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for paho-mqtt ... done
Successfully installed paho-mqtt-1.6.1

/opt/weewx$ python3 -m pip install pyephem
Collecting pyephem
  Using cached pyephem-9.99.tar.gz (1.4 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: ephem in ./weewx-venv/lib/python3.11/site-packages (from pyephem) (4.1.5)
Installing collected packages: pyephem
  DEPRECATION: pyephem is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

I am currently running it in simulation mode just to see what happens.

On my real system I have installed all my environment and data under /opt

eg /opt/weewx/weewx-venv and /opt/weewx/weewx-data

Other than that it seems to run.

My question is should I run the install of those packages above that had the error with the --use-pep517 option? I read the github information that was on the link contained in the error message but it made no sense to me.

My pip version is: 3.11

[Tom Keffer]

Thanks, all! Keep them coming. 

1. The syslog comment is annoying, but harmless. We've changed the weewx unit service file to specify StandardError of journal+console. Commit 940eff4.

2. Missing Vantage records. I've noticed variants of this problem when it takes a long time to download records from the console. My best is that while the console is busy emitting historic records, it neglects to create new ones. Hence, you miss a few records. I don't think it has anything to do with V5, as the driver code hasn't changed much.

3. Greg, I would guess that your problems are due to using a very old version of pip. You're at 3.11, but the current version is 23.3. See the wiki article Troubleshooting pip installs and see if that helps.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/3f813b17-0b9b-49b4-b9af-97dcd0a4f69bn%40googlegroups.com.

[Paul R Anderson]

Extremely nitpicky, this could be just me :)

On startup __init__.py now logs this:

Dec 21 06:47:19 hall-9000 weewxd[4484]: INFO weewx: Adding 'user' directory '/home/panders/weewx-data/bin'

First time I saw it I thought wait that directory should exist, it's where my extensions live, hope they're OK. Then I realized it's just adding the dir to PYTHONPATH

Perhaps change  __init__.py logging line to slightly more descriptive

log.info("Adding 'user' directory '%s' to PYTHONPATH" % lib_dir)

So it produces:

Dec 22 07:52:36 hall-9000 weewxd[508]: INFO weewx: Adding 'user' directory '/home/panders/weewx-data/bin' to PYTHONPATH

Paul

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zECFSEWf9immM3fFroWU79GcYyT65ibe8w1hWd-XJ92s1g%40mail.gmail.com.

[matthew wall]

On Friday, December 22, 2023 at 8:31:47 AM UTC-5 Paul R Anderson wrote:

On startup __init__.py now logs this:

Dec 21 06:47:19 hall-9000 weewxd[4484]: INFO weewx: Adding 'user' directory '/home/panders/weewx-data/bin'

First time I saw it I thought wait that directory should exist, it's where my extensions live, hope they're OK. Then I realized it's just adding the dir to PYTHONPATH

 

addressed in commit <a href="e402cd4">https://github.com/weewx/weewx/commit/e402cd4dee6fb9db397b224c4c05188b884c8d69</a>

[Steeple Ian]

Installed the release candidate using pip. All working fine using simulator driver. Then tried to install Gary's weewx-gw1000.py driver and repeatedly get this error: -

(weewx-venv) Ian@bookworm:~$ weectl extension install https://github.com/gjr80/weewx-gw1000/releases/download/v0.6.0b2/gw1000-0.6.0b2.tar.gz
Using configuration file /home/Ian/weewx-data/weewx.conf
Request to install 'https://github.com/gjr80/weewx-gw1000/releases/download/v0.6.0b2/gw1000-0.6.0b2.tar.gz'.
Extracting from tar archive /tmp/tmpdoa3q052
Traceback (most recent call last):
  File "/home/Ian/weewx-venv/bin/weectl", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/weectl.py", line 66, in main
    namespace.func(namespace)
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py", line 96, in dispatch
    namespace.action_func(config_dict, namespace)
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/weectllib/extension_cmd.py", line 116, in install_extension
    ext.install_extension(namespace.source)
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py", line 114, in install_extension
    extension_name = self._install_from_file(test_fd.name, info.get_content_subtype())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py", line 154, in _install_from_file
    extension_name = self.install_from_dir(extension_dir)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py", line 165, in install_from_dir
    installer_path, installer = weecfg.get_extension_installer(extension_dir)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/weecfg/__init__.py", line 738, in get_extension_installer
    installer = loader()
                ^^^^^^^^
  File "/tmp/tmpekv6izgc/gw1000/install.py", line 377, in loader
  File "/tmp/tmpekv6izgc/gw1000/install.py", line 382, in __init__
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/setuptools/_distutils/version.py", line 54, in __init__
    self.parse(vstring)
  File "/home/Ian/weewx-venv/lib/python3.11/site-packages/setuptools/_distutils/version.py", line 157, in parse
    raise ValueError("invalid version number '%s'" % vstring)
ValueError: invalid version number '5.0.0rc1'
(weewx-venv) Ian@bookworm:~$

I guess the second last line is the key one. Being trying to dig my way out of this one without success. I never had a problem with any of the other recent builds of version 5.

Any help would be appreciated,

Thanks,

Ian

[matthew wall]

I guess the second last line is the key one. Being trying to dig my way out of this one without success. I never had a problem with any of the other recent builds of version 5.

Any help would be appreciated,

hi ian,

we just discovered this as well.  the 'StrictVersion' parser does not handle 'rc' correctly.  here are some details:

 https://peps.python.org/pep-0386/

as a workaround, you can extract the gw1000 tarball/zip, modify the install.py, then use 'weectl extension' to load from the directory instead of directly from the url.

in install.py, replace the StrictVersion conditional at line 382 with a simple "if False:"

m

[Steeple Ian]

Thanks Matthew, I have a fork of gw1000 so I can do it directly from there.

Ian

[Steeple Ian]

Can confirm that worked,

Thanks,

Ian

[Vince Skahan]

On Friday, December 22, 2023 at 5:09:15 AM UTC-8 Tom Keffer wrote:

3. Greg, I would guess that your problems are due to using a very old version of pip. You're at 3.11, but the current version is 23.3. See the wiki article Troubleshooting pip installs and see if that helps.

On Fri, Dec 22, 2023 at 12:45 AM Greg <ubea...@gmail.com> wrote:

I installed weewx using the pip method and followed the instructions.

I did a pip list before installing to get the list of what other dependencies I am using.

 When I installed paho-mqtt and pyephem I got these errors:

Installing collected packages: paho-mqtt
  DEPRECATION: paho-mqtt is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for paho-mqtt ... done
Successfully installed paho-mqtt-1.6.1 

 

My question is should I run the install of those packages above that had the error with the --use-pep517 option? I read the github information that was on the link contained in the error message but it made no sense to me.

I see these too on debian12 with pip 23.0.1 for those two packages...

• paho-mqtt has an open issue on this but they seem to be thrashing among the devs a bit working it - https://github.com/eclipse/paho.mqtt.python/issues/706
  
• I couldn't find an issue on pyephem's github so I opened one there - https://github.com/brandon-rhodes/pyephem/issues/268

The warnings do go away if you add the switch ala:

(weewx-venv) vagrant@deb12:~$ pip install --use-pep517 paho-mqtt
Collecting paho-mqtt
  Using cached paho-mqtt-1.6.1.tar.gz (99 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: paho-mqtt
  Building wheel for paho-mqtt (pyproject.toml) ... done
  Created wheel for paho-mqtt: filename=paho_mqtt-1.6.1-py3-none-any.whl size=62120 sha256=f329e400c6f3932151a07b46b5f098cd025392b454e94d874d48d7cc6f2c2d91
  Stored in directory: /home/vagrant/.cache/pip/wheels/29/ea/a5/ba9a63aaf4cd4e16e8a87ee31fb4d11b04ff5e1735d312619a
Successfully built paho-mqtt
Installing collected packages: paho-mqtt
Successfully installed paho-mqtt-1.6.1

 

[Vince Skahan]

I set up a couple instances via weewx-multi following https://github.com/weewx/weewx/wiki/weewx-multi and it looks like the generated weewx@.service file needs a little tweak.

My pip installation generated a weewx@.service file that had ExecStart pointing to weewx.conf, but it should point to %i.conf so the instance name is used.  After making that change multi works just amazingly well.  So easy.

[Claudio]

does it work on Raspberry 5 and Debian version: 12 (bookworm)?
Claudio

[Vince Skahan]

yes

[Greg]

Oops...I meant python version is 3.11 not pip. Pip is at the latest version.

So I guess I install this way  pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517'?

I am running debian 12 by the way. Anyone else seen these messages?

[Tom Keffer]

I was able to install both paho-mqtt and ephem (NB: these days, it's just "ephem", not "pyephem") without warnings on Debian 12 by installing "wheel" first:

python3 -m pip install wheel

python3 -m pip install ephem

python3 -m pip install paho-mqtt

Does this work for others? If so, we can put "wheel" in the requirements list for pip installs.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/064a168f-7dd7-4c4d-9b2c-88d83926740fn%40googlegroups.com.

[Vince Skahan]

Works for me on deb12 vagrant and pi5 deb12 based raspios

[Greg]

Yes having install wheel first removes all my errors. I think it should be installed by default to save any questions from people like me who don't really know how this all works and what depends on what. :)

Because I was already running the beta V5 I just renamed my venv and recreated it so my weewx data would work and also if it didn't work I could rename it back and all would be good. (in theory) I have installed in /opt/weewx

All looks good so far.

https://weather.ubeaut.work/

Thanks for the support.

[Tom Keffer]

Hmmm, that didn't work. If you list "wheel" as a dependency, pip isn't smart enough to install it first before attempting the others. So, you get the same error. I'm reluctant to create yet another step in the install process.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/c0de7524-70ff-4168-bd97-b5736243ead5n%40googlegroups.com.

[Tom Keffer]

After a bit more investigation, it appears that some systems install python3-wheel when they install python3-pip, some don't. Hence the confusing results.

[Vince Skahan]

Tom - your xaggs extension isn't installing.  I see the identical issue with one of my custom extensions too.

(weewx-venv) vagrant@deb12:~/adds$ weectl extension install --config=/home/vagrant/weewx-data/simulator.conf weewx-xaggs-master/
Using configuration file /home/vagrant/weewx-data/simulator.conf
Request to install 'weewx-xaggs-master/'.

Traceback (most recent call last):

  File "/home/vagrant/weewx-venv/bin/weectl", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectl.py", line 66, in main
    namespace.func(namespace)
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py", line 96, in dispatch
    namespace.action_func(config_dict, namespace)
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/extension_cmd.py", line 116, in install_extension
    ext.install_extension(namespace.source)
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py", line 125, in install_extension
    extension_name = self.install_from_dir(extension_path)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weecfg/extension.py", line 201, in install_from_dir
    save_config |= self._inject_config(installer['config'], extension_name)
                                       ~~~~~~~~~^^^^^^^^^^
KeyError: 'config'

(weewx-venv) vagrant@deb12:~/adds$ pip3 list
Package            Version
------------------ ----------
certifi            2023.11.17
charset-normalizer 3.3.2
configobj          5.0.8
CT3                3.3.3
ephem              4.1.5
idna               3.6
paho-mqtt          1.6.1
Pillow             10.1.0
pip                23.0.1
pyephem            9.99
PyMySQL            1.1.0
pyserial           3.5
pyusb              1.2.1
requests           2.31.0
setuptools         66.1.1
six                1.16.0
urllib3            2.1.0
weewx              5.0.0rc1
wheel              0.42.0

[Tom Keffer]

Thanks for spotting that, Vince!

Fixed in commit 256cac5.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/73f123da-7608-46d5-add0-a2b9b870d1ffn%40googlegroups.com.

[Vince Skahan]

I hand-edited the diff into weewx-venv/lib/python3.11/site-packages/weecfg/extension.py which worked for installing the extensions, but I get a similar message trying to run reports manually....

You might try this on a current fully patched up version to see if it works with all the updates you made after release to pypi just in case...

(weewx-venv) vagrant@deb12:~/weewx-data$ weectl report run --config=simulator.conf
Using configuration file simulator.conf
All enabled reports will be run.
Generating as of last timestamp in the database.

Traceback (most recent call last):
  File "/home/vagrant/weewx-venv/bin/weectl", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectl.py", line 66, in main
    namespace.func(namespace)
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/__init__.py", line 96, in dispatch
    namespace.action_func(config_dict, namespace)

  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/report_cmd.py", line 92, in run_reports
    weectllib.report_actions.run_reports(config_dict,
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weectllib/report_actions.py", line 84, in run_reports
    engine = weewx.engine.DummyEngine(config_dict)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", line 89, in __init__
    self.loadServices(config_dict)
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weewx/engine.py", line 157, in loadServices
    obj = weeutil.weeutil.get_object(svc)(self, config_dict)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/weewx-venv/lib/python3.11/site-packages/weeutil/weeutil.py", line 1404, in get_object
    module = importlib.import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1142, in _find_and_load_unlocked

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> Hmmm, that didn't work. If you list "wheel" as a dependency, pip isn't
> smart enough to install it first before attempting the others. So, you get
> the same error.

Is it the case that new enough pip installs it? I just did

$ virtualenv-3.11 FOO
created virtual environment CPython3.11.7.final.0-64 in 451ms
creator CPython3Posix(dest=/home/gdt/FOO, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/gdt/.local/share/virtualenv)
added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.41.3
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ virtualenv-3.11 --version
virtualenv 20.25.0 from /usr/pkg/lib/python3.11/site-packages/virtualenv/__init__.py

so it may be that people with older virtualenv are having trouble?

> I'm reluctant to create yet another step in the install process.

I'd say you aren't creating it, you are describing something that the
python world thinks is a feature!

[Tom Keffer]

Hmm, "weectl report run" works perfectly on my system, using Python 3.11.

It's not clear from your stack trace what raised the exception. Can you dig deeper?

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/756a05b3-d693-41f5-9d59-411364316872n%40googlegroups.com.

[Vince Skahan]

It worked with the --config switch specified ?

If so, let me know (email is fine) and I can try to dig deeper tomorrow....

[Tom Keffer]

On Fri, Dec 22, 2023 at 5:46 PM Vince Skahan <vince...@gmail.com> wrote:

It worked with the --config switch specified ?

Yes. I tried macOS, Debian 12, Ubuntu 23.04, with a mix of Python 3.7 and 3.11 --- all worked with the --config switch.

[Vince Skahan]

Thanks.   I can't recreate yesterday's boom but I did find one bug.

In weewx@.service, the .conf file it uses for the exec line should say %i.conf (the config file for the instance being started)

[matthew wall]

On Saturday, December 23, 2023 at 1:09:50 PM UTC-5 Vince Skahan wrote:

In weewx@.service, the .conf file it uses for the exec line should say %i.conf (the config file for the instance being started)

this has been addressed in commit 33f18e8

[jpb...@gmail.com]

Install of 5.0.0rc1 worked just fine for me. I deleted my weewx-venv tree, and followed the instructions. No errors.

[Cameron D]

I tried an upgrade from a 4.10.2 simulator system just to see what happened. It looked OK, so I tried on my main system.

• Debian 12, using the apt package upgrade.
• I have two hardware units - Oregon WMR300 and EcoWitt 1000 (v0.6.0b2), both with custom databases and the gw1000 has remapped names.
• I have been running as non-root, but not user  or group named "weewx"
• I have been running the systems via systemd-multi (for just on 12 months).

I accept many of my problems relate to my non-standard systems and so are of my own making, but here goes...

User name:

I had WEEWX_USER defined in /etc/default/weewx  to be "weather" because that user is also attached to other software unrelated to weewx.  I was slightly surprised that the installer did not check, but that's not a big deal, bacause I doubt there are many in my situation. I modified my operation to use user 'weewx', who becomes a member of group 'weather'.  The installation process  changed ownership of  files under  /etc/weewx, but did not touch anything under HTML_ROOT, so reports failed from then on.

My only suggestion here is it would be good if you mention in the upgrade notes that the ownership change will happen unconditionally and that if people do have weewx running under a non-root user of any other name then convert first.

Udev rules:

The upgrade notes state that the example udev rules allow for group ownership. My installation did not get any rules with owner or group changes in them. The example rules just  give read-write permission to the world.

The sample rules under /etc/weewx/udev are mainly dated 2018 for individual devices, a vantage rule is dated Nov 2021 and an all-in-one "weewx.rules" dated Jan 2021.  None of them sets owner or group.

On my test system, the sample udev rules are all dated Feb 2023, which I guess is from v4.10.2

Reports not completing:

Both systems report via customised Seasons skins, and they appear to be giving reasonable updates to the reports. The WMR300 merges data from both databases and has a separate issue that I will report in detail later. The gw100 seems to give updated reports, including sensible plots and current conditions every 2 minutes.

They sample at 1 minute intervals and  every 2 minutes both systems independently report:

INFO weewx.engine: Launch of report thread aborted: existing report thread still running

Both instances are each consuming 60 to 100% cpu permanently.

When I stop (allow it to die) and restart, there is not much cpu activity for the first minute, so I would guess it is tied into thje report generation.

I'll have a go with the default skin...

[Cameron D]

A bit more info on the reports.

1. I disabled any reports from the GW1000 and cpu usage dropped to zero.
2. I set the WMR300 to just use the Seasons skin as-shipped, but there was no improvement.
3. I changed wmr300 config to write to a new empty directory (with suitable ownership and permissions) and things got much worse:

• NOAA dir was created and populated.
• A set of PNG files and inxex.html were written as expected.
• the reports now timeout every minute, rather than every 2 minutes.
• various support files are never copied across - seasons.css and js; background, icons, font, lang folders do not exist (not sure whether some are hangovers from prev versions.)
• Most files are not updated when expected.

 For example, I deleted all day*.png files and, at the next update, noted the following:

• the barometer and tempdew were the only images to reappear. 
•  rss.xml was updated
• html files celestial, tabular and telemetry were 1 minute old
• statistics.html was 2 minutes old
• index.html was 4 min old
• NOAA had not been updated for 7 min
• other files were older - css and js files were dated from when I copied them over manually.
• the files that do get updated within a report period are often dated within 0.1 seconds of each other, but sometimes are seconds apart.

[Cameron D]

I got basically this same result as Vince, trying to run a report manually.

Running as user weewx or root gave same result.

I have no venv, and no pip install.

A pointer to the cause would be that renaming /usr/share/weewx/user-2023xxx  back to user fixed the problem.

I also tried a symlink from /usr/share/weewx/user pointing to /etc/weewx/bin/user and that worked also.  Including being able to run as user weewx.

And got me back to my other problem, that the report took a long time to run - 1 minute and 30 seconds to generate the full list. I'll report on that separately.

[Cameron D]

Back to the reports themselves.

I am running the GW1000 system without any reports, and will probably end up running cron jobs to occasionally update the reports.

• I created an empty folder and pointed the config file there.
• This is a heavily customised Seasons report, because it is basically indoor air/living quality only.
• I ran    weectl report run SeasonsReport  --config=/etc/weewx/ewitt.conf
• then  ls -lrt --full-time to show the increment in file creation times.
• my customised report has 5 daily, 4 weekly, 3 monthly and 5 yearly plots, so not a particular burdon.

The results show the report  files are generated in short bursts, then a gap, then a few more, etc.  The sequence is...

• start at 2:14.6 seconds
• 14.8 s NOAA folder populated by 14.8 seconds
• 2.5 s later index.html, followed statistics
• 0.9 s gap to  telemetry, tabular &  celestial, which were within 0.02 s
• 40 s gap to rss.xml ,
• 0.04 sec gap to the start of the PNG files daybarom and daytempdew
• 13s gap to dayhumidity, dayPM, dayco2, weekbarom and weektempdew (all within 0.25s)
• 21s gap to weekPM, weekco2, monthbarom, monthtempdew (all within 0.2 s)
• 10.3 s gap to monthco2, yearbarom, and the remaining 4 year summaries. (within 0.1s)
• 10.2s gap to seasons.js, css, font and favicon.ico (all at identical times - to the nanosecond!)

[Cameron D]

The high cpu usage seems to relate to the size of the DB.

I tried with my schema and a new sqlite db - no problems

I migrated my mysql DB to sqlite and got the same problems, if not worse - elapsed time 18 minutes clock (10 min cpu user, 2 min sys)  to run a default Seasons report and it failed to create the index.html file.

Another time it took 10 minutes, created an index.html, but only generated a single png file.

I'm out of ideas.

[Vince Skahan]

Cameron several of us have run v5 with very large db of over 10 years data on pi4 or lesser boxes without such issue, so a bit more data from you would be helpful. How big a size are you running ? On what hardware ?

[Cameron D]

The wmr300 DB has 3.7million records, and the ecowitt DB has only 530k.  The system is a VM running on an i5 - the VM is allocated 8GB ram and 4 cores.  It was not stressed at all by weewx V4.

The CPU usage is all in the python, not in the mariadb server.

While the python code is thrashing around achieving nothing, the memory allocation for the report script is oscillating from under 200MB to 780M - no swapping, the machine still has 4GB free.

[Cameron D]

I ran some more tests with my main DB converted to sqlite, using weectl to run repots using the default Seasons skin.

The times remained basically constant  at about  4.5 minutes user+sys CPU as I kept deleting old records.

As I got below 500k records the times dropped proportionally to the number of archive records, extrapolating back to about 20 seconds for no records (which is still a crazy long time.

In addition the various gaps between the file creation times also dropped in proportion.

The bit I wrote in the previous post about memory use did not apply to this set of tests, as the value just stuck steadily to 120MB. I think I have lost track of some of the tests I was doing.

[Vince Skahan]

If you're running a throwaway vm, try running the benchmarks at https://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation and see what your numbers look like.  You should see numbers well under 3 seconds on that kind of gear.

[Tom Keffer]

I am trying to reproduce the result that Vince and Cameron have reported, but am unable to do so. More details would be helpful.

Did you both do an upgrade from v4.10.2 to v5.0.0rc1?

Did you have any extensions installed?

Vince, I know you were running "weectl report run" using Python 3.11, but, Cameron, what were you running? 

Cameron, do you have a stack trace?

It's hard to debug because the snippet that Vince has does not show the type of the underlying exception.

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/c5270d4d-0008-4b23-ac1b-5ab4647e2e8bn%40googlegroups.com.

[Vince Skahan]

What I did was install a clean beta via pip, then overwrite extension.py down buried in the venv from the later one you'd uploaded to github.  I 'think' that I hand-edited in one patch as you'd previously updated multiple things after the beta was released, but my memory is a little hazy there.

In thinking about it after the fact, I wish I had stopped weewx and deleted all the __pycache__ trees in the venv, then restarted weewx and retried the same command  to see if the issue went away.  A clean install worked.  A later try where I was more careful hand-editing the diff in worked.

[Tom Keffer]

So, it sounds like you can't reproduce it either...?

Because it looks like a hard crash of the Python interpreter, it is likely it was caused by a bad cache, or something like that.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/f24957bc-b805-4507-9fcb-794492cfe8dbn%40googlegroups.com.

[rcst...@gmail.com]

Hey All,

 

I’m going to give this a go, but want to confirm based on this: https://weewx.com/docs/5.0/upgrade/

 

I’m just running on a raspi, and installed via apt, so I should just be able to do as documented here and upgrade this way? I swear a few months back there wasn’t going to be a packaged version of 5, and the emphasis was pushing toward using pip… that changed along the way? (sorry for being oblivious to that change, I try to at least glance at all the emails via this list, I must have just missed it).

 

Thanks!

 

From: weewx-de...@googlegroups.com <weewx-de...@googlegroups.com> On Behalf Of Tom Keffer
Sent: Thursday, December 21, 2023 3:07 PM
To: weewx-development <weewx-de...@googlegroups.com>
Subject: [weewx-development] V5.0 release candidate available

 

Celebrate the solstice (coming up in 4 hours) and the start of Winter with a WeeWX release candidate!

 

Changes since b17

We gave up on a standalone logger and have gone back to whatever your system uses. This is more robust to permission problems, but may make it harder to get the logs back out. See the wiki article View logs for tips on how to convince your system to let you have a look.

 

As always, you can add a [Logging] section to weewx.conf and set up your own rotating log system. See the article WeeWX V4 and logging for how to do this.

 

Pip

 

For pip installs, please delete your old virtual environment, then install from scratch by following the pip install instructions. While upgrading should work, we are particularly interested in the experience of a new install, including setting up a daemon and udev files. Make sure to follow the new instructions that use a daemon setup script.

 

 

Debian

 

For Debian package installs, modify /etc/apt/sources.list as follows:

echo "deb [arch=all] https://weewx.com/apt-test/python3 buster main" | sudo tee /etc/apt/sources.list.d/weewx.list

Note the "apt-test". This tells apt to look there for the beta release, instead of the normal repository. You may want to change it back when you're done.

 

 

Red Hat

 

For Red Hat package installs, put this in /etc/yum.repos.d/weewx.repo

[weewx]

name=weewx

baseurl=http://weewx.com/yum-test/weewx/el9

enabled=1

Note the "yum-test". This tells yum to look there for the beta release, instead of the normal repository. Again, you may want to change things back when you're done.

 

 

What we're looking for

 

1. We are interested in your upgrade experience. Did the installers get your configuration file right? Skins? Is the daemon configured correctly?

 

2. With this release, we have included udev rules to set the correct permissions for devices. Did it work? If not, did unplugging then replugging the device work?

 

3. How about logging? Is it going to your system logger? Do the labels look reasonable?

 

Thanks!

 

-Tom & Matt

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zECFSEWf9immM3fFroWU79GcYyT65ibe8w1hWd-XJ92s1g%40mail.gmail.com.

[Tom Keffer]

Yes, there is a packaged version, so, yes, you do a package upgrade. NOTE: the release candidate is in a temporary apt repository, as outlined in the first message of this thread.

[rcst...@gmail.com]

Hey Tom,

 

Thanks! Should I assume I should stop the current weewx, backup the db, etc? And yup, realize it’s a temp repo. =)

 

Thanks!

[rcst...@gmail.com]

Hey All,

 

Okay, stopped weewx, did the change in sources, and an apt upgrade. Declined to replace weewx.conf, and then updated the existing one to have the weewx root properly, and start weewx. Started right away. Grabbed the one archive record that it missed during my process.

 

Files generated relatively quickly, and rsync worked. Fwiw, still seems to run as root on here, rather than weewx. Not sure that matters. I do see a weewx user in /etc/passwd, so it got created. Rebooted the raspi and same thing, so not too worried it’s going to break. I imagine if we have to flip from root to weewx that would require some chown’s and ssh key migrations for rsync.

 

Otherwise, seems just fine. =) I’m on a pretty “vanilla” setup, no real skin modifications, excess plugins, etc. Just a Vantage Vue pulling from a WeatherlinkIP.

 

Of interest, it does seem like image generation seems orders of magnitude slower than 4.10

 

Here’s 4.10

Dec 26 17:00:25 raspi-server-misc weewx[25329] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 1.79 seconds

Dec 26 17:00:27 raspi-server-misc weewx[25329] INFO weewx.imagegenerator: Generated 24 images for report SeasonsReport in 2.01 seconds

Dec 26 17:00:27 raspi-server-misc weewx[25329] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx

Dec 26 17:00:31 raspi-server-misc weewx[25329] INFO weewx.cheetahgenerator: Generated 2 files for report Emkubed in 3.72 seconds

Dec 26 17:00:31 raspi-server-misc weewx[25329] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/emkubed

Dec 26 17:00:33 raspi-server-misc weewx[25329] INFO weeutil.rsyncupload: rsync'd 34 files (567,397 bytes) in 2.03 seconds

 

Here’s 5.0rc1

Dec 26 17:21:47 raspi-server-misc weewxd[716]: INFO weewx.imagegenerator: Generated 12 images for report SeasonsReport in 74.31 seconds

Dec 26 17:21:47 raspi-server-misc weewxd[716]: INFO weewx.reportengine: Copied 5 files to /var/www/html/weewx

Dec 26 17:21:51 raspi-server-misc weewxd[716]: INFO weewx.cheetahgenerator: Generated 2 files for report Emkubed in 3.81 seconds

Dec 26 17:21:51 raspi-server-misc weewxd[716]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx/emkubed

Dec 26 17:21:53 raspi-server-misc weewxd[716]: INFO weeutil.rsyncupload: rsync'd 27 files (506,049 bytes) in 1.85 seconds

 

Any thoughts?

 

Thanks Tom and everyone!

 

From: Tom Keffer <tke...@gmail.com>
Sent: Tuesday, December 26, 2023 3:20 PM
To: rcst...@gmail.com
Cc: weewx-development <weewx-de...@googlegroups.com>

[Cameron D]

74 seconds clock time to run a seasons report looks like the same problem I am seeing.

On Wednesday 27 December 2023 at 11:25:17 am UTC+10 rcst...@gmail.com wrote:

Hey All,

...

Here’s 4.10

Dec 26 17:00:25 raspi-server-misc weewx[25329] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 1.79 seconds

 

...

[Tom Keffer]

Can you please run the benchmark that Vince linked to earlier: https://github.com/weewx/weewx/wiki/Benchmarks-of-file-and-image-generation

Unfortunately, it runs only under V5 (not v4.10). Still, it will give us an idea whether you're in the right ballpark for runtimes.

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/a8336c10-46c9-41f7-96ce-a6b248cc4431n%40googlegroups.com.

[Cameron D]

Mine is an upgrade from 4.10.2, and probably all the way from the original v4, at which stage I think is was a clean apt  install from V3 and I think that was when I switched to python 3

.

 My system is up-to-date Deb 12, so Python 3.11.2-1+b1 

I can reproduce it by simply removing the symlink from /usr/share/weewx/user to /etc/weewx/bin/user

The only user-defined in that conf file is schema = user.db_wmr300.schema

If I comment that out then it runs OK.  With it in , I get...

 weectl report run SeasonsReport  --config=/etc/weewx/wmr300-sqlite-mydb.conf
Using configuration file /etc/weewx/wmr300-sqlite-mydb.conf
The following reports will be run: SeasonsReport

Generating as of last timestamp in the database.
Traceback (most recent call last):

  File "/usr/share/weewx/weectl.py", line 74, in <module>
    main()
  File "/usr/share/weewx/weectl.py", line 66, in main
    namespace.func(namespace)
  File "/usr/share/weewx/weectllib/__init__.py", line 96, in dispatch
    namespace.action_func(config_dict, namespace)
  File "/usr/share/weewx/weectllib/report_cmd.py", line 92, in run_reports
    weectllib.report_actions.run_reports(config_dict,
  File "/usr/share/weewx/weectllib/report_actions.py", line 84, in run_reports
    engine = weewx.engine.DummyEngine(config_dict)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/weewx/weewx/engine.py", line 89, in __init__
    self.loadServices(config_dict)
  File "/usr/share/weewx/weewx/engine.py", line 157, in loadServices
    obj = weeutil.weeutil.get_object(svc)(self, config_dict)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/weewx/weewx/wxservices.py", line 103, in __init__
    self.db_manager = engine.db_binder.get_manager(data_binding=data_binding,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/weewx/weewx/manager.py", line 752, in get_manager
    manager_dict = get_manager_dict_from_config(self.config_dict,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/weewx/weewx/manager.py", line 886, in get_manager_dict_from_config
    manager_dict['schema'] = weeutil.weeutil.get_object(schema_name)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/weewx/weeutil/weeutil.py", line 1404, in get_object

    module = importlib.import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1142, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'user'

[Vince Skahan]

ok that looks like the old bug where extensions were in one place and some of weewx was looking in other places (at one point in the betas)

[Cameron D]

Benchmark results: 1.4s for files and 1.5s for images

Added to Wiki.

[Ryan Stasel]

In my case, benchmark: 

weectl report run --config=./weewx.conf

Using configuration file ./weewx.conf

All enabled reports will be run.

Generating as of last timestamp in the database.

2023-12-26 18:42:45 weectl[1787]: INFO weewx.cheetahgenerator: Generated 21 files for report SeasonsReport in 6.12 seconds

2023-12-26 18:42:53 weectl[1787]: INFO weewx.imagegenerator: Generated 68 images for report SeasonsReport in 7.56 seconds

Done.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/ba4e2612-c6c4-4592-8f4f-09f184165f3cn%40googlegroups.com.

--

-Ryan Stasel

[Ryan Stasel]

That time may be skewed by my prod weewx being in it's current state. Stopped the production instance and re-ran:

weectl report run --config=./weewx.conf

Using configuration file ./weewx.conf

All enabled reports will be run.

Generating as of last timestamp in the database.

2023-12-26 18:47:13 weectl[1845]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.45 seconds

2023-12-26 18:47:14 weectl[1845]: INFO weewx.imagegenerator: Generated 17 images for report SeasonsReport in 0.82 seconds

Done.

That seems much more inline with 4.10.2. And like Cameron, I've been updating from 4.x along the way, I think I started my DB with 4.x (I don't believe I upgraded from 3). My DB is over 10y of data, and is about 700MB. 

Hopefully this helps find a fix... 

--

-Ryan Stasel

[Vince Skahan]

Clean pi5 64bit installation of the beta dpkg.  I manually copied my prod weewx.sdb (17+ years of data) and its previously generated NOAA files into place before starting weewx.

Dec 26 18:20:15 raspberrypi weewxd[12806]: INFO weewx.manager: Added record 2023-12-26 18:20:00 PST (1703643600) to database 'weewx.sdb'
Dec 26 18:20:15 raspberrypi weewxd[12806]: INFO weewx.manager: Added record 2023-12-26 18:20:00 PST (1703643600) to daily summary in 'weewx.sdb'
Dec 26 18:20:16 raspberrypi weewxd[12806]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 0.78 seconds
Dec 26 18:20:17 raspberrypi weewxd[12806]: INFO weewx.imagegenerator: Generated 30 images for report SeasonsReport in 0.53 seconds
Dec 26 18:20:17 raspberrypi weewxd[12806]: INFO weewx.reportengine: Copied 5 files to /var/www/html/weewx

clean install of v4 dpkg after purging all traces of weewx, also putting my big prod weewx.sdb into place

Dec 26 18:50:15 raspberrypi python3[13902]: weewx[13902] INFO weewx.manager: Added record 2023-12-26 18:50:00 PST (1703645400) to database 'weewx.sdb'
Dec 26 18:50:15 raspberrypi python3[13902]: weewx[13902] INFO weewx.manager: Added record 2023-12-26 18:50:00 PST (1703645400) to daily summary in 'weewx.sdb'
Dec 26 18:50:16 raspberrypi python3[13902]: weewx[13902] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 0.58 seconds
Dec 26 18:50:16 raspberrypi python3[13902]: weewx[13902] INFO weewx.imagegenerator: Generated 18 images for report SeasonsReport in 0.24 seconds

then upgraded v4 => v5 selecting N when prompted about the weewx.conf file

Dec 26 18:55:14 raspberrypi weewxd[14576]: INFO weewx.manager: Added record 2023-12-26 18:55:00 PST (1703645700) to database 'weewx.sdb'
Dec 26 18:55:14 raspberrypi weewxd[14576]: INFO weewx.manager: Added record 2023-12-26 18:55:00 PST (1703645700) to daily summary in 'weewx.sdb'
Dec 26 18:55:16 raspberrypi weewxd[14576]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 0.76 seconds
Dec 26 18:55:16 raspberrypi weewxd[14576]: INFO weewx.imagegenerator: Generated 18 images for report SeasonsReport in 0.25 seconds
Dec 26 18:55:16 raspberrypi weewxd[14576]: INFO weewx.reportengine: Copied 5 files to /var/www/html/weewx

Dec 26 19:00:14 raspberrypi weewxd[14576]: INFO weewx.manager: Added record 2023-12-26 19:00:00 PST (1703646000) to database 'weewx.sdb'
Dec 26 19:00:14 raspberrypi weewxd[14576]: INFO weewx.manager: Added record 2023-12-26 19:00:00 PST (1703646000) to daily summary in 'weewx.sdb'
Dec 26 19:00:15 raspberrypi weewxd[14576]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 0.58 seconds
Dec 26 19:00:16 raspberrypi weewxd[14576]: INFO weewx.imagegenerator: Generated 36 images for report SeasonsReport in 0.91 seconds
Dec 26 19:00:16 raspberrypi weewxd[14576]: INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx

So I see really no significant differences in dpkg on 64bit raspios for v5, v4, and v4->v5 installations here.  The cpu is essentially idle.

I didn't look into the number of images differing it seems.  I don't run the Seasons skin typically so I don't really keep track of what it's doing in which situations.

[Ryan Stasel]

Ran this again, realizing I hadn't removed public_html between first and second run (thinking it was production weewx causing the issue). 

root@raspi-server-misc:/var/tmp/weewx-benchmark# weectl report run --config=./weewx.conf

Using configuration file ./weewx.conf

All enabled reports will be run.

Generating as of last timestamp in the database.

2023-12-26 20:23:48 weectl[2979]: INFO weewx.cheetahgenerator: Generated 21 files for report SeasonsReport in 6.09 seconds

2023-12-26 20:23:55 weectl[2979]: INFO weewx.imagegenerator: Generated 68 images for report SeasonsReport in 6.59 seconds

Done.

So Vince, you said something about a previous bug looking for extensions somewhere else? 

--

-Ryan Stasel

[Cameron D]

I copied the benchmark DB and the conf file onto my main weewx installation, edited the conf file accordingly, and got the same benchmark results, about 1.5 seconds each.

So I can only assume there is something in my customisations that V5 is having trouble  with.

[Cameron D]

I just wiped weewx from my test system and did a fresh install - I got 5.0.0-rc2.

There was no weewx.conf created, so I ran dpkg-reconfigure and there is still no weewx.conf.

I added back the benchmark conf and DB.  On trying the manual report gen it exited immediately, and syslog showed that is was complaining that user.extensions was missing. Correct - bin/user is empty.

[Cameron D]

I think the missing weewx.conf is because I did apt remove, then renamed all the folders, thinking that would clean things out.  

However it looks like dpkg keeps a bit of config information elsewhere, and makes some incorrect assumptions.

An apt purge seems to be required, otherwise it is not really a fresh install.

[wee...@gmx.de]

Hi,

I have installed version 5.0.0rc1 with the pip method on a new VM (Debian Bookworm). I want to use nginx as web server.
When calling xxx.xxx.xxx.xx I get from NGINX: 403 Forbidden
This means a rights problem for me. Where can I start?

The following is contained in /etc/nginx/sites-available/default, among other things:
---------
root /var/www/html;

# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;

server_name _;

location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}

location /weewx {
alias /home/michael/weewx-data/public_html;
        }
# pass PHP scripts to FastCGI server
#
------------

Many thanks in advance for any advice.

Greetings Michael

[Vince Skahan]

On Wednesday, December 27, 2023 at 10:37:13 AM UTC-8 wee...@gmx.de wrote:

I have installed version 5.0.0rc1 with the pip method on a new VM (Debian Bookworm). I want to use nginx as web server.
When calling xxx.xxx.xxx.xx I get from NGINX: 403 Forbidden
This means a rights problem for me. Where can I start? 

 

The same place you can always start.

Google.

Google for your exact error message.

But to help explain a bit hopefully...

• nginx runs as user www-data group www-data
  
• weewx in a pip installation is running as your user and group
  
• your $HOME likely does not permit the nginx process user to read down to your public_html tree
  

 You have at least two options....

• open up your $HOME permissions to let the world read down to your public_html tree (security risk!)
• or configure weewx to write to the /var/www tree that nginx permit (safer for security reasons)

I did mine this way on a pi:

• sudo mkdir /var/www/html/weewx
• sudo chown pi:pi /var/www/html/weewx                       #### <=== use your user:group here
• ln -s /var/www/html/weewx /home/pi/weewx-data/public_html  #### <=== use your username here

The reason I do it this way is that I do not need to edit anything in nginx sites-enabled or even in weewx.conf.  I just make a directory and set its permissions and create one symlink in $HOME before starting weewx and it all works.  Simpler for me to follow.

[Vince Skahan]

> The only user-defined in that conf file is schema = user.db_wmr300.schema

> If I comment that out then it runs OK.  With it in , I get...

Cameron - v5 expects that stuff in /etc/weewx/bin/user but v4 expected it in /usr/share/weewx/user, so the location has changed.   You need your user stuff in the new v5 location.

I did a clean v4 install and then an upgrade to the v5 rc dpkg and it 'did' correctly move things over and disable (rename) the old location, so the current dpkg looks good.  I'm guessing you restored something manually after the fact and put it in the wrong (v4) location rather than the new (v5) location.

[Tom Keffer]

I can confirm that V5 does have a performance problem with image generation. Here's how the various versions compare (using the file and image generation benchmark).

| Hardware                    | WeeWX    | Python | Files (21) | Images (68) 
|-----------------------------|----------|--------|-----------:|------------
| MacBook Air, M1 2020        | 4.10.2   | 3.7.16 |      0.69s |       0.78s 
| MacBook Air, M1 2020        | 5.0.0b13 | 3.7.16 |      0.71s |       0.97s 
| MacBook Air, M1 2020        | 5.0.0rc1 | 3.7.16 |      0.74s |       1.28s 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/122b7ed6-0a04-4947-8f51-9a603abcb688n%40googlegroups.com.

[Cameron D]

Vince,

 I think you misinterpreted my observations.

Everything had  been moved to  /etc/weewx/bin/user when V5 upgraded, but something in weectl was only looking in the old location.  I had to create a link at the  old location to make weectl work in V5.

I suppose I was a bit ambiguous in describing the direction of the link.

And, of course the existence of the symlink confused the upgrade to rc1-2 horribly.

[Vince Skahan]

The need to create the link 'was' the old bug I mentioned.  It was fixed quite a number of weeks ago, or so it seemed at the time that is.

I cannot duplicate what you ran into, and everything works as expected with no errors seen here.

• started with a clean deb system
• installed v4 dpkg via the procedure for that
• added one extension via wee_extension
• restarted weewx v4 - the extension works fine with no errors seen
• upgraded to v5 rc1 per that procedure, taking the default answers when prompted
• restarted weewx v5 - the extension still works fine with no errors seen
• and the old user tree under /usr/share/weewx was renamed to be user.yyyymmddhhmmss as expected

You're going to have to precisely document step-by-step exactly how you made something break in order for somebody to be able to work the issue if you're still having it.

[Tom Keffer]

Back to the benchmarks. I made a small technical mistake. The comparison now stands at:

| Hardware                  | WeeWX      | Python | Files (21) | Images (68) |

|---------------------------|------------|--------|-----------:|------------:|

| MacBook Air, M1 2020      | 4.10.2     | 3.7.16 |      0.69s |       0.78s |
| MacBook Air, M1 2020      | 5.0.0b13   | 3.7.16 |      0.71s |       0.97s |

| MacBook Air, M1 2020      | 5.0.0rc1   | 3.7.16 |      0.69s |       0.93s |

So, V5 image generation is slower, but then it's also doing more. V5 is able to generate plots of pure synthetic types, not just types in the database. That requires consulting the database to see whether it can do the type.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/b755bb2d-b64f-4f78-812e-dbae0224f1f9n%40googlegroups.com.

[Cameron D]

Tom,

is there any chance that it is confused by me using  mysql storage type of float rather than real?

[rcst...@gmail.com]

Chasing my issue, decided to remove some old extensions I wasn’t using. Rtldavis, sdr, fileparse. Removed the blocks from the now uninstalled extensions. During that process, the whole station block in weewx.conf seems to have been removed (unclear if extension uninstall did this, or me on accident)? Copied that back in from a backup. and fired up weewx.

 

No change in processing speed.

 

Dec 27 16:15:36 raspi-server-misc weewxd[16194]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 11.69 seconds

Dec 27 16:16:48 raspi-server-misc weewxd[16194]: INFO weewx.imagegenerator: Generated 12 images for report SeasonsReport in 71.77 seconds

 

Only other extension I have is Emkumed for the iOS app. I’m no python expert (or even amateur), but I don’t see anything in the files related to that extension that would cause it to generate any delay.

 

The only other thing of interest is I have weewx doing loop_request=3 since some values aren’t in loop 1 and some aren’t in loop 2 (I think it was battery voltage info that isn’t in loop 2).

 

Not sure what other things there might be. Happy to provide my weewx.conf if that would be helpful.  

 

Tom, if you’re testing on an M1, and I think Vince is testing on a Pi 5, could we be seeing some lack of certain hardware capabilities in the graph processing? Otherwise, I can grab Seasons from source and try dropping that in to see if maybe there’s something wonky with the version I have. Most of the files appear to be dated early 2022 except skin.conf, font, lang, and NOAA directories… but the files inside are also early 2022.

 

Thanks.

 

From: weewx-de...@googlegroups.com <weewx-de...@googlegroups.com> On Behalf Of Vince Skahan
Sent: Wednesday, December 27, 2023 3:50 PM
To: weewx-development <weewx-de...@googlegroups.com>

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/b755bb2d-b64f-4f78-812e-dbae0224f1f9n%40googlegroups.com.

[Vince Skahan]

I did tests on pi3+ pi4 pi5 and a m1 mac mini to compare v4 vs v5 on each hardware to check if it was hardware related. Not really.

The slower boxes just show the differences between v4 and v5 a little more obviously.

-----  vince...@gmail.com ----

[rcst...@gmail.com]

Thanks Vince for confirming that. So not worried about that being the case then, which is nice.

 

From: Vince Skahan <vince...@gmail.com>
Sent: Wednesday, December 27, 2023 4:29 PM
To: rcst...@gmail.com
Cc: weewx-development <weewx-de...@googlegroups.com>
Subject: Re: [weewx-development] Re: V5.0 release candidate available

 

I did tests on pi3+ pi4 pi5 and a m1 mac mini to compare v4 vs v5 on each hardware to check if it was hardware related. Not really.

[Tom Keffer]

On Wed, Dec 27, 2023 at 4:18 PM 'Cameron D' via weewx-development <weewx-de...@googlegroups.com> wrote:

Tom,

is there any chance that it is confused by me using  mysql storage type of float rather than real?

I'm not a MySQL expert, but I doubt it.

WeeWX does lots of little queries, which MySQL doesn't do as well as SQLite. It's possible that V5 is doing something that aggravates that.

[rcst...@gmail.com]

Other piece I have is ephem installed.

 

Diffin’g the current Seasons vs the version I have, the biggest difference seems to be in the Celestial.inc… Not sure if any of these would be causing massive slowdowns though (output below). Also seeing a lot of unit differences in skin.conf (things changing from “hour” to “1h” and “day” to “1d”, etc.

 

But the output in the logs seems to indicate the slowness is generating the images, not generating the html.

 

Thanks!

 

-Ryan Stasel

 

diff -r ./celestial.inc /etc/weewx/skins/Seasons/celestial.inc

10,11c10,11

<   #set $sun_altitude = $almanac.sun.alt

<   #if $sun_altitude < 0

---

>   #set $sun_altitude = $almanac.sun.altitude

>   #if $sun_altitude.raw < 0

59c59

<             <td class="data">$("%.1f&deg;" % $almanac.sun.az)</td>

---

>             <td class="data">$almanac.sun.azimuth.format("%03.1f")</td>

63c63

<             <td class="data">$("%.1f&deg;" % $sun_altitude)</td>

---

>             <td class="data">$sun_altitude.format("%.1f")</td>

67c67

<             <td class="data">$("%.1f&deg;" % $almanac.sun.ra)</td>

---

>             <td class="data">$almanac.sun.topo_ra.format("%03.1f")</td>

71c71

<             <td class="data">$("%.1f&deg;" % $almanac.sun.dec)</td>

---

>             <td class="data">$almanac.sun.topo_dec.format("%.1f")</td>

119c119

<             <td class="data">$("%.1f&deg;" % $almanac.moon.az)</td>

---

>             <td class="data">$almanac.moon.azimuth.format("%.1f")</td>

123c123

<             <td class="data">$("%.1f&deg;" % $almanac.moon.alt)</td>

---

>             <td class="data">$almanac.moon.altitude.format("%.1f")</td>

127c127

<             <td class="data">$("%.1f&deg;" % $almanac.moon.ra)</td>

---

>             <td class="data">$almanac.moon.topo_ra.format("%.1f")</td>

131c131

<             <td class="data">$("%.1f&deg;" % $almanac.moon.dec)</td>

---

>             <td class="data">$almanac.moon.topo_dec.format("%.1f")</td>

161c161

<     <p>Install <em>pyephem</em> for detailed celestial timings.</p>

---

>     <p>Install <a href=https://pypi.org/project/ephem/><em>ephem</em></a> for detailed celestial timings.</p>

 

diff -r ./skin.conf /etc/weewx/skins/Seasons/skin.conf

8c8

< SKIN_VERSION = 4.10.2

---

> SKIN_VERSION = 5.0.0rc1

29c29

<     # If you have a Google Analytics ID, uncomment and edit the next line, and

---

>     # If you have a Google Analytics GA4 tag, uncomment and edit the next line, and

31c31

<     #googleAnalyticsId = UA-12345678-1

---

>     #googleAnalyticsId = G-ABCDEFGHI

269c269

<         time_length = 97200 # 27 hours

---

>         time_length = 27h

350c350

<                 aggregate_interval = hour

---

>                 aggregate_interval = 1h

359c359

<                 aggregate_interval = hour

---

>                 aggregate_interval = 1h

384c384

<                 aggregate_interval = hour

---

>                 aggregate_interval = 1h

410c410

<         aggregate_interval = hour

---

>         aggregate_interval = 1h

488c488

<                 aggregate_interval = day

---

>                 aggregate_interval = 1d

496c496

<                 aggregate_interval = day

---

>                 aggregate_interval = 1d

521c521

<                 aggregate_interval = day

---

>                 aggregate_interval = 1d

546c546

<         aggregate_interval = 10800 # 3 hours

---

>         aggregate_interval = 3h

625c625

<                 aggregate_interval = day

---

>                 aggregate_interval = 1d

633c633

<                 aggregate_interval = day

---

>                 aggregate_interval = 1d

658c658

<                 aggregate_interval = day

---

>                 aggregate_interval = 1d

683c683

<         aggregate_interval = day

---

>         aggregate_interval = 1d

762c762

<                 aggregate_interval = week

---

>                 aggregate_interval = 1w

770c770

<                 aggregate_interval = week

---

>                 aggregate_interval = 1w

795c795

<                 aggregate_interval = week

---

>                 aggregate_interval = 1w

 

From: weewx-de...@googlegroups.com <weewx-de...@googlegroups.com> On Behalf Of Tom Keffer
Sent: Wednesday, December 27, 2023 4:52 PM
To: Cameron D <Cgo...@davidsoncj.id.au>
Cc: weewx-development <weewx-de...@googlegroups.com>
Subject: Re: [weewx-development] Re: V5.0 release candidate available

 

On Wed, Dec 27, 2023 at 4:18 PM 'Cameron D' via weewx-development <weewx-de...@googlegroups.com> wrote:

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEDdK_EBZQ4Qx-kx5--PJUy0N-9ame3Yur-KQcWSCBOcvg%40mail.gmail.com.

[Cameron D]

Tom,

my question was not from the MySQL side, but because you said the weewx code consults the database to see if it can "do that type". 

In any case, I since realised tha I am seeing the same problem with the DB converted to sqlite, so that cannot be a cause.

On a side question, is there much significance to the "version" number in the DB metadata?  I see one of my DBs is version 1 and the other is V4?

Cameron.

[Cameron D]

Updated to rc1-2 and the problem of weectl looking in the wrong place for  user modules seems to be fixed.

[Tom Keffer]

The database version number only comes into play if you change your archive interval. Higher versions "weight" the result by the length of the interval.

One of my databases, now 18 years old, is still at V1. 

On Wed, Dec 27, 2023 at 6:12 PM 'Cameron D' via weewx-development <weewx-de...@googlegroups.com> wrote:

Tom,

my question was not from the MySQL side, but because you said the weewx code consults the database to see if it can "do that type". 

In any case, I since realised tha I am seeing the same problem with the DB converted to sqlite, so that cannot be a cause.

On a side question, is there much significance to the "version" number in the DB metadata?  I see one of my DBs is version 1 and the other is V4?

Cameron.

On Thursday 28 December 2023 at 10:52:26 am UTC+10 Tom Keffer wrote:

On Wed, Dec 27, 2023 at 4:18 PM 'Cameron D' via weewx-development <weewx-de...@googlegroups.com> wrote:

Tom,

is there any chance that it is confused by me using  mysql storage type of float rather than real?

I'm not a MySQL expert, but I doubt it.

WeeWX does lots of little queries, which MySQL doesn't do as well as SQLite. It's possible that V5 is doing something that aggravates that.

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/db5da043-0192-417a-ab27-838a82ebb47bn%40googlegroups.com.

[Vince Skahan]

On Wednesday, December 27, 2023 at 4:24:35 PM UTC-8 rcst...@gmail.com wrote:

Dec 27 16:15:36 raspi-server-misc weewxd[16194]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 11.69 seconds

Dec 27 16:16:48 raspi-server-misc weewxd[16194]: INFO weewx.imagegenerator: Generated 12 images for report SeasonsReport in 71.77 seconds

What hardware are you on again ?  What version of weewx was this for ?

I don't think we've seen 'files' with any significant difference in v4 vs. v5 

 

[Cameron D]

In the cases of my problematic reports, it never prints the diagnostic summary lines at all.

But I can tell that the largest gaps, based on file dates is generating each of  index.html, then rss.xml, statistics.html  and all the images are generated over a similar time in total  to each of those gaps.

[Ryan Stasel]

I’m on an Rpi 4 4gb. Currently using 5.0.0rc1. The seasons skin appears to all be files from 4.10.2 based on dates and versions. Seems like the Deb didn’t update the skin (pretty sure I told it to during the upgrade). 

On Dec 27, 2023, at 18:50, Vince Skahan <vince...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/ac57b45d-fd3a-4195-9417-a0c003db2a41n%40googlegroups.com.

[Cameron D]

OK, this "ability to generate synthetic types"  was the clue to the excessive cpu and time.

I started with a copy of my main DB converted to sqlite and trimmed it to 5 minute intervals and ~100k archive records, so it is close to the benchmark db.

I did a bit of (my first ever) python profiling and compared the benchmark times with the same reports on my DB.

• benchmark: The busiest weewx function xtypes.py:128(get_aggregate) was called ~16000 times for a total duration of 0.4 seconds.
• my DB: the top function was: xtypes.py:76(get_scalar), called 6 milllion times, for a duration of 130 seconds, then   xtypes.py:27(get_scalar), was called 35 milllion times, total time 84 seconds.

Further down I could see lots of time spent in routines with names based on windchill or heatindex.

From the start, I had decided I didn't want to see either of these parameters, so I had removed them from my DB.  But now Seasons is forcing a years' worth of their recalculation every report.  Hence durations of up to several minutes.

I gradually trimmed down the skin.conf, and only by removing every trace of windchill, heatindex and tempfeel could I get the execution times down from 59 seconds to about 10 seconds - if I left a single reference remaining then it hung around the 56-59 seconds mark.

The other, more minor, but still quite significant factor, is that all the other items listed like extratemp1, extratemp2, etc have been removed from my db. Referencing them in charts seems to take more cpu to not plot them than to have data and have them plotted.  By removing all the chart references to non-existent columns from my skin I got the elapsed time down from ~10 seconds to 2.3s.

It is still refusing to print the times, either to console or the system log. (but I can see the debug output).  The times I quoted were user+system cpu time consumed, as reported by "time command..."

[Tom Keffer]

Very useful data, Cameron.

Right now, if you want one year's worth of say, windchill, and it's not in the database, it is calculated one record at a time. For a 5 minute archive interval, that's 100k+ database accesses. It's a general algorithm, but it's slow.

Alternatively, one could write a specialized algorithm for windchill. The sensible thing to do would be to read a year's worth of temperature and wind speed, all in one go --- one database access. Then use the results to calculate the year's worth of windchill. The downside is that it's not general at all: it would only know how to calculate windchill. The upside is that the existing xtypes API can be used right now to do this. You'd have to write extensions for all of your missing synthesized types. 

How to write a more general algorithm that uses only a single database access? It might be possible to set up a mechanism where an xtype specifies which types it needs from the database first. Then get_series() calls get_scalar() with the results. I'd have to think about that. It would definitely be an extension to the xtypes API.

Thanks for setting us on the right course, Cameron.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/14b64e0a-2550-4e82-b8b9-6184eb888485n%40googlegroups.com.

[Greg Troxel]

Tom Keffer <tke...@gmail.com> writes:

> Very useful data, Cameron.
>
> Right now, if you want one year's worth of say, windchill, and it's not in
> the database, it is calculated one record at a time. For a 5 minute archive
> interval, that's 100k+ database accesses. It's a general algorithm, but
> it's slow.
>
> Alternatively, one could write a specialized algorithm for windchill. The
> sensible thing to do would be to read a year's worth of temperature and
> wind speed, all in one go --- one database access. Then use the results to
> calculate the year's worth of windchill. The downside is that it's not
> general at all: it would only know how to calculate windchill. The upside
> is that the existing xtypes API can be used right now to do this. You'd
> have to write extensions for all of your missing synthesized types.

I wonder if this could be organized into

a database accessor to get multiple columns, that results in a stream
of N values

functions that accept N values to compute something else
(e.g. windchill from temp/wind).

a function that takes a computing function and a stream

so that most of the code that is the same can be only once. I gather
there are pythonic idioms for the stream.

That may be saying the same thing, but not having a clue about the weewx
details, just a CS back bencher kind of comment :-)

[Cameron D]

My assumption would be that any derivable value that you want to plot should be a column in the database.  Even if you could speed it up, the calculation it would need to be by a factor approaching 100 to be a useful choice.

I think that using derived parameters for only a "current" display should be OK, but as soon as you want to plot it or include it in stats then each report has to process the year's worth.

If I did decide that  I wanted to add windchill at a later date, it looks like it would be just a  weectl database add-column  and then   weectl database calc-missing.  Does that seem right?

[Greg Troxel]

"'Cameron D' via weewx-development" <weewx-de...@googlegroups.com>

writes:

> My assumption would be that any derivable value that you want to plot
> should be a column in the database. Even if you could speed it up, the
> calculation it would need to be by a factor approaching 100 to be a useful
> choice.
> I think that using derived parameters for only a "current" display should
> be OK, but as soon as you want to plot it or include it in stats then each
> report has to process the year's worth.

While I more or less see your point, have you benchmarked the cost of
extra database columns vs computing on the fly? In most computers these
days I would expect CPU is much faster than storage.

Stats though there is a stronger argument for storing it.
1

[Tom Keffer]

I wonder if this could be organized into

  a database accessor to get multiple columns, that results in a stream
  of N values

  functions that accept N values to compute something else
  (e.g. windchill from temp/wind).

  a function that takes a computing function and a stream

This is pretty much what we have now.

dbmanager is the first;

xtypes.get_scalar() is the second;

XTypeTable.get_series() is the third.

What's missing is the ability to specify the N observation types. However, one could just assume them all. That is, XTypeTable.get_series() would just ask for all the data for a year, then call get_scaler(). Of course, that might require a lot of memory. You could relax that requirement a bit by doing the calculation in tranches --- perhaps a month's worth of data at a time --- then stitch together the pieces.

Cameron: your comment that a savings of 100x would be needed for this to be useful is about right. But, database accesses are very expensive. I think you'd be surprised by the savings of reducing the number of db calls from 100k to one.

-tk

[Cameron D]

That comparison was basically what I was investigating - on-the-fly calcs taking minutes, vs fractions of a seconds to retrieve from a DB.
Add to that... to do the calculation you might need to retrieve twice as much data anyway and redo the day-archive calculations every cycle.

As you suggest, I am sure there are ways, such as numpy perhaps, that allow you to handle vectors far more efficiently, but I am doubtful that it would  be worth the trouble. Especially now that sqlite can add columns far more easily.

Cameron.  

[Cameron D]

Well, I would not place too much value on my interpretation of the profile, but here is the sorted list of calls that totalled over 1 second. The total within sqlite.py is less than 20 seconds. It spent longer than that just converting to and fro between C and F

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  5717318  127.420    0.000  477.916    0.000 xtypes.py:76(get_scalar)
 34303908   81.864    0.000   81.864    0.000 xtypes.py:27(get_scalar)
  5717318   40.281    0.000  217.476    0.000 wxxtypes.py:80(get_scalar)
  5718892   35.100    0.000   66.585    0.000 manager.py:503(genBatchRecords)
  5718892   31.380    0.000   31.485    0.000 manager.py:463(genBatchRows)
     1582   30.922    0.020  575.233    0.364 xtypes.py:913(get_aggregate)
  5727031   26.915    0.000   40.451    0.000 units.py:534(__new__)
  2575348   24.718    0.000   42.936    0.000 wxformulas.py:122(windchillMetricWX)
  2575348   24.592    0.000   44.146    0.000 wxformulas.py:221(heatindexC)
  2575348   19.911    0.000   80.997    0.000 wxxtypes.py:157(calc_windchill)
  2575348   19.714    0.000   81.975    0.000 wxxtypes.py:174(calc_heatindex)
5729946/5727456   14.264    0.000   14.315    0.000 {built-in method builtins.getattr}
5736977/5736604   13.563    0.000   13.597    0.000 {built-in method __new__ of type object at 0x9584a0}
  1161605   13.158    0.000   35.678    0.000 sqlite.py:36(guarded_fn)
  5150696   12.203    0.000   12.203    0.000 units.py:47(CtoF)
  5151122   12.065    0.000   12.065    0.000 units.py:50(FtoC)
   577121    8.167    0.000    8.167    0.000 {function guard.<locals>.guarded_fn at 0x7fe282fe5120}
  2575348    7.448    0.000    7.448    0.000 wxformulas.py:144(heatindexF)
  2575348    6.056    0.000    6.056    0.000 wxformulas.py:73(windchillF)
   575069    5.854    0.000   42.773    0.000 manager.py:565(getSql)
   566622    5.294    0.000   51.152    0.000 wxxtypes.py:287(get_scalar)
  1782028    4.221    0.000    4.221    0.000 {method 'startswith' of 'str' objects}
   577121    3.118    0.000   11.285    0.000 sqlite.py:231(execute)
   577121    2.794    0.000    4.404    0.000 sqlite.py:146(cursor)
   577121    1.610    0.000    1.610    0.000 {method 'cursor' of 'sqlite3.Connection' objects}
   585132    1.403    0.000    1.403    0.000 {method 'lower' of 'str' objects}
   577121    1.367    0.000    1.367    0.000 {method 'close' of 'sqlite3.Cursor' objects}
   571755    1.339    0.000    1.339    0.000 {method 'endswith' of 'str' objects}
     7691    0.526    0.000    1.099    0.000 xtypes.py:1070(get_aggregate)

[rcst...@gmail.com]

Swapping out default Seasons with the old version resulted in no change. I do notice skin.conf still says 4.10.2. The previous Seasons skin I believe I had modified because the station info defaults to saying “Vantage Pro2” (since that’s the station type in weewx.conf), but I have a Vantage Vue, and only way I can see to set that is by hard setting it in the template.  

 

Given Cameron’s report of calculating missing values, I’m having to wonder if that could be the issue. Is there a way to have WeeWX actually report what it’s doing in this regard?

 

Weectl debug doesn’t seem to have any info in the output that is helpful from what I can tell. =/ 

 

I do notice in logs “Type beaufort has been deprecated. Use unit beaufort instead.” I’m unsure where this is coming from and if it doing some type of conversion could be causing the slowdown…

 

Ran “weectl database calc-missing –dry-run”, which didn’t really produce any output, unsure if it was supposed to say it calculated missing values or not (even though it’s a dryrun and didn’t actually put anything into the DB). Since I have a Vantage Vue, wondering if there’s some other value that’s not actually provided that is being expected (like, ET I believe isn’t provided by the Vue, but it isn’t graphed because it’s empty in the DB… I would hope weewx isn’t spending time trying to generate it or look for it even though it’s not there).  

 

Would love to figure out the issue here. Having Python pegged at 100% for over a minute every 5 minutes isn’t great (but also isn’t actually breaking anything).

 

Thanks.

 

From: 'Cameron D' via weewx-development <weewx-de...@googlegroups.com>
Sent: Wednesday, December 27, 2023 6:59 PM
To: weewx-development <weewx-de...@googlegroups.com>
Subject: Re: [weewx-development] Re: V5.0 release candidate available

 

In the cases of my problematic reports, it never prints the diagnostic summary lines at all.

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/92d4ec47-5c56-4b0e-ba9b-333c03861df1n%40googlegroups.com.

[Vince Skahan]

On Thursday, December 28, 2023 at 5:26:09 AM UTC-8 Tom Keffer wrote:

Alternatively, one could write a specialized algorithm for windchill. The sensible thing to do would be to read a year's worth of temperature and wind speed, all in one go --- one database access. Then use the results to calculate the year's worth of windchill. The downside is that it's not general at all: it would only know how to calculate windchill. The upside is that the existing xtypes API can be used right now to do this. You'd have to write extensions for all of your missing synthesized types. 

I see two things here.  One is extension(s) to handle the missing synthesized types in archive periods moving forward.  The second is some kind of standalone utility to 'one time' catch up a legacy db with those items that you wished it would have calculated in the ancient past.  Have the standalone utility to get the legacy pain over with 'once' so you don't have to feel that pain every 5 minutes moving forward....

Kinda like how rebuild-daily or calc-missing work (?)

[Cameron D]

I'm not sure.   The reason mine are missing is that I have chosen them to be missing.  I don't want them automatically re-created. 

With the V4 skin it was happy checking for the existence of each parameter and then ignoring any that weren't there.

In my case, I have customised DBs and a heavily customised skin that combines data from two bindings. In my case I am happy to remove references to the problematic parameters in the skins.  I have just gone through and identified the rogue entry to windchill that was left in the RSS feed (which I forgot to disable anyway).

The main question prior to release I would think is how many people have a customised DB but use default skins?

Has anybody checked the benchmark timings for users who may have retained the old non-extended DB structure?

[Ryan Stasel]

Testing this some more, and based on suggestion from Cameron, I have made a copy of the default Seasons template, and enabled in weewx.conf. 

Going through and removing pieces I don't have (ET, UV, etc) from skin.conf, [DisplayOptions], got my generation from 2m27s to 2m20s (no appreciable change) and subsequent runs of 1m45s. So it doesn't appear to be anything in DisplayOptions. 

Going a step further and commenting out pieces I don't have from [ImageGenerator] got me down to 1m7s. with subsequent runs of 3s. 

This was gathered running "time weectl report run SeasonsTest", and removing the output after each run. 

Going through and toggling specific ImageGenerator stanzas, issue appears to be ET. My station doesn't provide ET, so the column is blank... but it's there because I have a Vantage (weewx assumes vantagepro2, or the loop packets include just a null value, super unclear here). 

Looking at my DB, I don't seem to have an ET column (maybe I dropped it at some point in the past... vague recollection of there being bad data in there, and ). Maybe this explains the behavior? Is my best bet a "weectl database reconfigure" to bring things back to default, or just re-add via "weectl database add-column ET"? 

Would love some help! 

Thanks! 

-Ryan Stasel

Here's what I get from listing columns in archive:

pragma table_info(archive);
0|dateTime|INTEGER|1||1
1|usUnits|INTEGER|1||0
2|interval|INTEGER|1||0
3|altimeter|REAL|0||0
4|appTemp|REAL|0||0
5|appTemp1|REAL|0||0
6|barometer|REAL|0||0
7|batteryStatus1|REAL|0||0
8|batteryStatus2|REAL|0||0
9|batteryStatus3|REAL|0||0
10|batteryStatus4|REAL|0||0
11|batteryStatus5|REAL|0||0
12|batteryStatus6|REAL|0||0
13|batteryStatus7|REAL|0||0
14|batteryStatus8|REAL|0||0
15|cloudbase|REAL|0||0
16|co|REAL|0||0
17|co2|REAL|0||0
18|consBatteryVoltage|REAL|0||0
19|dewpoint|REAL|0||0
20|dewpoint1|REAL|0||0
21|extraHumid1|REAL|0||0
22|extraHumid2|REAL|0||0
23|extraHumid3|REAL|0||0
24|extraHumid4|REAL|0||0
25|extraHumid5|REAL|0||0
26|extraHumid6|REAL|0||0
27|extraHumid7|REAL|0||0
28|extraHumid8|REAL|0||0
29|extraTemp1|REAL|0||0
30|extraTemp2|REAL|0||0
31|extraTemp3|REAL|0||0
32|extraTemp4|REAL|0||0
33|extraTemp5|REAL|0||0
34|extraTemp6|REAL|0||0
35|extraTemp7|REAL|0||0
36|extraTemp8|REAL|0||0
37|forecast|REAL|0||0
38|hail|REAL|0||0
39|hailBatteryStatus|REAL|0||0
40|hailRate|REAL|0||0
41|heatindex|REAL|0||0
42|heatindex1|REAL|0||0
43|heatingTemp|REAL|0||0
44|heatingVoltage|REAL|0||0
45|humidex|REAL|0||0
46|humidex1|REAL|0||0
47|inDewpoint|REAL|0||0
48|inHumidity|REAL|0||0
49|inTemp|REAL|0||0
50|leafTemp1|REAL|0||0
51|leafTemp2|REAL|0||0
52|leafWet1|REAL|0||0
53|leafWet2|REAL|0||0
54|lightning_distance|REAL|0||0
55|lightning_disturber_count|REAL|0||0
56|lightning_energy|REAL|0||0
57|lightning_noise_count|REAL|0||0
58|lightning_strike_count|REAL|0||0
59|luminosity|REAL|0||0
60|maxSolarRad|REAL|0||0
61|nh3|REAL|0||0
62|no2|REAL|0||0
63|noise|REAL|0||0
64|o3|REAL|0||0
65|outHumidity|REAL|0||0
66|outTemp|REAL|0||0
67|pb|REAL|0||0
68|pm10_0|REAL|0||0
69|pm1_0|REAL|0||0
70|pm2_5|REAL|0||0
71|pressure|REAL|0||0
72|radiation|REAL|0||0
73|rain|REAL|0||0
74|rainBatteryStatus|REAL|0||0
75|rainRate|REAL|0||0
76|referenceVoltage|REAL|0||0
77|rxCheckPercent|REAL|0||0
78|signal1|REAL|0||0
79|signal2|REAL|0||0
80|signal3|REAL|0||0
81|signal4|REAL|0||0
82|signal5|REAL|0||0
83|signal6|REAL|0||0
84|signal7|REAL|0||0
85|signal8|REAL|0||0
86|snow|REAL|0||0
87|snowBatteryStatus|REAL|0||0
88|snowDepth|REAL|0||0
89|snowMoisture|REAL|0||0
90|snowRate|REAL|0||0
91|so2|REAL|0||0
92|soilMoist1|REAL|0||0
93|soilMoist2|REAL|0||0
94|soilMoist3|REAL|0||0
95|soilMoist4|REAL|0||0
96|soilTemp1|REAL|0||0
97|soilTemp2|REAL|0||0
98|soilTemp3|REAL|0||0
99|soilTemp4|REAL|0||0
100|supplyVoltage|REAL|0||0
101|txBatteryStatus|REAL|0||0
102|UV|REAL|0||0
103|uvBatteryStatus|REAL|0||0
104|windBatteryStatus|REAL|0||0
105|windchill|REAL|0||0
106|windDir|REAL|0||0
107|windGust|REAL|0||0
108|windGustDir|REAL|0||0
109|windrun|REAL|0||0
110|windSpeed|REAL|0||0

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/d07cd082-dd2c-42bf-bef0-a051241b5388n%40googlegroups.com.

--

-Ryan Stasel

[Matthias Manhart]

I run weewx 5.0 on a test Raspberry Pi 4 since a few months and made the upgrades continuously. I'm now on 5.0.0rc1. I used the installation with pip and the virtual environment.

I have tried to execute the command "weectl database rebuild-daily", but i get the error message "No module named 'user'".

(weewx-venv) pi@meteo:~ $ weectl database rebuild-daily
Using configuration file /home/pi/weewx-data/weewx.conf
Traceback (most recent call last):
  File "/home/pi/weewx-venv/bin/weectl", line 8, in <module>
    sys.exit(main())
  File "/home/pi/weewx-venv/lib/python3.9/site-packages/weectl.py", line 66, in main
    namespace.func(namespace)
  File "/home/pi/weewx-venv/lib/python3.9/site-packages/weectllib/__init__.py", line 96, in dispatch
    namespace.action_func(config_dict, namespace)
  File "/home/pi/weewx-venv/lib/python3.9/site-packages/weectllib/database_cmd.py", line 328, in rebuild_daily
    weectllib.database_actions.rebuild_daily(config_dict,
  File "/home/pi/weewx-venv/lib/python3.9/site-packages/weectllib/database_actions.py", line 85, in rebuild_daily
    manager_dict = weewx.manager.get_manager_dict_from_config(config_dict, db_binding)
  File "/home/pi/weewx-venv/lib/python3.9/site-packages/weewx/manager.py", line 886, in get_manager_dict_from_config
    manager_dict['schema'] = weeutil.weeutil.get_object(schema_name)
  File "/home/pi/weewx-venv/lib/python3.9/site-packages/weeutil/weeutil.py", line 1404, in get_object
    module = importlib.import_module(module_name)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'user'

Any idea what is wrong with my installation or weewx ?

[Tom Keffer]

If you don't have an ET column in the database, but request a plot of one in [ImageGenerator], then the image generator will calculate ET in software using data in the database. That's likely to be expensive.

Running "weectl station reconfigure" will not add it back to the database. You need "weectl database add-column", followed by "weectl database calc-missing". 

Keep track of what you're doing, including report run times. It will be very interesting to see if it makes a big difference.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CALSG5j46SfviYdb3Ob8QX59wWzKpk_edYV8cH3QK_%3DmXvi_Zag%40mail.gmail.com.

[Tom Keffer]

Matthias, what do you get for the following:

ls -l /home/pi/weewx-data

ls -l /home/pi/weewx-data/bin

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/16d8970b-5359-4726-9b54-ee571a8c41bcn%40googlegroups.com.

[Cameron D]

Matthias,

I had that problem with rc1-1 and it went away with rc1-2.  I notice we are now at rc1-4

[Matthias Manhart]

This is the output:

(weewx-venv) pi@meteo:~ $ ls -l /home/pi/weewx-data
insgesamt 112
drwxr-xr-x  2 pi pi  4096  2. Sep 20:33 archive
drwxr-xr-x  3 pi pi  4096  2. Sep 11:57 bin
drwxr-xr-x 18 pi pi  4096  2. Sep 11:57 docs
drwxr-xr-x  9 pi pi  4096  2. Sep 11:57 examples
drwxr-xr-x 15 pi pi  4096 29. Dez 14:13 public_html
drwxr-xr-x 11 pi pi  4096  2. Sep 17:51 skins
drwxr-xr-x 17 pi pi  4096  2. Sep 11:57 util
-rw-r--r--  1 pi pi 82219 28. Okt 10:26 weewx.conf
(weewx-venv) pi@meteo:~ $ ls -l /home/pi/weewx-data/bin
insgesamt 4
drwxr-xr-x 4 pi pi 4096  2. Sep 22:21 user
(weewx-venv) pi@meteo:

[Matthias Manhart]

How can i upgrade to rc1-4 ? I made the pip-upgrade 2h ago and i can see only the release "5.0.0rc1" in my installation.

[Cameron D]

maybe they were only released as deb packages, which is how I installed mine.

[matthew wall]

On Friday, December 29, 2023 at 8:20:25 AM UTC-5 Matthias Manhart wrote:

How can i upgrade to rc1-4 ? I made the pip-upgrade 2h ago and i can see only the release "5.0.0rc1" in my installation.

the rc1-2, rc1-3, and rc1-4 were for redhat and debian packages.

we just released 5.0.0rc2, which includes a few path/logging changes in the weewx code, as well as the platform-specific changes for redhat, debian, and suse.

matthias, please do a pip upgrade to 5.0.0rc2

cameron and other debian package users - the debian (and redhat and suse) packages are now smarter about how they handle previous installs.  in particular, if you were running weewx as a non-root user, the new package should respect your previous user and permissions.  if you were running weewx as root, the new package should maintain that.  if you do a new weewx install, it will run as the weewx user and weewx group (it will create these if they do not exist).

when you upgrade deb/rpm you will now get a 'paper trail' of weewx.conf files that should make it easy for you to see any changes in each weewx release.  the installer will not touch your existing weewx.conf file, and we work really hard to keep weewx backward-compatible with conf files (and a bit less so with skins).  but the upgrade process will leave a weewx.conf-OLD-NEW file that shows what your conf file *would* look like if you were to upgrade it.

skins and 'user' directory should be handled better now, both when migrating from a 4.x (or 3.x) installation, and when updating weewx in the future.  only the weewx.conf file and 'util' helpers will be under deb/rpm package control, so you *should* be able to use weectl to explicitly update skins in future weewx releases - otherwise your skins should remain untouched (as they would in a pip install).  (i'm still working on the vagrant and pytest plumbing to ensure this part of the behavior, so any testing in this area would be helpful)

apologies for so much rc churn.

m