OS and weewx upgrade - performance issues fixed

[Bob Atchley]

Raspbian BullsEye is end of life at the end of August  2026 (2 weeks time as I write this).
With the number of security issues being dug up by AI, doing nothing is a high risk strategy.

I decided to make the move to Ubuntu 26.04 on my RPi4 (giving a upgrade path going forward) and of course the latest weewx version (5.5.0).

The biggest hurdle is that weewx doesn't break, so not touched it for years and I have forgotten everything.
Great documentation and the Wiki are the saving graces.

On BullsEye I was running a very early (beta) release of weewx 5.0 using the old database schema and just using the standard Seasons Skin,
I realised I had a serious performance issue after the upgrade:

Before:
user.log.1:Aug  8 23:25:21 rpi4 weewx[965] INFO weewx.imagegenerator: Generated 10 images for report SeasonsReport in 0.74 seconds
user.log.1:Aug  8 23:30:21 rpi4 weewx[965] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.28 seconds

After:
026-08-14T14:37:47.234310+00:00 rpi4 weewxd[5648]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 144.66 seconds
2026-08-14T14:43:52.117916+00:00 rpi4 weewxd[5648]: INFO weewx.imagegenerator: Generated 10 images for report SeasonsReport in 364.88 seconds

so 4 seconds to write the reports before, 500 seconds after, with 100% CPU usage.  Disappointing.

Reading between the lines more use is being made of the extended database schema, causing a large overhead if still using the old schema.
So I followed the Wiki:
https://github.com/weewx/weewx/wiki/Switching-to-the-new-wview_extended-schema

This solved the performance issues:
2026-08-14T18:40:28.340408+00:00 rpi4 weewxd[7306]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 5.37 seconds
2026-08-14T18:40:29.342990+00:00 rpi4 weewxd[7306]: INFO weewx.imagegenerator: Generated 10 images for report SeasonsReport in 1.00 seconds

This is perfectly acceptable performance.

The downside is my database when backed up is now 1.4GB (95MB gzipped) - basically double the size from before.

My backup strategy has been to use google cloud (via rclone) but I was getting bandwidth limit errors before this increase of size.
Still need to work out how to fix this.

Hope this helps anyone else doing something similar

Cheers

Bob

[Tom Keffer]

Have you tried the VACUUM command?

Also, you did not need to switch to the wview-extended schema, just add columns for any derived types. See Troubleshooting V5 performance issues.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/430e8248-f7a5-401d-8fb1-d3807c2baed5n%40googlegroups.com.

[Bob Atchley]

Thanks

I should have said i'm using mariadb ... I think that has an OPTIMISE TABLE equivalent to VACUUM, but no I have not used it (I'll experiment on an off line database).

It wasn't clear to me what extra derived types I needed, and the extended schema should give me future compatibility ... so no regrets other working out how I will back it up.

[Tom Keffer]

Ah! I see.

That does not strike me as a particularly large MariaDB database, but I can't claim to be an expert.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/392bae75-b470-4027-b088-0064ec27bc4an%40googlegroups.com.

[John Smith]

Before:
user.log.1:Aug  8 23:25:21 rpi4 weewx[965] INFO weewx.imagegenerator: Generated 10 images for report SeasonsReport in 0.74 seconds
user.log.1:Aug  8 23:30:21 rpi4 weewx[965] INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.28 seconds

After:
026-08-14T14:37:47.234310+00:00 rpi4 weewxd[5648]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 144.66 seconds
2026-08-14T14:43:52.117916+00:00 rpi4 weewxd[5648]: INFO weewx.imagegenerator: Generated 10 images for report SeasonsReport in 364.88 seconds

When there is constant excessive processing that's usually a sign of one or more columns missing in the database and weeWX has to calculate it every report cycle instead.

While the wview-extended schema does have a lot of columns, it doesn't have an exhaustive list either, and nor should it because not everyone needs the same ones.

To fix your problem you need to figure out what's in the template and as Tom suggested just add columns for those fields or otherwise it will keep taking a long time to generate reports.

[Pablo Sanchez]

Hi Bob,

You probably know this but if I want to find non-performant queries, I use the Slow Query Log. 

---
pablo

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/392bae75-b470-4027-b088-0064ec27bc4an%40googlegroups.com.