Reports don't show old data anymore.

[Zed]

I had to upgrade my VEnv-installation from 4.9.1 to 5.2.0.
The data is stored in mariadb since ever.
Now in the reports i see the the values since the upgrade but none before. Not in the view week, month or year.
The database backup contains several hundreds of thousand of records. So i guess the data is still there.
What causes the old data to be hidden? 

S. issues #1043

[John Smith]

You didn't configure weeWX for MariaDB

Added record 2025-12-01 23:05:00 CET (1764626700) to database 'weewx.sdb'

Specifically this section:

[DataBindings]
    [[wx_binding]]
        database = archive_sqlite
        table_name = archive
        manager = weewx.manager.DaySummaryManager
        schema = weewx.schemas.wview_extended.schema

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/b1f919c7-439a-42f8-b952-1e96fc5a5366n%40googlegroups.com.

[Zed]

@John Smith You made my day! Looks like i'm getting old ;-)

[Zed]

Hm, now i'm suffering:

"INFO weewx.engine: Launch of report thread aborted: existing report thread still running"

How can i figure out, what's wrong?

[Tom Keffer]

It's possible your reports are taking too long to process. Easy to do with a RPi.

But it's hard to say without seeing a log.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/c779e7ed-b1b6-4a1d-97eb-31abdb5406a5n%40googlegroups.com.

[vince]

If it broke when you switched from (local) sqlite to (remote)  mariadb, you likely have network or configuration issues in your mariadb setup between the weewx system and that other system.

What kind of pi are you running on ?  What kind of system is running the db ?   Are you seeing errors on the remote db server ?

As Tom said, we need to see your 'current' logs from weewx startup through 'two' full report cycles.

You might also attach the 'current' output of "weectl diag --output=filename" after checking the output file has no usernames/passwords in it.  Weectl is pretty good about obfuscating such things but it can't be perfect in all cases.

[Zed]

@vince 
attached please find the diag-log.
WeeWX is running on a RPI-4b 4GB. Right now the load is high because i'm migrating data. 
Normaly is works perfectly well since years.

[vince]

Need more information.

• What kind of system is running the db ?
  

• Where is this db system located - on your LAN ?  On internet ?

• Are you seeing errors on the remote db server ?
  

• why do you need to use mariadb at all ?
  

And also...

• We need to see your 'current' logs from weewx startup through 'two' full report cycles.

If it was ok using sqlite3 as your db, and it is not ok using mariadb as your db, the issue is likely a mariadb/weewx misconfiguration or a network issue.

If you (temporarily) switch back to sqlite3 and it is then ok again, it is 'very' likely you have a mariadb/weewx misconfiguration or a network issue.

[Zed]

The db is running on another RPI-4b 4GB in the home LAN.
No errors in the db-log.
I like mariadb. This worked perfectly for years.
Right now the system load is high as i am migrating data. I will check again as soon as the load settels down again ...

[vince]

If you are not going to provide the logs we requested, then I am going to give up trying here.  Best of luck.

[Tom Keffer]

Let us know what you find. I'm betting that there was too big a load for your RPi.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/222b6e08-be18-4d64-99d2-af7c247186dfn%40googlegroups.com.

[John Smith]

Why didn't you copy your old config when you updated, or did it get deleted somehow and there is no backup of it?

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/222b6e08-be18-4d64-99d2-af7c247186dfn%40googlegroups.com.

[John Smith]

Is this line correct and the hostname listed in /etc/hosts or a LAN resolver?

host = mariadb.local

[Zed]

Attached please find the requested log.

When the service starts everything looks well and the system-load is low.

As soon as the first report is created, the process "/home/weewx/weewx-venv/bin/python3 /home/weewx/weewx-venv/lib/python3.13/site-packages/weewxd.py /home/weewx/weewx-data/weewx.conf" consumes 100% CPU of one kernel. It stays at this level and the system-load rises to ~1.8.

Please note the messages:
"WARNING weewx.engine: Previous report thread has been running 600.9363927841187 seconds.  Launching report thread anyway."

I guess 
"ERROR weewx.cheetahgenerator: Evaluation of template /home/weewx/weewx-data/skins/Seasons/index.html.tmpl failed with exception '<class 'FileNotFoundError'>'"

and the following messages are a result of a leftover of the pending thread.

Is there a way to monitor these  report threads and gather more information?

Allover the configuration is as before and works well.

I should mention, that i just migrated the system to Debian 13 Trixie.
Maybe this is relevant.

[John Smith]

If your system is doing that much work, are you missing DB columns to store previously calculated values that templates are asking for?

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/abff0369-6907-42f6-8a4b-98590b546bfdn%40googlegroups.com.

[Zed]

Hi John,
how can i check the columns? Attached please find the db-columns currently defined.

[John Smith]

On Fri, 5 Dec 2025 at 00:38, Zed <welf.t...@gmail.com> wrote:

how can i check the columns? Attached please find the db-columns currently defined.

I have no direct experience with using 3rd party templates, but you'd have to look for things in the template(s) that have no corresponding columns in the DB, especially for things like monthly or yearly stats etc...

When debugging things in general you should disable everything, and then turn features back on one by one until you hit the same behaviour.

[Zed]

Is the report generation dependent from the database layout?
What makes you think,  3rd party templates are in use?

There is a  difference  in "schema" in weewx.conf:
Before  (v4.9.1):

[DataBindings]
    [[wx_binding]]
        database = archive_mysql

        table_name = archive
        manager = weewx.manager.DaySummaryManager

        schema = schemas.wview_extended.schema

Now (v5.2.0):

[DataBindings]
    [[wx_binding]]
        database = archive_mysql

        table_name = archive
        manager = weewx.manager.DaySummaryManager
        schema = weewx.schemas.wview_extended.schema

Could this cause this issue?

[John Smith]

On Fri, 5 Dec 2025 at 01:28, Zed <welf.t...@gmail.com> wrote:

Is the report generation dependent from the database layout?

Getting derived values from a DB is a lot faster than weeWX continuously deriving or calculating them.

 

What makes you think,  3rd party templates are in use?

Cause the standard or extended schemas include all possible columns needed.

 

There is a  difference  in "schema" in weewx.conf:
Before  (v4.9.1):

[DataBindings]
    [[wx_binding]]
        database = archive_mysql
        table_name = archive
        manager = weewx.manager.DaySummaryManager
        schema = schemas.wview_extended.schema

Now (v5.2.0):

[DataBindings]
    [[wx_binding]]
        database = archive_mysql
        table_name = archive
        manager = weewx.manager.DaySummaryManager
        schema = weewx.schemas.wview_extended.schema

Could this cause this issue?

No, it's a problem with the report generation, because it's not finished by the time another report generation starts making the problem even worst by having 2 or more reports running in parallel.

Not having derived or calculated columns in the database will cause weeWX to regenerate them during every report cycle rather than just reading the previously calculated and stored data.

All this would be triggered by a template asking for data that's not in the database.

[Tom Keffer]

Yes, reports that use derived types can be slow if the type is not in the database. The Belchertown skin has this problem because of its use of type 'appTemp'. However, the Seasons skin does not do this. (Unless, of course, you extended it.)

The declaration for schema

        schema = schemas.wview_extended.schema

does not make any difference. It's only used when the database is created.

What type of Raspberry Pi are you using? I've used everything from an RPi Zero on with no problem, but it will be slow.

These performance problems can be difficult to diagnose. I would suggest commenting out the template declarations in Seasons/skin.conf one by one until you find the culprit.

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/4ad2d41d-1343-4daa-9443-4bf567c2435fn%40googlegroups.com.

[Zed]

The differences in Seasons/skin.conf are:

8c8
< SKIN_VERSION = 5.2.0
---
> SKIN_VERSION = 4.9.1
29c29
<     # If you have a Google Analytics GA4 tag, uncomment and edit the next line, and
---
>     # If you have a Google Analytics ID, uncomment and edit the next line, and
31c31
<     #googleAnalyticsId = G-ABCDEFGHI
---
>     #googleAnalyticsId = UA-12345678-1
269c269
<         time_length = 27h
---
>         time_length = 97200 # 27 hours
350c350
<                 aggregate_interval = 1h
---
>                 aggregate_interval = hour
359c359
<                 aggregate_interval = 1h
---
>                 aggregate_interval = hour
384c384
<                 aggregate_interval = 1h
---
>                 aggregate_interval = hour
410c410
<         aggregate_interval = 1h
---
>         aggregate_interval = hour
488c488
<                 aggregate_interval = 1d
---
>                 aggregate_interval = day
496c496
<                 aggregate_interval = 1d
---
>                 aggregate_interval = day
521c521
<                 aggregate_interval = 1d
---
>                 aggregate_interval = day
546c546
<         aggregate_interval = 3h
---
>         aggregate_interval = 10800 # 3 hours
625c625
<                 aggregate_interval = 1d
---
>                 aggregate_interval = day
633c633
<                 aggregate_interval = 1d
---
>                 aggregate_interval = day
658c658
<                 aggregate_interval = 1d
---
>                 aggregate_interval = day
683c683
<         aggregate_interval = 1d
---
>         aggregate_interval = day
762c762
<                 aggregate_interval = 1w
---
>                 aggregate_interval = week
770c770
<                 aggregate_interval = 1w
---
>                 aggregate_interval = week
795c795
<                 aggregate_interval = 1w
---
>                 aggregate_interval = week

[vince]

If your weewx was fast with v4 and it is very slow with v5, you are very likely trying to display something in a skin that is calculated yet not an element in the db.  This is a change that occurred in v5.

The schema definition in the db only creates new columns if you are 'creating' a 'new' database, if I remember correctly.

• Your database has 53 items in the archive record.  That means you are using the old 'small' schema.
• My db has 114 items in the archive record. That means I am using the newer wview-extended schema.

Your db is missing the following elements that were added in the wview-extended schema.

appTemp appTemp1
batteryStatus1 batteryStatus2 batteryStatus3 batteryStatus4 batteryStatus5 batteryStatus6 batteryStatus7 batteryStatus8
cloudbase
co
co2
dewpoint1
extraHumid3 extraHumid4 extraHumid5 extraHumid6 extraHumid7 extraHumid8
extraTemp4 extraTemp5 extraTemp6 extraTemp7 extraTemp8
forecast
hailBatteryStatus
heatindex1
humidex humidex1
illuminance
inDewpoint
lightning_distance lightning_disturber_count lightning_energy lightning_noise_count lightning_strike_count luminosity
maxSolarRad
nh3
no2
noise
o3
pb
pm1_0 pm10_0 pm2_5
signal1 signal2 signal3 signal4 signal5 signal6 signal7 signal8
snow snowBatteryStatus snowDepth snowMoisture snowRate
so2
uvBatteryStatus

windrun 

[Zed]

Thank you vince. That makes sense.
These are the columns of table archive?

Are you using mariadb/mysql? Could you please post the result of "describe archive;" then.

Could somebody please post a script to upgrade a mariadb from v4 to v5?
Is there a migration guide describing this step?
Is it sufficient to add these columns?

[Zed]

I guess "weectl database reconfigure --binding=wx_binding -y" will do this job.
But "weectl database reconfigure --binding=wx_binding --dry-run -y" fails:

Using configuration file /home/weewx/weewx-data/weewx.conf
This is a dry run. Nothing will actually be done.
Copying database 'weewx' to 'weewx_new'
The new database will use the same unit system as the old ('METRICWX').
Traceback (most recent call last):
  File "/home/weewx/weewx-venv/bin/weectl", line 7, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/weewx/weewx-venv/lib/python3.13/site-packages/weectl.py", line 67, in main
    namespace.func(namespace)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/weewx/weewx-venv/lib/python3.13/site-packages/weectllib/__init__.py", line 90, in dispatch
    namespace.action_func(config_dict, namespace)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/weewx/weewx-venv/lib/python3.13/site-packages/weectllib/database_cmd.py", line 371, in reconfigure_database
    weectllib.database_actions.reconfigure_database(config_dict,
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                                                    db_binding=namespace.binding,
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                    dry_run=namespace.dry_run,
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                    no_confirm=namespace.yes)
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/weewx/weewx-venv/lib/python3.13/site-packages/weectllib/database_actions.py", line 295, in reconfigure_database
    weewx.manager.reconfig(manager_dict['database_dict'],
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           new_database_dict,
                           ^^^^^^^^^^^^^^^^^^
                           new_unit_system=target_unit_system,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           new_schema=manager_dict['schema'],
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           dry_run=dry_run)
                           ^^^^^^^^^^^^^^^^
  File "/home/weewx/weewx-venv/lib/python3.13/site-packages/weewx/manager.py", line 711, in reconfig
    record_generator = weewx.units.GenWithConvert(old_archive.genBatchRecords(),
                       ^^^^^
UnboundLocalError: cannot access local variable 'weewx' where it is not associated with a value

[Zed]

"weectl database reconfigure --binding=wx_binding -y" (without " --dry-run") fails with the same message.

[Zed]

Just found this conversion. Looks like the manager.py fixes this issue.

[Zed]

Ops,  "this conversation".

[Zed]

Solved!
Thanks to everbody for pushing me in the right direction!

But two more questions:
" weectl database reconfigure ..." created a db weewx_new containing just a single table named "archive". Whereas the old db had 52 tables like:

archive                      
archive_day_ET                
archive_day_UV                
archive_day__metadata        
archive_day_altimeter        
archive_day_barometer        
archive_day_consBatteryVoltage
...

Does this mean all tables but archive are obsolete?

"weectl database check --binding=wx_binding" returns:

Using configuration file /home/weewx/weewx-data/weewx.conf
Checking daily summary tables version...
Daily summary tables are at version 4.0.
Interval Weighting Fix is not required.

I'd expected "... version 5.x".

[John Smith]

On Fri, 5 Dec 2025 at 10:40, Zed <welf.t...@gmail.com> wrote:

Does this mean all tables but archive are obsolete?

No, the archive table holds the data, the other tables are created to cache daily summaries to speed up report generation times, if you now only have the archive table you probably need to run weectl database rebuild-daily

[Zed]

"weectl database rebuild-daily --binding=wx_binding -y" returns:

Using configuration file /home/weewx/weewx-data/weewx.conf
All daily summaries will be rebuilt.
Rebuilding daily summaries in database 'weewx' ...
Daily summaries up to date in 'weewx'.

Whereas "weectl database check --binding=wx_binding" returns:

Using configuration file /home/weewx/weewx-data/weewx.conf
Checking daily summary tables version...
Daily summary tables are at version 4.0.
Interval Weighting Fix is not required.

Looks like this didn't work. Do i miss something?

[John Smith]

On Fri, 5 Dec 2025 at 21:07, Zed <welf.t...@gmail.com> wrote:

Daily summary tables are at version 4.0.

I don't think this is the same as the weeWX version.

Did the daily summary tables get created after all that? 

[Zed]

As mentioned before  "weectl database reconfigure ..." created just a single table "archive" which i moved from weewx_new to weewx.
All other tables are left as before.

[vince]

The "Daily summary tables are at version 4.0" has nothing to do with the weewx version.  Just ignore it.

To reconfigure your database you need to do the following (tested with sqlite3):

# stop weewx

sudo systemctl stop weewx

# edit weewx.conf to set "schema = weewx.schemas.wview_extended.schema"

# reconfigure the db to use the specified schema

weectl database reconfigure

# copy the new database into place

# create and populate the summary tables

weectl database rebuild-daily

# start weewx

sudo systemctl start weewx

You should have 115 tables when complete.

I don't know the mariadb command to do this, but for sqlite3

echo ".tables" | sqlite3 weewx.sdb | wc -w

[Zed]

Thank you vince.

i rebuild the daily-tables using

weectl database drop-daily --binding=wx_binding -y

weectl database rebuild-daily --binding=wx_binding -y

weectl database check --binding=wx_binding

still reports version 4.0. But anyway!

WeeWx is running well again!

[vince]

As multiple people replied, the version 4.0 of the tables is DIFFERENT from the weewx version. Simply stop worrying about it.

[John Kline]

Perhaps this will help.

Version 4.0 is the LATEST version of DaySummaryManager.  See the very latest copy of weewx/manager.py or look at the snippet from that file below.

Note the lat line in the snippet: version = "4.0"

class DaySummaryManager(Manager):

    """Manage a daily statistical summary.

            

    The daily summary consists of a separate table for each type. The columns of each table are

    things like min, max, the timestamps for min and max, sum and sumtime. The values sum and

    sumtime are kept to make it easy to calculate averages for different time periods.

            

    For example, for type 'outTemp' (outside temperature), there is a table of name

    'archive_day_outTemp' with the following column names: 

                          

        dateTime, min, mintime, max, maxtime, sum, count, wsum, sumtime

            

    wsum is the "Weighted sum," that is, the sum weighted by the archive interval. sumtime is the

    sum of the archive intervals.

            

    In addition to all the tables for each type, there is one additional table called

    'archive_day__metadata', which currently holds the version number and the time of the last

    update.

    """ 

            

    version = "4.0"

On Dec 5, 2025, at 2:59 PM, vince <vince...@gmail.com> wrote:

As multiple people replied, the version 4.0 of the tables is DIFFERENT from the weewx version. Simply stop worrying about it.

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/f657678c-d476-4574-b9e0-693ffd273ee3n%40googlegroups.com.