Unrecoverable error "no such table: forecast_day_wind"

[Michael Waldor]

I seem to have a very specific problem due to my complex changes within weewx. First I'll shortly describe my problem, then I'll explain my weewx environment:

My problem: weewx 4.10.2 is working fine during daytimes, but imagegenerator fails daily between 0:00 and 5:15 all 5min with the same set of error messages, e.g.

Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine: Caught unrecoverable exception in generator 'weewx.imagegenerator.ImageGenerator'
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****  no such table: forecast_day_wind
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weedb/sqlite.py", line 39, in guarded_fn
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      return fn(*args, **kwargs)
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weedb/sqlite.py", line 233, in execute
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      return sqlite3.Cursor.execute(self, *args, **kwargs)
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****  sqlite3.OperationalError: no such table: forecast_day_wind
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****  During handling of the above exception, another exception occurred:
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 197, in run
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      obj.start()
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/reportengine.py", line 385, in start
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      self.run()
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 44, in run
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      self.gen_images(self.gen_ts)
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 109, in gen_images
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      plot = self.gen_plot(plotgen_ts,
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/imagegenerator.py", line 248, in gen_plot
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      start_vec_t, stop_vec_t, data_vec_t = weewx.xtypes.get_series(
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 100, in get_series
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      return xtype.get_series(obs_type, timespan, db_manager, aggregate_type,
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 920, in get_series
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      return ArchiveTable.get_series(obs_type, timespan, db_manager, aggregate_type,
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 167, in get_series
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      agg_vt = get_aggregate(obs_type, stamp, do_aggregate, db_manager)
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 127, in get_aggregate
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      return xtype.get_aggregate(obs_type, timespan, aggregate_type, db_manager,
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/xtypes.py", line 1102, in get_aggregate
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      row = db_manager.getSql(sql, timespan)
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weewx/manager.py", line 579, in getSql
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      _cursor.execute(sql, sqlargs)
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****    File "/usr/share/weewx/weedb/sqlite.py", line 47, in guarded_fn
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****      raise weedb.NoTableError(e)
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****  weedb.NoTableError: no such table: forecast_day_wind
Nov 13 00:00:33 imurr9 python3[579]: weewx[579] ERROR weewx.reportengine:         ****  Generator terminated

My weewx environment:

I'm using the current weewx version 4.10.2 with a highly customized Seasons skin on a raspberry pi 3 with current bullseye linux.

I also use the German DWD forecast module to get local forecasts, being stored within a separate sqlite DB (updated each hour by cron).

I want to see the forecast data within the same diagrams as the measured data, i.e. those diagrams contain data from 2 sqlite databases. Sadly there seeme to be no standard mechanism to extend the x-axis (time) into the "future". Thus I've tweaked the function scaletime within utities.py to include an optional parameter offset. This offset is forwarded from skin.conf to extend the x-axis by 10h (for daily reports) resp. 5days (for weekly reports).

This complicated setup works fine - except every neight between 0:00 and 5:15. I have noticed that exactly after 5:15 the forecast DB no longer contains data from the previous day, i.e. exactly after midnight the current time and the first data within forecast DB are from different dates.

At first I've asked the same question within the German GIT project https://github.com/roe-dl/weewx-DWD/issues/57, but that module does not access a table named forecast_day_wind. (The DWD sqlite table is named forecast).

Then I've tried to debug my code changes of scaletime by

cd /usr/share/weewx/weeplot
PYTHONPATH=/usr/share/weewx python3 -i utilities.py

and calling scaletime step by step at some critical dates after midnight. But my modifications to extend the time interval exactly work as intended (and I see completly correct images during daytime including all lnes from both sqlite DBs).

I've checked the created forecast DB - it's fine all over the day, especially nightly.

Of course I might post my changes if you ask for. But it would make my current post even more unreadable:-) . Here only 2 code lines from utilities.py (I do more)

def scaletime(tmin_ts, tmax_ts, offset=0):

...

    tmax_dt = datetime.datetime.fromtimestamp(tmax_ts + offset)

...

Any ideas?

[Tom Keffer]

Imagegenerator is trying to calculate a wind vector. This typically happens for a plot with a spec that looks something like this:

[[[weekwindvec]]]
    [[[[windvec]]]]
        plot_type = vector

Normally, it uses the "wx_binding" database binding to do this, which is normally pointed at the "archive" table. For whatever reason, your installation is attempting to do this with the "forecast" table. It could be that you are using the wrong binding, or it could be that the wx_binding is pointing to the "forecast" table.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2baf8c90-d283-4e06-9d33-1daa7a6f50can%40googlegroups.com.

[Michael Waldor]

Thanks for your quick reply. But why does my setup work fine after 5:15 all over the day? From my perspective a wrong configuration should fail always. 

[Tom Keffer]

Hard to say. Plots are regenerated only as often as their aggregation interval. Perhaps it is the "year" windvec plot that is failing: it would be regenerated only once a day.

Just guessing. You have a complicated configuration and we haven't seen your configuration files.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b39ac44e-25cf-4983-8f12-860eb479473bn%40googlegroups.com.

[Michael Waldor]

During this night I did save the weewx HTML output from 3 o'clock. And now I know what images are not updated - the weekwindvec diagram (and all week diagrams being rendered "afterwards").

The time axis of all diagrams are fine, i.e. my tweak of utilities.py seems NOT to be the cause of my problem.

Since the rendering of even the weekwindvec diagram  works fine if the current time is later then 5:15 I guess that there might be a hidden bug within imagegenerator.py itself. I have no idea howto debug that. I only know that imagegenerator fails for weekwindvec if the current time is between 0:00 and 5:15 - exactly during that time the forecast DB contains data from the previous day. But that doesn't disturb creation of daywindvec being fine at 3:00.

As a starter I give here the most relevant parts of the involved configuration files.

First weewx.conf:

    [[dwd_binding]]
        database = dwd_sqlite
        table_name = forecast
        manager = weewx.manager.Manager
        schema = schemas.dwd.schema

Next my skin.conf:

        [[[weekwindvec]]]
            [[[[windvec]]]]
                plot_type = vector

            [[[[forecast_windvec]]]]
                data_type = windvec
                data_binding = dwd_binding
                plot_type = vector
label = ' '
color = blue

BTW, within the original Seasons skin for week/month/year [[[wind]]] is used and should be replaced by [[[windDir]]]

And below you see the not updated diagram itself - it has timestamp 23:00, but should be 3:00. Here you also see my intention of merging data from two DBs - the "real" life data are in Seasons blue, and the forecast data are in dark blue. Keep in mind current date is 11-14, and you see data till 11-18 from forecast. There is a small overlap (not visible here) since the forecast data contains "old outdated" data of some hours - but I like that to compare outdated forecast with real data.

[Michael Waldor]

One additional hint: I've checked if there is forecast info available at the right border of my timeline. Sadly, it is - the forcast roughly covers 10 days into the future w.r.t. current date. And most other week diagrams are rendered correctly when weekwindvec is failing.

[Graham Eddy]

suggestions:

 (1) maybe the weekly graph is only being calculated in your 0-5 daily hour interval → really failing all the time. try hourly or daily graph to verify it works during those & other hours

 (2) maybe another deeper exception is happening which happens to be percolated up as sqlError that reports the last sqlite error not this error

⊣GE⊢

[Michael Waldor]

I believe that your item (1) can't be my problem because all diagrams have the correct (and current) timestamp below their time axis. Within my failing sample diagram in my previous post you can see the timestamp 23:00 where it should be 03:00. And of course after 5:15 even that diagram is updated and marked with the correct and current timestamp. To enforce recreation of monrh/year diagrams I intentonally cleared all diagrams within /var/www/html/weewx, and weewx creatwd all diagrams during next report generation cycle. 

[Tom Keffer]

First weewx.conf:

    [[dwd_binding]]
        database = dwd_sqlite
        table_name = forecast
        manager = weewx.manager.Manager
        schema = schemas.dwd.schema

You are using the database manager weewx.manager.Manager for the "forecast" table. That manager will not include daily summaries, which is why you are getting the error that the daily summary table forecast_day_wind is missing. Is this intentional?

Nevertheless, it should be possible to calculate the wind vectors without the daily summaries. It would be frightfully slow, but it would work. 

Two questions:

1. What does the schema schemas.dwd.schema look like?

2. What is the actual schema used in the database used by forecast? You didn't give enough information for me to determine the path to the actual database, so you'll have to do that.

echo .schema | sqlite3 path-to-the-forecast-database

-tk

[Michael Waldor]

I have to admit that I do not understand the details howto setup sqlite databases within weewx - I've tried to follow the recipe for DWD forecasts as given within https://github.com/roe-dl/weewx-DWD , and I did retain all database settings as given by standard delivery weewx.conf (I do a manual diff with my private copy of weewx.conf each time when I update weewx to a new release).

Here the full part w.r.t. sqlite from my weewx.conf (comments dropped)

[DataBindings]
     [[wx_binding]]
         database = archive_sqlite
         table_name = archive
         manager = weewx.manager.DaySummaryManager
         schema = schemas.wview_extended.schema

    [[dwd_binding]]
        database = dwd_sqlite
        table_name = forecast
        manager = weewx.manager.Manager
        schema = schemas.dwd.schema

[Databases]
    [[archive_sqlite]]
        database_name = weewx.sdb
        database_type = SQLite

    [[dwd_sqlite]]
        database_name = dwd-forecast-K2714.sdb
        database_type = SQLite
   

And the forecast schema /usr/share/weewx/schemas/dwd.py contains

schema = [('dateTime','INTEGER NOT NULL PRIMARY KEY'),
          ('usUnits','INTEGER NOT NULL'),
          ('interval','INTEGER NOT NULL')]

The sqlite DB itself is stored at /var/lib/weewx/dwd-forecast-K2714.sdb side by side with weewx.sdb

echo .schema | sqlite3 /var/lib/weewx/dwd-forecast-K2714.sdb  returns

CREATE TABLE forecast (dateTime INTEGER NOT NULL PRIMARY KEY,usUnits INTEGER NOT
NULL,interval INTEGER NOT NULL,hour INTEGER,outTemp REAL,dewpoint REAL,outHumid
ity REAL,windchill REAL,heatindex REAL,windDir REAL,windSpeed REAL,windGust REAL
,pop REAL,cloudcover REAL,barometer REAL,rain REAL,rainDur REAL,sunshineDur REAL
,visibility REAL,ww INTEGER);

[Karen K]

@Tom Keffer: I would like to add that the table dwd-forecast-K2714.sdb is created outside WeeWX by a separate program. It creates the core table only, no daily summaries. That is the reason, weewx.manager.Manager is used. I use it myself, and I never got an error message like that the thread starter experiences.

dwd-forecast-K2714.sdb has another interval than weewx.sdb: For dwd-forecast-K2714.sdb it is always an hour, for weewx.sdb it is 300 seconds, I guess. This may be an issue if mixing them within the same plot.

[Tom Keffer]

I don't know why this is working for Karen, but not Michael.

In any case, try this version of xtypes.py. It explicitly tests whether a daily summary is available when calculating wind vectors.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7c1b30f5-638f-4d17-96e9-3a60e0559b03n%40googlegroups.com.

[Michael Waldor]

Thanks for your fix. It's now activated within my environment (I did restart weewx afterwards).

Tomorrow I'll report back...

[Karen K]

Tom Keffer schrieb am Freitag, 17. November 2023 um 14:43:37 UTC+1:

I don't know why this is working for Karen, but not Michael.

In any case, try this version of xtypes.py. It explicitly tests whether a daily summary is available when calculating wind vectors.

I guess, because I do not plot wind vectors or because I do not mix measurements with forecast data.

 

[Michael Waldor]

Heureka - your fix within xtypes.py did the trick. I no longer get errors during the night. Sadly, I forgot to store nightly copies of the weewx HTML output, thus I currently can't confirm if the weekwindvec is really created. But I'm quite sure:-) 

Thank you very much, Michael

[Michael Waldor]

Just as a final conclusion - during this night the weekwindvec diagram has been created also at 3:00 as expected (no longer error messages within journalctl).

Its timestamp 03:00:00 proves it (to be compared with 23:00:00 some posts ago without the xtypes.py fix).