empty week images due to end of year?

[Michael Waldor]

Since (today?) all week images contain no measured data, only the forecast data are visible.

I have to admit that I've patched utilities.py to shift the right border of the timescale into the "future". It works fine all over the year. To my surprise all (measured) data is missing. If I drop my shift the data re-appears. But - as you see from the image above - the timescale looks fine and is exactly as intended. I've even debugged my modification for the current date, and everythind works as intended. But why are there no data from the past? Even from the above image one can see that imagegenerator.py can cross the year boundary perfectly.

And there IS data available within the past (I've checked this by dropping my change, and I've got a plot between 23. and 31.).

Why does weewx don't see the missing data? Images without forecast data aren't generated at all, e.g. weekUV.

For more details on my patch see https://groups.google.com/g/weewx-user/c/76qdbSy3bbQ/m/oIlk3sKrBgAJ

Regards, Michael

[Michael Waldor]

And since now I encounter the same problem with my day images. And the timescale is extended into the first hours of next year. Thus I'm quite confident that my problems will disappear as soon as the next year starts. 

To me it looks as if the access to weewx.sdb searches at a wrong time. Sadly I can't look deeper to debug that. There are no error messages within journalctl.

I'm using two sqlite DBs - the original weewx.sdb of course and a second one with forecast data. Maybe if the border of a year is crossed the weewx.sdb becomes unavailable? I can even report the time - it happens exactly at 14:00. Afterwards the timeaxis starts to go  till 3 o'clock in the night (till 14:00 it goes till midnight). And only forecast data is drawn (but containing data from today and tomorrow - no problem with year boundary). Even dayUV is no longer updated despite not using forecast data at all (but with the same timeaxis crossing the year boundary). 

[Tom Keffer]

Your modified version of scaletime() is surely causing the problem. Its semantics were never clear to me, and now it appears we're discovering the side effects. 

If you have a debugger, use it to step through running wee_reports. It will probably become evident what the problem is.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b4d802ee-c929-4ebb-aee9-e52c040a24c0n%40googlegroups.com.

[Michael Waldor]

I would agree, but why is the forecast data drawn fine and only the measured data not? My change to scaletime only calculates the time range - and that looks fine to me (see the included image within my first post). scaletime does return a triple of data - start, stop, and stepwidth. Works fine for forecast. AND works fine if there is no change of the year. I assume furthermore that scaletime does not use any global variables. But your program is very clean designed, thus I do not expect, that scaletime could create sideeffects. Tomorrow I'll know more... 

[Michael Waldor]

As expected all images are fine as soon as current time has reached the new year 2024. To be more precise - it has to be 5 min past midnight before the images are fine.

I've  tried to understand the logic within imagegenerator.py near its call of scaletime. Since the x-axis is labelled fine & day/night rendering works I assume that x_domain is calculated correctly.

There is a loop over all line_name - it might miss the entry for measured data. Here I should debug, but:

I've tried to reproduce my problems using e.g.

sudo wee_reports --date=2023-12-31 --time=22:00

Now it is generated perfectly as expected. Of course I do no longer have the same input data - the measurement data now is available for times within 2024, and the forecast data is more recent. Thus it's more difficult to reproduce the problem - I've to use older weewx.sdb (luckily I do have a recent backup)....

BTW Happy new year, Michael

[gjr80]

I suspect you will not get too much feedback of value until you post the exact code you are using along with the plot definition from skin.conf. The post you linked is very general in nature and it's not clear if the code you are using is that same as the PRs you submitted or otherwise. I am happy to have a look through your code but I don't want to be chasing my tail or going down the wrong path.

Gary

[Michael Waldor]

My code is available as a (rejected) pull request https://github.com/weewx/weewx/pull/906 - but the code itself works fine. It enhances the timescale some days into the future to plot forecast data. See my first post for a sample (only forecast data is plotted, measurement data are missing). I assume that my change has a sideeffect when weewx checks if there is data to be drawn. But I have no idea why my problem only arise for the measurement data. From the sample you clearly see that the forecast data contains data from 2023 and 2024, and is plotted. The measuretdata of course only contains data from the past. And even images where no forecast data is to be plotted (e. g dayUV) are missing completely, as soon as the timescale has parts in 2024 and current time is 2023.

And the problem is difficult to reproduce - as soon as the measurement data has reached 2024, even images generated for old times are fine. 

[gjr80]

Thank you, so that is the code, just need the plot definition. In fact, the entire [ImageGenerator] stanza from the config file in use would be best.

Gary

[Michael Waldor]

Thanks for your intrest in my problem. I do have two major changes within section [imagegenerator]. I've increased the image_width from 500 to 600. And I've added a second line within some images to include forcast data from a different database like e.g.

        [[[daybarometer]]]
            [[[[barometer]]]]
            [[[[forecast_barometer]]]]
                data_type = barometer
                data_binding = dwd_binding
label = ' '
color = blue

And to draw that forecast data I use my new option "time_offset = 36000  # 10 hours". It's intention is to use the additional 100pixel to be filled by forecast.

See my full skin.conf appended here.

Regards, Michael

[Michael Waldor]

I'm trying to setup my linux laptop as a degugging station for my weewx issue. I've copied all relevant files into my homedirectory resembling the weewx directory structure. Did the same for /var and the like. I enabled python3.11 to be /usr/bin/python. I also installed some python tools not commonly available within my linux distribution (e.g. pillow or cheetha). Finally my call to wee_reports is working nearly (and creates some files within var/www/html/weewx, but no images):

PYTHONPATH=MYROOT/weewx-debug/usr/share/weewx MYROOT/weewx-debug/usr/share/weewx/wee_reports --
config etc/weewx/weewx.conf --date 2023-12-10 --time=15:00

But I encounter the error

Traceback (most recent call last):
  File "MYROOT/weewx-debug/usr/share/weewx/weewx/reportengine.py", line 197, in run
    obj.start()
  File "MYROOT/weewx-debug/usr/share/weewx/weewx/reportengine.py", line 385, in start
    self.run()
  File "MYROOT/weewx-debug/usr/share/weewx/weewx/imagegenerator.py", line 42, in run
    self.gen_images(self.gen_ts)
  File "MYROOT/weewx-debug/usr/share/weewx/weewx/imagegenerator.py", line 114, in gen_images
    image = plot.render()
            ^^^^^^^^^^^^^
  File "MYROOT/weewx-debug/usr/share/weewx/weeplot/genplot.py", line 212, in render
    self._renderBottom(draw)
  File "MYROOT/weewx-debug/usr/share/weewx/weeplot/genplot.py", line 404, in _renderBottom
    bottom_label_size = draw.textsize(self.bottom_label, font=bottom_label_font)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "MYROOT/weewx-debug/usr/share/weewx/weeplot/genplot.py", line 646, in textsize
    return ImageDraw.ImageDraw.textsize(self, string, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'ImageDraw' has no attribute 'textsize'
Generating for requested time 2023-12-31 15:00:00 CET (1704031200)
Using configuration file etc/weewx/weewx.conf

I modified WEEWXROOT to point onto MYROOT.

I admit my current approach is a little bit unconventionally, but I seem to be near to a working debugging environment.

Any hint w.r.t. the above python error?

Regards, Michael

[michael.k...@gmx.at]

https://groups.google.com/g/weewx-user/search?q=pil%20attribute%20error

[Michael Waldor]

Thanks a lot for this hint - I wasn't aware that there are such complex dependencies among python versions  - its extensions - and weewx. Since I'm on tumbleweed, too, I couldn't simply switch the pillow version. After tweaking genplot.py I also had to manipulate utilities.py as described here https://groups.google.com/g/weewx-user/c/pOOAfh6ct-I

But now I have a working wee_reports within my test environment AND can reproduce my problem. Thus now I will start to debug ...

Regards, Michael

[Michael Waldor]

I could idenify the main reason for my problem. If the right border of the timeline has already reached the new year 2024 while the current date (last database entry) is still in 2023 that line is intentionally skipped triggered by skip_if_empty - see the the debugging of imagegenerator.py within the program "wee_reports  .... --date 2023-12-31 --time=15:00"

-> if skip:
(Pdb) print(line_name)
outHumidity
(Pdb) print(x_domain)
[2023-12-22 00:00:00 CET (1703199600) -> 2024-01-02 00:00:00 CET (1704150000)]  <---- as intended by my change
(Pdb) print(skip)
True
(Pdb) print(binding)
wx_binding
(Pdb) print(db_manager)
<weewx.manager.DaySummaryManager object at 0x7f22e1464650>
(Pdb) print(check_domain)
[2024-01-01 00:00:00 CET (1704063600) -> 2025-01-01 00:00:00 CET (1735686000)]    <-- unexpectedly checking the "wrong" year
(Pdb) exit

The x_domain correctly starts in 2023 and ends in 2024. But the check_domain only looks into 2024 - of course without data at end of 2023.

There is a workaround in setting skip_if_empty to false within skin.conf. By default within Seasons it is set to year.

But setting skip_if_empty to false triggers another problem for many observables like e.g. extTemp4. wee_reports crashes with the error message:

Using configuration file etc/weewx/weewx.conf

Traceback (most recent call last):

  File "/shared/purzel/miwaldor/weewx-debug/usr/share/weewx/weewx/reportengine.py", line 197, in run
    obj.start()
  File "/shared/purzel/miwaldor/weewx-debug/usr/share/weewx/weewx/reportengine.py", line 385, in start
    self.run()
  File "/shared/purzel/miwaldor/weewx-debug/usr/share/weewx/weewx/imagegenerator.py", line 42, in run
    self.gen_images(self.gen_ts)
  File "/shared/purzel/miwaldor/weewx-debug/usr/share/weewx/weewx/imagegenerator.py", line 107, in gen_images
    plot = self.gen_plot(plotgen_ts,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/purzel/miwaldor/weewx-debug/usr/share/weewx/weewx/imagegenerator.py", line 250, in gen_plot
    start_vec_t, stop_vec_t, data_vec_t = weewx.xtypes.get_series(
                                          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/purzel/miwaldor/weewx-debug/usr/share/weewx/weewx/xtypes.py", line 116, in get_series
    raise weewx.UnknownType(msg)
weewx.UnknownType: extraTemp4

The same error occurs for all those observables (really many, e.g. all extTemp? > 3) which are not within the weewx.sdb database. Maybe that's an ommission from my side. I'm using weewx for the last ~5 years and never did any database migration (except those maybe enforced by the apt installation).

Conclusion: I now have a working solution. I will drop all failing (and from me not used) observables and set check_if_empt to false. But maybe two changes should be applied to weewx-4.10.2 to resolve those problems for other users (did not investigate the upcoming 5.0 version of weewx).

Regards, Michael