Trying to troubleshoot v4 alpha4 reports

[Tom Mitchell]

Playing around with the v4 alpha inside a Docker container. Polling works well talking to a VP2, database works well. I am struggling with getting any reports to work with my config.

Here is what I know.

If I use the default weewx.conf that ships with the alpha, it complains about logging but does populate public_html/

I visually inspected and diff'd the default weewx.conf with mine, and they seem the same in the places that matter.

But the reporting process that runs every five minutes fails:

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.drivers.vantage: DMPAFT complete: page timestamp 2019-10-18 08:50:00 EDT (1571403000) less than final timestamp 2019-10-27 12:10:00 EDT (1572192600)

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.drivers.vantage: Catch up complete.

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Running reports for latest time in the database.

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Running report 'SeasonsReport'

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.drivers.vantage: Requesting 200 LOOP packets.

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Found configuration file /home/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'

2019-10-27 12:15:25  weewx[39]/ERROR weewx.reportengine:         ****  Report ignored

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Running report 'SmartphoneReport'

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Found configuration file /home/weewx/skins/Smartphone/skin.conf for report 'SmartphoneReport'

2019-10-27 12:15:25  weewx[39]/ERROR weewx.reportengine:         ****  Report ignored

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Running report 'MobileReport'

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Found configuration file /home/weewx/skins/Mobile/skin.conf for report 'MobileReport'

2019-10-27 12:15:25  weewx[39]/ERROR weewx.reportengine:         ****  Report ignored

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Running report 'StandardReport'

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Found configuration file /home/weewx/skins/Standard/skin.conf for report 'StandardReport'

2019-10-27 12:15:25  weewx[39]/ERROR weewx.reportengine:         ****  Report ignored

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Running report 'FTP'

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Found configuration file /home/weewx/skins/Ftp/skin.conf for report 'FTP'

2019-10-27 12:15:25  weewx[39]/ERROR weewx.reportengine:         ****  Report ignored

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.reportengine: reportengine: Report 'RSYNC' not enabled. Skipping.

2019-10-27 12:15:25  weewx[39]/DEBUG weewx.drivers.vantage: Gentle wake up of console successful

LOOP:   2019-10-27 12:15:26 EDT (1572192926) appTemp: 37.26554343340425, barometer: 30.104, cloudbase: 1511.4556564232578, consBatteryVoltage: 4.77, dateTime: 1572192926, dayET: 0.0, dayRain: 0.23, dewpoint: 39.87799511173767, extraAlarm1: 0, extraAlarm2: 0, extraAlarm3: 0, extraAlarm4: 0, extraAlarm5: 0, extraAlarm6: 0, extraAlarm7: 0, extraAlarm8: 0, forecastIcon: 6, forecastRule: 63, heatindex: 41.2, humidex: 41.2, inDewpoint: 35.71774930009776, inHumidity: 58.0, insideAlarm: 0, inTemp: 49.9, leafWet4: 0.0, maxSolarRad: 502.17728515573157, monthET: 1.27, monthRain: 4.7, outHumidity: 95.0, outsideAlarm1: 0, outsideAlarm2: 0, outTemp: 41.2, radiation: 26.0, rain: 0.0, rainAlarm: 0, rainRate: 0.09, soilLeafAlarm1: 0, soilLeafAlarm2: 0, soilLeafAlarm3: 0, soilLeafAlarm4: 0, stormRain: 0.23, stormStart: 1572148800, sunrise: 1572175632.0, sunset: 1572214128.0, txBatteryStatus: 0, usUnits: 1, windchill: 41.2, windDir: 150.0, windGust: 3.0, windGustDir: 150.0, windSpeed: 3.0, windSpeed10: 2.0, yearET: 8.7, yearRain: 4.7

Why are they being skipped? Where should I look for more detail to see what I need to fix?

Should I be posting this to the developers group since it is on the alpha?

Any advice appreciated!

[Tom Mitchell]

More on this - some snippets from weewx.conf

[StdReport]

    # Where the skins reside, relative to WEEWX_ROOT

    SKIN_ROOT = skins

    # Where the generated reports should go, relative to WEEWX_ROOT

    HTML_ROOT = public_html

    # The database binding indicates which data should be used in reports.

    data_binding = wx_binding

    # Whether to log a successful operation

    log_success = True

    # Whether to log an unsuccessful operation

    log_failure = True

    # Each of the following subsections defines a report that will be run.

    # See the customizing guide to change the units, plot types and line

    # colors, modify the fonts, display additional sensor data, and other

    # customizations. Many of those changes can be made here by overriding

    # parameters, or by modifying templates within the skin itself.

    [[SeasonsReport]]

        # The SeasonsReport uses the 'Seasons' skin, which contains the

        # images, templates and plots for the report.

        skin = Seasons

        enable = true

    [[SmartphoneReport]]

        # The SmartphoneReport uses the 'Smartphone' skin, and the images and

        # files are placed in a dedicated subdirectory.

        skin = Smartphone

        enable = true

        HTML_ROOT = public_html/smartphone

    [[MobileReport]]

        # The MobileReport uses the 'Mobile' skin, and the images and files

        # are placed in a dedicated subdirectory.

        skin = Mobile

        enable = true

        HTML_ROOT = public_html/mobile

    [[StandardReport]]

        # This is the old "Standard" skin. By default, it is not enabled.

        skin = Standard

        enable = true

    [[FTP]]

        # FTP'ing the results to a webserver is treated as just another report,

        # albeit one with an unusual report generator!

        skin = Ftp

        # If you wish to use FTP, set "enable" to "true", then

        # fill out the next four lines.

        # Use quotes around passwords to guard against parsing errors.

        enable = true

        user = hidden

        password = "hidden"

        server = 192.168.1.xx    # The ftp server name, e.g, www.myserver.org

        path = /vtwx-dev-html     # The destination directory, e.g., /weather

        # Set to True for an FTP over TLS (FTPS) connection. Not all servers

        # support this.

        secure_ftp = False

        # To upload files from something other than what HTML_ROOT is set

        # to above, specify a different HTML_ROOT here.

        #HTML_ROOT = public_html

        # Most FTP servers use port 21

        port = 21

        # Set to 1 to use passive mode, zero for active mode

        passive = 1

        debug = 1

And

[Engine]

    [[Services]]

        # This section specifies the services that should be run. They are

        # grouped by type, and the order of services within each group

        # determines the order in which the services will be run.

        prep_services = weewx.engine.StdTimeSynch

        data_services = ,

        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate

        archive_services = weewx.engine.StdArchive, weewx.engine.StdPrint   # , user.raw.RawService

        restful_services = weewx.engine.StdPrint, weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS#, user.mqtt.MQTT

        #report_services = weewx.engine.StdPrint, weewx.engine.StdReport

        report_services = weewx.engine.StdReport, weewx.engine.StdPrint

[vince]

We might need to see your docker-compose.yml or bash script etc. you used to start up the container.

Grepping the sources I find the following in reportengine.py:

        # Iterate over each requested report

        for report in self.config_dict['StdReport'].sections:

            # Ignore the [[Defaults]] section

            if report == 'Defaults':

                continue

            # See if this report is disabled

            enabled = to_bool(self.config_dict['StdReport'][report].get('enable', True))

            if not enabled:

                log.debug("reportengine: Report '%s' not enabled. Skipping.", report)

                continue

            log.debug("reportengine: Running report '%s'", report)

            # Fetch and build the skin_dict:

            try:

                skin_dict = self._build_skin_dict(report)

            except SyntaxError:

                log.error("        ****  Report ignored")

                continue

[Tom Mitchell]

Thanks. So that suggests something wrong with the skin, which I have changed since 3.9.2. I will strip it down to a minimum (the defaults that are in v4a4 and see what happens.

The Docker container and startup are unchanged for a long time, and they seem to work, so I do not suspect them. Also, I started up the container and exec'd a shell on it and ran wee_reports manually with the config and it failed just like when I just docker run it (in fact, I am running my four 3.9.2 sites on Kubernetes).

I will play and report back, and thanks Vince for the pointer!

[Tom Mitchell]

OK, report back. I methodically backed this config off until I thought it resembled the default weewx, and it still did not work. However, one other configuration I had on hand did work, though it was minimal. The one I am working with is busy - it runs raw.py, Belchertown, mgtt, and more.

So I spent way too much time backing things off one by one until I thought everything was gone and it looked like the OOTB one. But it still did not work.

I said screw it and started with the one that worked and I will build it back step by step and will find the problem that way, eventually.

[Tom Mitchell]

I will add that when the time is right, it might be helpful to the developers when I reproduce this all again. I am happy to do that - I have all of this under version control. I just don't know when the time is right, meaning I would want to do this once the v4 software is thought to be ready for that. I would guess that's the first beta. but might be wrong.

[gjr80]

Just for future reference, if you get a fault like this could you provide more of the log, in particular the WeeWX startup through until the error occurs. The startup and what goes on in the lead up to the error gives a lot of useful contextual info.

Gary

[Tom Mitchell]

You bet. It all looked like usual to me leading up to the error point. Thanks for letting me know.

--
Tom Mitchell
t...@tom.org
Real Time Beverly, MA Weather
http://weather.tom.org

"The whole problem with the world
  is that fools and fanatics are
  always so certain of themselves,
  and wiser people so full of doubts."
  - Bertrand Russell

On Sun, Oct 27, 2019 at 4:11 PM gjr80 <gjrod...@gmail.com> wrote:

Just for future reference, if you get a fault like this could you provide more of the log, in particular the WeeWX startup through until the error occurs. The startup and what goes on in the lead up to the error gives a lot of useful contextual info.

Gary

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/Tvwo1N14pS0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/1d59e7ee-da86-427c-8d2e-583c1e85f10e%40googlegroups.com.

[vince]

On Sunday, October 27, 2019 at 1:18:11 PM UTC-7, Tom Mitchell wrote:

You bet. It all looked like usual to me leading up to the error point. Thanks for letting me know.

FWIW - I'm running 4.0.0a7 on a pi4 and it works fine on real gear.  Also works fine in VirtualBox/vagrant.  So it (probably) has to be something do with your particular configuration and/or your docker setup.  Might even be something as simple as permissions trying to read the skins directory.

[gjr80]

Agree it is likely a config issue, looking at what is reported the log somewhere in the following code (circa line 251 in reportengine.py) a SyntaxError exception is being thrown:

        # Now add on the [StdReport][[Defaults]] section, if present:
        if 'Defaults' in self.config_dict['StdReport']:
            merge_dict = copy.deepcopy(self.config_dict['StdReport']['Defaults'])
            weeutil.config.merge_config(skin_dict, merge_dict)

        # Inject any scalar overrides. This is for backwards compatibility. These options should now go
        # under [StdReport][[Defaults]].
        for scalar in self.config_dict['StdReport'].scalars:
            skin_dict[scalar] = self.config_dict['StdReport'][scalar]

        # Finally, inject any overrides for this specific report. Because this is the last merge, it will have the
        # final say.
        weeutil.config.merge_config(skin_dict, self.config_dict['StdReport'][report])

I would be looking carefully at the [StdReport] stanza, not just the reports but the defaults etc that come after.

Gary