MQTT Metric vs Metricwx conundrum

[Colin Larsen]

Hi all

I'm putting this out on the Weewx forum as I'm not sure if there is a simple answer within Weewx for this problem.

Down in NZ it is common to report rain in mm and wind speed in kp/hr

In publishing MQTT data for Pats Belchertown skin this is causing an issue whereby if I set MQTT to publish Metric I get wind speed and gust in kp/h ..... but rain is reported in CM. Pats skin looks at the group units which I have set to mm for rain and publishes the data, which is CM as he sees the MQTT data is Metric and I get 0.14 mm of rain instead of the 1.4mm it should be. Same for rain rate of course.

I can fix the rain by using Metricwx which publishes rain in mm - but then wind is in m/s so that reports wrong as the group units are kph.

Can anyone suggest a solution within Weewx to the conundrum?

Many thanks

Colin

[gjr80]

Colin,

US, METRIC or METRICWX merely defines the units used in the weeWX database, it really should have no bearing on the presentation of data. For example, in the weeWX Standard skin (and for that matter other skins that simply use the weeWX report system for generating reports) the weeWX tag system and config settings in skin.conf/weewx.conf determine what presentation units are used. The user can freely (and simply) change these settings to display whatever units he/she requires in the resulting reports. Of course once you step outside extensions that only use the weeWX reporting system things become a little more complex, chnaces are you cannot rely solely on the weeWX tag system. In my view it really behoves those that produce extensions to weeWX to provide a means of selecting the units displayed by their extension, this may be as simple as the tags and skin/weewx config as previously mentioned, or for more complex extensions that say use MQTT or some other external app/system then some other mechanism is required. I am not fully across Pat's Belchertown extension but given it is still relatively young and being actively developed I would hope Pat would include some sort of user selectable settings/config options that allow you to display in whatever units you wish irrespective of the underlying database units.

In getting back to your original question I suspect there is not too much that can be done by you to change things unless the Belchertown extension has some as yet unknown config settings to do what you want. Pat or other Belchertown users are the best placed to help with this. I also suspect there is little you can do directly to weeWX. If it was me I would be using US customary units in my database unless you have good reason not to (* see note) and I would be approaching Pat for a means to allow the user to alter the Belcherton presentation units.

Note: FWIW I am in Australia and prefer to be able to view metric data on the odd occasion I want to look at my raw database data so I run METRIC (when I first started using weeWX there was only US or METRIC, METRICWX came later), I just have to keep reminding myself that rain is in cm. I also do all my development work using a US units database and since I mostly develop for my requirements that way I am forced to make sure I have a mechanism to display metric units.

Gary

[Thomas Keffer]

+1 on Gary's comments.

The right way to do this is to have the MQTT extension specify what unit system it is using to transmit values. Alternatively, it can hardwire in a unit system, then always send values in that system and only that system. Either way, the consumer of the data knows what unit system to expect. 

From there, it becomes a conversion problem to the final, target system used by the data display.

In practice, the "unit group" system used by WeeWX for the final data display works very well. Forcing all units to be in US Customary or Metric is too rigid. At the other extreme, expecting the user to select the unit to be used by every observation is too tedious. The group system seems to be a good compromise.

WeeRT  uses essentially the same system.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pat]

The Belchertown skin extension allows users to specify which group units they want (much like the standard skin). I think this Issue on my GitHub repo (which Colin started) tries to a) get my own head wrapped around this and b) suggest that the weewx-mqtt extension have the ability to also send specific observations in the unit they want. Similar to a skin setup. 

The MQTT extension can specify which units to transmit values in - but it's an all or nothing conversion (US, METRIC, METRICWX). However based on the table in the GitHub link (also attached here), I think what's trying to be determined is if a hybrid scenario can exist somehow? Is that up to the skin or up to the MQTT extension's transmit values?

So would the answer be to have weewx-mqtt transmit in METRICWX and then specify in the skin options group_wind = km_per_hour? Or would that cause some miscalculations of wind data?

An additional layer on top of all of this is that my skin just displays what's present in MQTT. So if MQTT is sending dayRain_cm, I display that even if the user wants dayRain_mm. So I think it would make sense for the mqtt extension to be able to have granular ability on altering certain observations. Otherwise the live-updated data coming from MQTT would have to do in-situ conversions within the JavaScript functions (outside of weewx conversion utilities) - which is another layer of complexity I'd like to avoid if possible. 

Lastly, if I'm not mistaken I think Colin's database is in US units based on previous conversations we've had. 

[Pat]

Thinking about this further, I said:

So would the answer be to have weewx-mqtt transmit in METRICWX and then specify in the skin options group_wind = km_per_hour? Or would that cause some miscalculations of wind data?

This would be invalid because the base skin would probably do the conversion, but the MQTT published message would have wind in m/s since the weewx-mqtt is setup for METRICWX (in this example). So no matter what I would display that observation as the published m/s but with a label of km/h on the page. 

I think I keep going back to the idea of having the weewx-mqtt send specific observations in the unit the user wants. Similar to a skin's [[[Units]]] [[[[Groups]]]] setup.

[Pat]

...and it looks like the weewx-mqtt extension already has the ability to do just that - specify unit overrides. Taking a look at the weewx-mqtt code has this comment:

[StdRestful]
    [[MQTT]]
        ...
        unit_system = METRIC # default to metric
        [[[inputs]]]
            [[[[outTemp]]]]
                name = inside_temperature  # use a label other than outTemp
                format = %.2f              # two decimal places of precision
                units = degree_F           # convert outTemp to F, others in C
            [[[[windSpeed]]]]
                units = knot  # convert the wind speed to knots

So, Colin, you could try altering the mqtt settings in your weewx.conf to this example below. This example worked for me. I had 0.3 inches of rain today, after adding the below and a weewx restart it changed it from dayRain_in to dayRain_mm and was showing 7.62 which converts correctly. Everything else stayed as it was. 

[StdRestful]
    [[MQTT]]
        ...
        unit_system = METRIC # default to metric

        [[[inputs]]]

            [[[[dayRain]]]]

                name = dayRain_mm

                units = mm

And based on your comment on GitHub, I think this is what you want. 

METRIC wind (km/h) - default by the unit_system = metric

METRICWX rain (mm) - overridden by the input above

[Colin Larsen]

This is my MQTT section in weewx.conf, the skin just says "Connected. Waiting for data" and never gets any. I'll have a look for errors

[[MQTT]]

        #This section is for the MQTT service setup

        server_url = mqtt://192.168.20.121:1883/

        topic = weather/41south

        unit_system = METRIC

        [[[Inputs]]]

                [[[[dayRain]]]]

                   name = dayRain_mm

                   units = mm

        binding = loop

        aggregation = aggregate

        log_success = False

        log_failure = True

--

[Pat]

Log onto your broker from your phone or computer and subscribe to the topic and check for messages. Specifically the dateTime message is what changes the status. Also after a weewx restart it may take it a minute or two to populate all observations.

[Colin Larsen]

With those added fields it went to only publishing a record every 2 minutes or so, take them out and it's back to normal :/

Will try again. Is it case sensitive?

On Wed, Nov 21, 2018 at 4:50 PM Pat <p...@obrienphoto.net> wrote:

Log onto your broker from your phone or computer and subscribe to the topic and check for messages. Specifically the dateTime message is what changes the status. Also after a weewx restart it may take it a minute or two to populate all observations.

[Colin Larsen]

Ok it works as far as changing the units goes but MQTT now only publishes every 5 minutes with the rest of the outputs

"monthRain_cm": "0.0", "outsideAlarm1": "0.0", "windSpeed_kph": "0.0", "outsideAlarm2": "0.0", "forecastRule": "0.0", "windrun_km": "8.72121676916", "dayRain_mm": "1.6000000004", "outHumidity": "41.2083333333",

Nov 21 17:00:15 raspberrypi weewx[6939]: restx: MQTT: Published record 2018-11-21 17:00:00 NZDT (1542772800)

Nov 21 17:00:20 raspberrypi weewx[6939]: cheetahgenerator: Generated 14 files for report StandardReport in 4.81 seconds

Nov 21 17:00:23 raspberrypi weewx[6939]: imagegenerator: Generated 24 images for StandardReport in 2.74 seconds

Nov 21 17:00:23 raspberrypi weewx[6939]: copygenerator: copied 9 files to /var/www/html/weewx

Nov 21 17:00:32 raspberrypi weewx[6939]: cheetahgenerator: Generated 4 files for report Highcharts_Belchertown in 9.49 seconds

Nov 21 17:00:32 raspberrypi weewx[6939]: copygenerator: copied 0 files to /var/www/html/weewx/belchertown

Nov 21 17:00:32 raspberrypi weewx[6939]: Belchertown Extension: version 0.8rc2

Nov 21 17:00:40 raspberrypi weewx[6939]: cheetahgenerator: Generated 12 files for report Belchertown in 7.22 seconds

Nov 21 17:00:40 raspberrypi weewx[6939]: copygenerator: copied 22 files to /var/www/html/weewx/belchertown

Nov 21 17:05:16 raspberrypi weewx[6939]: manager: Added record 2018-11-21 17:05:00 NZDT (1542773100) to database 'weewx.sdb'

Nov 21 17:05:16 raspberrypi weewx[6939]: manager: Added record 2018-11-21 17:05:00 NZDT (1542773100) to daily summary in 'weewx.sdb'

Nov 21 17:05:16 raspberrypi weewx[6939]: restx: MQTT: Published record 2018-11-21 17:05:00 NZDT (1542773100)

Nov 21 17:05:18 raspberrypi weewx[6939]: cheetahgenerator: Generated 14 files for report StandardReport in 1.75 seconds

Nov 21 17:05:19 raspberrypi weewx[6939]: imagegenerator: Generated 12 images for StandardReport in 0.89 seconds

Nov 21 17:05:19 raspberrypi weewx[6939]: copygenerator: copied 0 files to /var/www/html/weewx

Nov 21 17:05:28 raspberrypi weewx[6939]: cheetahgenerator: Generated 4 files for report Highcharts_Belchertown in 9.04 seconds

Nov 21 17:05:28 raspberrypi weewx[6939]: copygenerator: copied 0 files to /var/www/html/weewx/belchertown

Nov 21 17:05:29 raspberrypi weewx[6939]: cheetahgenerator: Generated 12 files for report Belchertown in 1.61 seconds

Nov 21 17:05:29 raspberrypi weewx[6939]: copygenerator: copied 1 files to /var/www/html/weewx/belchertown

[mwall]

On Tuesday, November 20, 2018 at 10:45:39 PM UTC-5, Colin Larsen wrote:

This is my MQTT section in weewx.conf, the skin just says "Connected. Waiting for data" and never gets any. I'll have a look for errors

[[MQTT]]

        #This section is for the MQTT service setup

        server_url = mqtt://192.168.20.121:1883/

        topic = weather/41south

        unit_system = METRIC

        [[[Inputs]]]

                [[[[dayRain]]]]

                   name = dayRain_mm

                   units = mm

        binding = loop

        aggregation = aggregate

        log_success = False

        log_failure = True

colin,

the indents do not matter - the square brackets do matter.

the configuration you posted is equivalent to this:

[[MQTT]]

    server_url = mqtt://192.168.20.121:1883/

    topic = weather/41south

    unit_system = METRIC

    [[[Inputs]]]

        [[[[dayRain]]]]

            name = dayRain_mm

            units = mm

            binding = loop

            aggregation = aggregate

            log_success = False

            log_failure = True

try this instead:

[[MQTT]]

    server_url = mqtt://192.168.20.121:1883/

    topic = weather/41south

    unit_system = METRIC

    binding = loop

    aggregation = aggregate

    log_success = False

    log_failure = True

    [[[Inputs]]]

        [[[[dayRain]]]]

            name = dayRain_mm

            units = mm

m

[Colin Larsen]

Boom! Thanks Matt, that was the problem. Also [[[inputs]]] needs to be all lower case

Now - do I need to do that for all of the rain measurements?

--

[Colin Larsen]

Thinking about it the only other one will be the rain rate for Pats skin and MQTT

Thanks for the help all - really really appreciate getting to the bottom of that one!

[Colin Larsen]

And just to close this off - this fixed my problems

[[MQTT]]

        #This section is for the MQTT service setup

        server_url = mqtt://192.168.20.121:1883/

        topic = weather/41south

        unit_system = METRIC

        binding = loop

        aggregation = aggregate

        log_success = False

        log_failure = True

        [[[inputs]]]

                [[[[dayRain]]]]

                        name = dayRain_mm

                        units = mm

                [[[[rainRate]]]]

                        name = rainRate_mm_per_hour

                        units = mm_per_hour