Vantage LOOP2 packets (for THSW index)

[Deborah Pickett]

Summary: seeking expressions of interest in having the Vantage driver also poll for the LOOP2 packet, to get (among other things) the THSW index value.

Hi everyone,

One of the peculiar omissions from the Vantage LOOP packet is the Temperature-Humidity-Sun-Wind (THSW) index, a "feels-like" apparent temperature that the console displays if you have a solar radiation sensor on your ISS.

Where I live (Melbourne Australia) I find the THSW index is a good indicator of what it feels like if I were to step outside.  It's often close to the Australian Bureau of Meteorology's "feels like" temperature.  To me it feels more "like" the apparent temperature than what I get from the Heat Index value that weewx already calculates.

I'd like to display THSW index on my weewx page.  But...

1. No one knows the formula that the Vantage Pro 2 console firmware uses to compute the THSW index.

2. The venerable LOOP packet doesn't include the THSW index value.

But there is a data packet that does include THSW: the LOOP2 packet.  This packet also includes a few other interesting values, like 0.1mph-precision average wind speeds, and the speed and direction of the 10-minute wind gust.  On the other hand, there are things that the LOOP2 packet omits, like the "extra" temperature and humidity sensor values, so a wholesale migration from LOOP to LOOP2 isn't going to do it.

I'm going to have a play with having weewx ask the console to alternate LOOP and LOOP2 packets (command: "LPS 3 <n>").  It's been a long time since I last touched the weewx code base (vintage 2.6, I updated wee_config for transmitter types and calibrations) but I feel like a challenging project.

Who's with me?

[rich T]

Did you check this doc for the formula

https://www.davisinstruments.com/product_documents/weather/app_notes/AN_28-derived-weather-variables.pdf

[Deborah Pickett]

On Monday, February 11, 2019 at 12:52:30 AM UTC+11, rich T wrote:

Did you check this doc for the formula

I did, and it isn't in there.  There are references to academic papers, which I no longer have access to, but I recall the last time I looked into this whole thing several years ago, those papers didn't have a clean formula I could use either. Hence why I'm planning to pull the value precomputed out of the LOOP2 packet.

[Deborah Pickett]

Some observations about the LOOP2 packet:

LOOP2 contains values for "dewpoint", "windchill" and "heatindex", which are absent in LOOP and which have until now been generated in software.  The LOOP2 values are integer-degree-Fahrenheit precision, which is less precision than the values generated in software.  I think most Vantage users are going to want to set the [StdWxCalculate] preferences to "software" for these three.  Currently they are "prefer_hardware" which falls back to software because the hardware (LOOP packets) doesn't generate them.

LOOP contains a value "windSpeed10" (ten-minute average wind speed) to 1-mph precision.  LOOP2 contains the same measurement to a 0.1-mph precision.  It's going to be interesting* mixing these two values into the same variable.

LOOP2 doesn't contain "monthRain", so the shifty little trick that the driver does to work around the packet not containing actual rainfall won't work.  It does contain "dayRain" though.

It wasn't super-hard to get the Vantage driver to pull LOOP2 packets from the console.  Now it's off to read the docs on expanding the database schema to insert the values into the database.

[gjr80]

Hi,

It sounds like you are where I was a few years ago. I wanted THSW (it was one of the fields required in some output generated by WeeWX-WD), I saw LOOP2 could provide but also went looking for a formula and never found anything. I often wondered how the likes of Weather Display and Cumulus provided THSW, presumably through LOOP2 but they must have been doing something with LOOP as well given the limited data in LOOP2. In the end I overcame my OCD and left it, I was happy with apparent temperature. At least with apparent temperature I have something local to compare with through BoM (I'm in Brisbane). But I guess you can't display apparent temperature on your console :)

A couple of thoughts. I think you will find that a lot of Vantage users use hardware record generation, I for one would be loathed to change that, that being said loop data can be accumulated and if record_augmentation = True under [StdArchive] then WeeWX should add anything it can get from the accumulators to the (hardware generated) archive record.

The other thing I have been continually reminded of since contributing to WeeWX is to be wary of complexity. This is going to make the Vantage driver more complex, just don't make it complicated - PEP 20.

I am sure Tom will have some thoughts.

Gary

[Thomas Keffer]

The LOOP2 packets were not introduced until about 6 or 7 years ago, so there are a LOT of Vantage stations out there that cannot support it. So, it has to be optional, probably set through an option in [Vantage].

Unfortunately, LOOP2 packets are missing a lot of data (e.g., extraTemps), so you gain some variables, but lose others. The only way around that is to have the console emit both LOOP and LOOP2. But then, as you say, that will cause things like windSpeed10 to jump around, unless the driver includes special logic for duplicated variables and emits only the LOOP2 values. Complexity.

There are several observation types that can be calculated more accurately in software than hardware, but most users prefer the latter. Why? So, the output matches what they see on the display. This is why prefer_hardware is the default.

Personally, I am busy doing the Python 3 port, so I have no time for this. If you want to do this, make sure you stay current with the python3 branch, as the Python 3 port touched the Vantage driver pretty heavily. 

-tk

[Deborah Pickett]

Thanks, Gary and Tom, for your thoughts and advice.

Anyone who's curious to see how the THSW index tracks actual temperature can view my site at http://www.icemoonprison.com/weather/.

I'll probably end up throwing the code I've written and redo it with the benefit of my first-time experience.  Based on Tom's remarks about the python3 branch, I'll use that as my base.  The strategy for compatibility will be:

- Any console with NVER < 1.90 will continue to receive the LOOP command.

- Any console with NVER >= 1.90 will consult a configuration in the [Vantage] section, which defaults to asking only for LOOP packets.

- If the configuration option in [Vantage] allows LOOP2 packets as well, the driver will use the "LPS" command to alternate LOOP and LOOP2 packets.

- If LOOP2 packets are received, the "windSpeed10" value will be ignored in LOOP packets because the LOOP2 packet has the same value to a higher precision.

- If LOOP2 packets are received, "windchill", "heatindex" and "dewpoint" values will respect the [StdWxCalculate] preference for hardware or software generation.  Currently these values are always generated in software for Vantage consoles because neither the LOOP nor archive packets contain these values.

While I have your attention, a bikeshed-style question about schema naming.  Is there a preferred naming style for new database columns?  THSWIndex? thswIndex? thswindex? Or something else more verbose?

[Thomas Keffer]

Take a look at this ancient issue: Issue #115, 

That issue pointed out all the bits and pieces necessary to do an extended schema. Now 2 years later, they are (I think) all in place. It would be wonderful if you could finish it off! That would clear the way for your new observation types.

-tk

[Deborah Pickett]

On Tuesday, February 12, 2019 at 10:26:01 AM UTC+11, Tom Keffer wrote:

Take a look at this ancient issue: Issue #115,

That definitely seems relevant.  I'll raise a couple of issues on GitHub to track the whole LOOP2 enhancement, and include a reference to #115.

Something that was brought up with me off-list, but which I am not equipped to answer: THSW index is a proprietary Davis calculation, and so far the schema has been unpolluted with vendor-specific terms.  What if another vendor invents an almost-but-not-identical heat index value? Does that get added to the schema too?

[mwall]

On Monday, February 11, 2019 at 6:03:22 PM UTC-5, Deborah Pickett wrote:

While I have your attention, a bikeshed-style question about schema naming.  Is there a preferred naming style for new database columns?  THSWIndex? thswIndex? thswindex? Or something else more verbose?

i like bikes :)

if you want to maintain compatibility with wview, then you probably should follow the (mostly) camelCase style used by the wview schema.

however, if you are doing a new implementation, then definitely use alllowercase or use snake_case.  snake case is much easier to use across databases (mssql, mysql, postgres, sqlite) and will not cause any of the case problems that camelCase has.

(i wish we could eliminate the column name 'interval', but that horse is so far out of the barn it has evolved to its own new species)

m

[Greg Troxel]

Deborah Pickett <deb...@icemoonprison.com> writes:

> Something that was brought up with me off-list, but which I am not equipped
> to answer: THSW index is a proprietary Davis calculation, and so far the
> schema has been unpolluted with vendor-specific terms. What if another
> vendor invents an almost-but-not-identical heat index value? Does that get
> added to the schema too?

[I had written to Deborah offlist; she quite reasonably abstracted the
main point]

I had not really heard of THSW; I don't think we use it in the US. Just
"heat index" and "wind chill" (aside from content from particular TV
channels).

There are two separate concerns here:

1) scales that are well defined in the scientific community

Generally, weewx is reporting values that are measured by instruments,
but the values are well-defined and recognized in the meterological
community. Yes, many people have their wind sensors other than at 10m,
but still there is a broad intent to follow the consensus definitions.
Various countries have adopted different apparent temperature
definitions, and probably this is true for UV index, but still these are
well defined in the literature. THSW apparently has an original
definition, but Davis documents that they calculate something else, and
they don't really define it. So there is apparently no way to compute
the same THSW scale from the constituents. I find this problematic, as
its dealing with something that we don't really know what it means, and
we are reduced to "this is the number that Davis reports for THSW".

In the US, there are at least two proprietary apparent scales, each
promulgated by a company. They don't seem to be publically documented,
and thus can't have had open review, and can't be calculated by others.
It would not shock me if there was some licensing arrangement to have
them in branded personal weather stations.

2) namespace collisions

The second concern is about collisions between various things called
THSW. Here, I would say that there is no scale called THSW, because
there is no scientific consensus definition. So I would call this
"davis_THSW" or something like that.

I am unclear on the plans for scheme exension. If the idea is that
people who want to record this value run some command to add a column,
and everybody else continues as is, that sounds fine. But I would see
it as the common schema being polluted by proprietary definitions if it
were part of databases by default.



I don't mean to sound so negative about people who find this useful and
want to record it; they are of course welcome to do that, and support
for getting any info from a station that can be gotten is progress. I
might even log it. I am merely trying to point out that crossing into
proprietary measurements feels like a departure from past practice.

Greg

[Thomas Keffer]

Not only is THSW used in the US, it was invented here by Robert Steadman 40 years ago.

Here are papers on how to calculate it using tables, and using formulas.

In the interests of calculating it in a low-powered instrument, Davis simplified the tables slightly, but there is nothing proprietary about it.

-tk

[Greg Troxel]

Thomas Keffer <tke...@gmail.com> writes:

> Not only is THSW used in the US, it was invented here by Robert Steadman 40
> years ago.
>
> Here are papers on how to calculate it using tables

> <https://journals.ametsoc.org/doi/pdf/10.1175/1520-0450%281979%29018%3C0861%3ATAOSPI%3E2.0.CO%3B2>,
> and using formulas
> <https://journals.ametsoc.org/doi/pdf/10.1175/1520-0450%281979%29018%3C0874%3ATAOSPI%3E2.0.CO%3B2>
> .

>
> In the interests of calculating it in a low-powered instrument, Davis
> simplified the tables slightly, but there is nothing proprietary about it.

Thanks - I'm very glad to hear I was confused. I had read some of the
Davis documentation and understood it as saying that they were
calculating an intentionally modfied index, but managed to have the part
about the original index in the literature leave my head before I got to
writing - sorry about that.

In:

https://www.davisinstruments.com/product_documents/weather/app_notes/AN_28-derived-weather-variables.pdf

I misinterpreted

The Vantage Pro and Vantage Pro2 calculation is an
improvement over the THSW Index in the Health EnviroMonitor because the
Health system:

* only calculates THSW Index when air temperature is at or above 68°F.
• assumes the sky is clear.
• assumes the elevation is sea level.

as an indication they were trying to calculate something different. But
I guess they mean they are trying to more closely follow the paper,
rather than the cruder approximation in one of their other products.

So I guess it would be interesting to have the original forumulas in
StdCalculate, and perhaps compare to the hardware values.

[Hartmut Schweidler]

my way

def calculate_heat_index(temperature, relative_humidity):

        """
        Uses the temperature and relative humidity to calculate the heat index, the purpose of which is to represent a
        "felt-air temperature" close to what a human actually feels given the temperature and humidity. This index does
        not take into account the wind speed or solar radiation, and so is not the most accurate measure of a true
        "feels-like" temperature. For that, see `calculate_thw_index` and `calculate_thsw_index`. The algorithm used and
        its constants are sourced from http://www.wpc.ncep.noaa.gov/html/heatindex_equation.shtml. In this algorithm:
       
        T is the temperature in degrees Fahrenheit
        RH is the relative humidity percentage

        This function is tested against the NOAA/NWS heat index chart found at
        http://www.nws.noaa.gov/os/heat/heat_index.shtml. It returns `None` if the input temperature is less than 70F.
        Experts disagree as to whether the heat index is applicable between 70F and 80F, but this function returns a heat
        index calculation for those values.

        if temperature < HEAT_INDEX_THRESHOLD:
                return None

        T = temperature
        RH = _as_decimal(relative_humidity)

        heat_index = HI_0_5 * (T + HI_61_0 + ((T - HI_68_0) * HI_1_2) + (RH * HI_0_094))
        heat_index = (heat_index + T) / TWO  # This is the average

        if heat_index < HI_SECOND_FORMULA_THRESHOLD:
                return heat_index.quantize(ONE_TENTH, rounding=decimal.ROUND_CEILING)

        heat_index = (
                HI_C1 + (HI_C2 * T) + (HI_C3 * RH) + (HI_C4 * T * RH) + (HI_C5 * T * T) +
                (HI_C6 * RH * RH) + (HI_C7 * T * T * RH) + (HI_C8 * T * RH * RH) + (HI_C9 * T * T * RH * RH)
        )

        if (HI_FIRST_ADJUSTMENT_THRESHOLD[0] <= T <= HI_FIRST_ADJUSTMENT_THRESHOLD[1] and
                                RH < HI_FIRST_ADJUSTMENT_THRESHOLD[2]):
                heat_index -= (
                        ((HI_13 - RH) / FOUR) * ((HI_17 - _abs(T - HI_95)) / HI_17).sqrt()
                )
        elif (HI_SECOND_ADJUSTMENT_THRESHOLD[0] <= T <= HI_SECOND_ADJUSTMENT_THRESHOLD[1] and
                                                        RH > HI_SECOND_ADJUSTMENT_THRESHOLD[2]):
                heat_index += (
                        ((RH - HI_85) / TEN) * ((HI_87 - T) / FIVE)
                )

        return heat_index.quantize(ONE_TENTH, rounding=decimal.ROUND_CEILING)

# Constants used by Davis Instruments for its THW calculations
THW_INDEX_CONSTANT = decimal.Decimal('1.072')

# Constants used by the Australian Bureau of Meteorology for its apparent temperature (THSW) calculations
THSW_0_25 = decimal.Decimal('0.25')
THSW_0_348 = decimal.Decimal('0.348')
THSW_0_70 = decimal.Decimal('0.70')
THSW_4_25 = decimal.Decimal('4.25')
THSW_6_105 = decimal.Decimal('6.105')
THSW_17_27 = decimal.Decimal('17.27')
THSW_237_7 = decimal.Decimal('237.7')

HEAT_INDEX_THRESHOLD = decimal.Decimal('70.0')  # degrees Fahrenheit
WIND_CHILL_THRESHOLD = decimal.Decimal('40.0')  # degrees Fahrenheit

DEGREE_DAYS_THRESHOLD = decimal.Decimal('65.0')  # degrees Fahrenheit

I found that on the internet