Meteostick driver for weewx

[Lucas Heijst]

Begin february 2016 we started developing a weewx driver for the meteostick.

From the meteostick manual, see http://www.smartbedded.com/files/manual-en.pdf

Meteostick operates as a receiver for radio frequency data from your Davis®1 Vantage Pro2TM, Vantage Pro2 PlusTM, Vantage VueTM RF sensors or the Fine Offset HP-100x outdoor sensor array RF sensor WH24. Data is sent out by these devices in license free RF bands (915 MHz in the US/AU, 868 MHz in Europe) where Meteostick listens and makes data available for computation. Meteostick can report data as

• explanatory text or

• ready computed data to feed your programs or

• raw data 1:1 as the sensors send it without any processing

Our team

• Frank Bandle who is testing the meteostick driver on several Raspberry PI systems and with both Vantage Vue and Vantage Pro2 stations

• Matthew Wall who wrote the initial driver and maintain the github repository
  https://github.com/matthewwall/weewx-meteostick

• Luc Heijst who co-writes part of the logic (which is usually not very pythonic!)
  Recently Luc also tests the meteostick driver with his own Vantage Pro2. This configuration has also a solar sensor, an Anemometer Transport Kit, an Leaf-Soil station with Soil moisture and Soil temperature sensors and a leaf wetnes sensor.

Via this topic we will inform you about the progress.

We started first our driver with parsing the so called machine readable data ('o1').

An example of this data is:

T 3 20.8 36 -53 L

W 3 0.00 261 -53 L

W 1 0.44 241 -67

R 2 128 -67

R 3 128 -53 L

W 3 0.00 261 -53 L

R 1 44 -66

B 35.2 1023.57

W 1 0.44 233 -68

L 7 2 0 -52

W 2 0.00 83 -72

T 1 19.6 41 -66

W 3 0.00 261 -54 L

M 7 1 49 -51

O 7 1 22.3 -51

W 2 0.00 83 -66

W 1 0.00 234 -66

T 3 20.8 37 -54 L

P 1 21.1 -68

R 3 128 -52 L

W 1 0.00 234 -67

U 3 0.4 -58 L

Where "T" stands for the message with outside temperature and outside humidity

and "W" for a message with wind and gust etc.The "L" at the end of some lines indicates a low battery.

Problems with the machine readable output format

This format had two problems as we know by now:

• The throughput of wind samples appear to be 4 times less than it should be. A very poor average of 1 wind sample per 9.4 seconds!

• The data of the leaf wetness channels 1 and 2 are reversed.

NB. I wrote the author, Boris Pasternak, about these two problems. I'm still waiting for an answer.

Firmware

The firmware of the meteostick can be loaded via a module embedded in the meteohub software. No need to buy or activate a licence for this.

The problem with the firmware versions is that we could not find a list with release notes.

The website of the meteostick: http://www.smartbedded.com/wiki/index.php/Meteostick offers a link to the latest firmware file. This file has a general name. No information whatsoever which version this is.

All firmware versions can be downloaded from this site: http://tau.meteobridge.com/files/

User manual

The small user manual is not updated for a while.

• Missing are several configuration commands with their meanings.

• The wrap around of the rain counter is at 127 and not as 255 as the manual says.

• The "B" data for both machine readable output as raw output (more later) describe parameters for number of goof packets and nummer of failed RF packtets which now have another meaning. It's a guess to find out their current meaning.

Parameters not described in the user manual

?

# Show current settings (here: default) settings

# Possible commands are: r, ?, m, t, r, v, d, l, b, x, o, p, c, f

v0 or v1

# Set verbose: 0=off, 1=on; default off

d0 or d1

# Set debug: 0=off, 1=on; default off

l0 or l1

# Set ledmode 0=led active high, 1=led active low; default low

Bug: setting l1 presets in fact the bright mode; the texts are reversed!

b0 or b1

# Set bandwidth 0=narrow, 1=width; default narrow

p0 or p1

# Set probe 0, 0=probe mode off, 1=probe mode on; default off

# mode = on allows moisture sensors without temp probe for Davis Vantage

r0 - r255

# Set repeater 0-255; 8 bits, same concept as with transmitters; default 255

c0 - c255

# Set channel 255; function and use not known yet; default 255

x0 - x250 (or x255?)

# Set rf threshold

RF threshold

The most important command of the 'hidden features' above is the set rf threshold.

The given value is twice the desired rf threshold in [minus] dB.

Example: a threshold of -80 dB is set with command x160.

Each received message from the meteostich contains the current signal strength.

This is very handy with trouble shooting because:

• When the rf threshold is is too low (e.g. x100 for - 50 dB) only messages with a stronger signal than -50 db will come through. Worst case: no single message will be received.

• When the rf threshold is too high (e.g. x250 for -125 dB) the meteostick receiver will pick up so much noise that (again) no single message will be received.

• The advice is to set the rf threshold 10 dB higher than the weakest received signal.

  Example: the messages have signal strengths varying from -63 dB to -78 dB.

  The rf threshold should then be set to 2 * (78 + 10) = x176 for a threshold of -88 dB.

Raw message format

After the detection of the bad throughput of the "machine readable format" we switched to the "raw format". In fact this format is the same as Davis uses between the stations and the console or envoy.

The raw format has two flavours:

• 8 bytes data of which 2 bytes contain a CRC check.

• 10 bytes data. The first 8 bytes of this format are the same as the 8-byte format. It contains two extra bytes which are always zero when no repeaters are used.

We choosed the 8-bit raw data, selectable with command 'o0'.

Documentation of the raw data messages

Neither Davis nor Smartbedded publish the details of the raw format.

We found several publications on the internet which helped us start.

The document of dekay "Davis RFM69" contain a lot of useful data and also links to other information. See: https://github.com/dekay/DavisRFM69/wiki/Message-Protocol

OK, enough for now as a start of this topic. To be continued.

Luc Heijst 

[Lucas Heijst]

Solar sensor connected to Anemometer Transport Kit.

Today I connected my solar sensor on my anemometer transport kit. 

The idea is to place the solar sensor on a higher level than the iss where the solar sensor is surrounded by trees and buildings.

It is known that this won't work on the Davis equipment because you can define only one iss per console / datalogger.

With the meteostick this wouldn't be a problem: you don't even have to configure which stations are connected to the channels.

Attached the picture of my solar sensor on the south side of my wind mast and the solar graph. Left on the graph you see the readings of both Davis and Meteostick. At 18:00 you only see the green line of the Meteostick data.

Luc 

[Lucas Heijst]

Formula's to calculate rain rate from raw values.

The formula's I found on the internet to extract the rain rates from the raw Davis data were all not what I had expected.

Below my formula's based upon analysis of the received raw data and the presented loop data of a Vantage driver listening to the same weather station as the meteostick.

Note: routine logparse is a refined debug printing call with levels: 0=no print; 1=minimum, 2=nornal, 3=detailed print

Luc

                    if message_type == 5:

                        """ The rain_rate formula's on the internet differ from each other

                        For both light and heavy rain we like to know a 'time between ticks' in s.

                        The rain_rate then would be: 3600 [s/h] / time_between_tips [s] * 0.2 [mm] = xxx [mm/h]

                        Note: The values received from the vantage console are the contents of one rain bucket

                        higher than the theoretical value, so we add one (1) as well to the rain rate value.

                        """

                        # rain rate

                        time_between_tips_raw = ((pkt[4] & 0x30) << 4) + pkt[3]  # typical: 64-1022

                        logparse(3, "time_between_tips_raw: %03x (%s)" % (time_between_tips_raw, time_between_tips_raw))

                        if time_between_tips_raw == 0x3FF:

                            # no rain

                            data['rain_rate'] = 0

                            logparse(3, "no_rain: %s mm/h" % data['rain_rate'])

                        else:

                            if pkt[4] & 0x40 == 0:

                                # heavy rain: typical value: 64/16 - 1020/16 = 4 - 63.8 (180.0 - 11.1 mm/h)

                                time_between_tips = time_between_tips_raw / 16

                                data['rain_rate'] = ((3600 / time_between_tips) + 1) * rain_per_tip #  mm/h

                                logparse(1, "heavy_rain: %s mm/h, time_between_tips: %s s" % (data['rain_rate'], time_between_tips))

                            else:

                                # light rain:  typical value: 64 - 1022 (11.1 - 0.8 mm/h)

                                time_between_tips = time_between_tips_raw

                                data['rain_rate'] = ((3600 / time_between_tips) + 1) * rain_per_tip #  mm/h

                                logparse(1, "light_rain: %s mm/h, time_between_tips: %s s" % (data['rain_rate'], time_between_tips))

[kobuki]

Interesting find. There's some confusion around the leaf/soil calculations as well, IIRC.

However, I cannot find your above changes at https://github.com/matthewwall/weewx-meteostick/ - am I looking at the rigth repo?

[mwall]

On Friday, May 6, 2016 at 8:01:49 PM UTC-4, kobuki wrote:

Interesting find. There's some confusion around the leaf/soil calculations as well, IIRC.

However, I cannot find your above changes at https://github.com/matthewwall/weewx-meteostick/ - am I looking at the rigth repo?

that is the right repo.  luc's 0.33-3 most recent changes are now committed at 0c0ca77.

m

[Lucas Heijst]

I made some progress today with the conversion of the leaf-soil sensors raw data to the "davis" data format. See attached graphs.

The formulas for the temperature of the soil moisture and leaf wetness sensors were found on the internet and appear to be accurate.

The formulas for the soil moisture sensor of Dr. Richard G. Allen (Univ. of Idaho, Kimberly, Idaho) look good, but they are based upon a soil moisture sensor resistance (R) while the meteostick driver only receives soil_potential_raw values.
Via trial and error - where I compared 7 close to each other alternatives with the received LOOP values of the vantage driver - I came close to a usable formula for R. See graph: hoursoilmoist.png.
Note: the soil moisture sensor has three different formulas of which the last one now looks OK.

The values for the leaf wetness sensor are pure experimental. No info found on the internet so far.
In an Excel spreadsheet all received dat from the meteostick and vantage drivers were collected and analyzed.
The outcome were three areas:

• leaf_potential_raw < 847: set potential = 1
• 847 < leaf_potential_raw < 1010: use formula based upon regression of the collected data to set potentional
• leaf_potential_raw >= 1010: set potential = 15

More to come.

Cheers,
Luc 

[kobuki]

My MS has been running fine for about 1-2 days and I have a couple of comments after regular visual comparison and some code review. I have the Davis Envoy-fed one and the MS one running side-by-side as 2 separate daemons of WeeWx on the same server. Reception ratio is ~90-95% where it sits near the server.

The Envoy consistently produces higher values for temperature and wind/wind gust. No numerical analysis yet, just eyeballing using a switching comparison with the same scale on screen, it's about 5-10% higher. I can see 2 reasons for that: the code is converting wind data to metric - it's not recommended since Davis devices always use imperial as native format, with the exception of the 0.2 mm rain bucket and probably the leaf/soil calculations. The conversion produces weird values in the archive db and raw table (I have a raw data table running for MesoWx), like 10.99999993 for 11, 19.99999912 for 20 and so on. It's because WeeWx converts the values back to imperial for storage and the original values were 11 and 20, respectively, for example. I think the driver should keep and pass everything in native format without conversions and only convert to metric or anything else where calculations require it. I also suspect that this conversion causes a rounding error somewhere, causing the consistently lower values for wind and temperature (and most probably all other converted values as well). I see some magic constant for such conversions - WeeWx has built-in functions for that, I'd recommend using those instead where necessary.

The wind gust coming from the ISS is not stored - why? We lose important and interesting data! I can see that for yesterday's wind peak the Envoy reported 38 km/h, while the stick-based driver only 33 km/h. That's more than 10%, which is pretty bad, I think. All it should be needed is to store the wind gust data coming from the stick, possibly also considering the negative index that's also transmitted in the same packet.

Well, that's all for now :) I can probably fix these things in the code and make a pull request (the wind gust data might need a little more consideration and time, though), and if you don't object, I might just do that this week unless someone beats me to it.

[Lucas Heijst]

We started our driver with parsing the machine readable code where all units were in METRICWX format so the choice for a target_unit of METRICWX was obvious. Now that the driver defaults to the raw format with imperial units we might consider to change the target_unit to US.

The conversion back and forth from metricwx to imperial will produce errors but they should be far from the 5-10% you mentioned below. When 11 is represented by10.99999993 this gives an error of 0,0000006%. That is about eight million times less than 5%!.

Before we change anything in the driver we should evaluate where the differences come from. 

As an example the rain rate data. My syslog show both rain rate loop data of the vantage and meteostick drivers. When it rains, the values are equal or differ the value of one bucket (in my case 0.2 mm) during a short period. I did not checked this in detail, but the general impression is that the LOOP data are about the same. Yet the produced rain rate graphs show big differences. As I wrote on the weewx forum, I will investigate what causes these differences of 200-800%!

You asked why the wind gust value isn't stored in the database. The reason is simple: I thought this value is a 10-minute value.(based upon data I found on the internet) and wouldn't be the same (type of data) as the wingGust LOOP values received by the vantage driver. I will check these differences in LOOP data. BTW You mention a negative index stored in the wind gust data packet. Can you explain what the meaning is of this index?

As I said earlier: before we change anything in the driver we should come with thorough analisys why the data differs from the expected data (and what causes it).

Luc

[kobuki]

OK, I get it about the conversion to metric now. But as soon as the decision have been made, by principle the code should process the data as accurately as possible, without any conversion, as imperial values, as long as possible. It should have been part of the decision to go with the raw data. Well, at least that's what my programmer ego makes me say ;) But I think it should be changed as soon as possible, exactly to avoid problems stemming from eg. improper rounding - taking the integer part instead of numerical rounding is one of the common mistakes I've seen many times (not necessarily in the MS driver). For any sensor with a received value below 10, such as low winds, could cause an error of > 10%. In short, I think these things should be fixed first, then algorithmical issues or other bugs, should any exist.

About the "negative index", please see the following sub-thread in this forum, starting with this message: http://www.wxforum.net/index.php?topic=10739.msg243565#msg243565

The current driver decodes this value as gust_index_raw, I believe. It's an indexed reference to a previous interval between 2 consecutive #9 messages to which the current transmitted gust value applies. It can be ignored and gust value used as the current value, or to update a previous interval with the interpreted data. I think the gust value itself is safe to use as-is, without minding the index, though.

[Lucas Heijst]

Thanks.

I made a mistake with the gust data. They are not LOOP data but archive data in the vantage driver. May be we can use the genArchiveRecords call in the meteostick driver to yield the windGust and windGustDir data to the weewx driver. Those two fieldd would be the only archive data we have in the meteostick.

Luc

[kobuki]

Hm, yeah, I understand what you're proposing by calling the gust value "archive" in the SIM board, but they're also transmitted in the Davis LOOP2 packets as well, at offsets 22 and 24, gust value and direction, respectively (no indexed references there, though). When WeeWx uses software record generation (and thus ignoring archive packets at normal runtime), these are the values used for gust metrics. We can leave them as they are, as loop record data for further processing. It should work fine with the sw. record generator.

Technically in the SIM you only need to store the current gust value for the last 10-minute span and a counter to record how many #9 messages ago it was recorded - no archiving or buffering is necessary.

Also, a quick note regarding the rain ticks counter. I think it's just normal. What you're seeing might be caused by the difference in delays between subsequent Davis LOOP records and on-air packets. The former aims at a regular 2.5 sec delay, compressing all important values for a short period into a single record, but the latter is dependent on several factors, such as number of stations, delay between receptions of different stations, etc.

[Lucas Heijst]

For the moment we leave the wind registrations te same. I see a kind of 'flattening' in the windSpeed and windDirection graphs of the vantage data. Because of that alone the data of the meteostick driver won't be the same as that of the Vantage driver.

The target_unit of METRICWX also remain unchanged because the handling of the raw data still has more units in METRICWX than in US!

We discovered a few other things about eralier published formula's to translate the Davis raw data.

outsideTemp

The formula found on the internet divided 'their' raw data by 160.

The smallest resolution was here 25 / 160 = 0.4 degrees F.

The formula below gives the full resolution of 0.1 degrees F: 

               elif message_type == 8:
                   # temperature
                   # message examples:
                   # I 103 80 0 0 33 8D 0 25 11  -78 2562444 -25
                   # I 104 81 0 DB FF C3 0 AB F8  -66 2624980 125 (no sensor)
                   temp_f_raw = ((pkt[3] << 4) + (pkt[4] >> 4))
                   if temp_f_raw != 0xFFC:
                       temp_f = temp_f_raw / 10.0
                       data['temperature'] = weewx.wxformulas.FtoC(temp_f) # C
                       dbg_parse(2, "temp_f_raw: 0x%03x, temp_f: %s, temp_c: %s" % (temp_f_raw, temp_f, data['temperature']))

Solar Power Cel voltage

Also here found wrong formulas on the internet. Instead divide the raw value by 100 we divide it now by 300.

This results in maximum values of 3.3 V, the same values the machine readable format presents.

The new formula:

elif message_type == 7:
    # solar cell output / solar power (Vue only)
    # message example:
    # I 102 70 1 F5 CE 43 86 58 E2  -77 2562532 173
    """When the raw values are divided by 300 the voltage comes in
    the range of 2.8-3.3 V measured by the machine readable format
    """
    solar_power_raw = ((pkt[3] << 2) + (pkt[4] >> 6)) & 0x3FF
    if solar_power_raw != 0x3FF:
        data['solar_power'] = solar_power_raw / 300.0

Super Capacitor voltage

Same story here: divide by 300 instead of 100, see formula below:

if message_type == 2:
    # supercap voltage (Vue only) max: 0x3FF (1023)
    # message example:
    # I 103 20 4 C3 D4 C1 81 89 EE  -77 2562520 -70
    """When the raw values are divided by 300 the maximum voltage
    of the super capacitor will be about 2.8 V. This is close to
    its maximum operating voltage of 2.7 V
    """
    supercap_volt_raw = ((pkt[3] << 2) + (pkt[4] >> 6)) & 0x3FF
    if supercap_volt_raw != 0x3FF:
        data['supercap_volt'] = supercap_volt_raw / 300.0

More to come...

With thanks to Kobuki for his assistance!

Luc

[kobuki]

Regarding outsideTemp:

Old formula:
temp_f_raw = (pkt[3] * 256 + pkt[4]) & 0xFFC0
temp_f = temp_f_raw / 160.0

"New" formula:
temp_f_raw = (pkt[3] << 4) + (pkt[4] >> 4)

temp_f = temp_f_raw / 10.0

Old equation rewritten:
temp_f_raw
    = ((pkt[3] << 8) + pkt[4]) / 16 / 10.0
    = (((pkt[3] << 8) + pkt[4]) >> 4) / 10.0
    = ((pkt[3] << 4) + pkt[4] >> 4) / 10.0

As we've discussed earlier, they were always the same, but written a little more ambiguously, using division where simple shifting would have been more adequate. Apart from a better written formula, neither of the variants have less precision. They're exactly the same.

[kobuki]

OK, I need to amend my last post here. The mask in the original equation should have been 0xfff0, not 0xffc0 - the latter cuts off 2 important bits. Only the lowest 4 bits are not important. This is the mistake with it. Sorry for the noise.

[Lucas Heijst]

We are making progress with the parsing and translation of the soil moisture values.

The leaf-wetness and soil-moisture temperatures are calculated with the found formules

in Chris Materi's CC1101 Weather Receiver.

For the calculation of the soil-moisture values in centibar we couldn't find 

any useful formulas, so we made our own.

First the raw soil-moisture readings were temperature compensated.

Then the compensated raw soil-moisture value is looked up in a table with both

raw soil-moisture values and corresponding soil moistutre values in centibar** 

and intterpolated with the values of two adjacent columns which can be 

considered as a straight line.

** The corresponding soil-moisture values were extracted from the loop values 

of a vantage driver connected to a Vantage Pro2 console listening to the 

same weather station.

The attached graph shows two runs of the soil-moisture sensor going from soaking wet 

- and drying in open air - to the 'maximum' dryness of 200 centibar. 

Each run took about 3 days. 

The red values show the settings of the lookup table during the first run.

The green values show the measured values during the second run.

We need a few more test runs to see if the values are reciprocable.

Below the used program sections of the meteostick driver***.

*** This part is not implemented yet in driver version 0.36 on github. Please be patient.

More to come.

Luc

=====================================================================================================================

# Lookup table for soil_moisture_raw values to get a soil_moisture value based upon a linear formula

sm_cor_table =     [100.0, 213.8, 222.5, 263.2, 389.3, 473.0, 534.2, 593.0, 750.0, 1024.0]

sm_out_table =     [  0.0,   9.0,  10.0,  15.0,  35.0,  55.0,  75.0, 100.0, 200.0,  220,0]

def calculate_leaf_soil_temp(leaf_soil_temp_raw):

    """ Decode the raw leaf-soil temperature, then calculate the

    actual leaf-soil temperature and the leaf_soil potential,

    using Davis' formulas.

    When the sensor is not populated, leaf_soil_temp_raw and

    leaf_soil_potential_raw are set to their max values (0x3ff).

    see: https://github.com/cmatteri/CC1101-Weather-Receiver/wiki/Soil-Moisture-Station-Protocol

    """

    # Convert leaf_soil_temp_raw to a resistance (R) in kiloOhms

    a = 18.81099

    b = 0.0009988027

    r = a / (1.0 / leaf_soil_temp_raw - b) / 1000 # k ohms

    # Steinhart-Hart parameters

    s1 = 0.002783573

    s2 = 0.0002509406

    try:

        leaf_soil_temp = 1 / (s1 + s2 * math.log(r)) - 273

        dbg_parse(3, 'leaf_soil_temp thermistor resistance: %s kohm, leaf_soil_temp: %s, leaf_soil_temp_raw: %s' %

                  (r, leaf_soil_temp, leaf_soil_temp_raw))

        return leaf_soil_temp

    except ValueError, e:

        logerr('leaf_soil_temp failed for leaf_soil_temp_raw %s, r: %s: %s' %

               (leaf_soil_temp_raw, r, e))

    return DEFAULT_SOIL_TEMP

def calculate_soil_moisture(soil_moisture_raw, soil_temp):

    # The calculation of the soil soil_moisture in cb is based upon empirical

    # values in two lookup tables. The values between two points can be

    # interpolated as a straight line.

    soil_moisture = 0.0 # preset soil moisture

    # Correct the soil moisture raw value for the temperature

    # The correction value of 0.0079 % is based upon observations: the corrected raw values

    # during a temperature 'jump' must show as less temperature influences as possible.

    sm_raw_corr = soil_moisture_raw / (1 - 0.0079 * (soil_temp - DEFAULT_SOIL_TEMP))

    # lookup sm_raw_corr value in table

    for x in range(0, 10):

        if sm_raw_corr < sm_cor_table[x]:

            if x == 0:

                # 'pre zero' phase; soil_moisture = 0.0

                dbg_parse(1, "soil_moisture_raw %s sm_raw_corr %s soil_moisture_temp %s x %s sm %s x %s"

                          % (soil_moisture_raw, sm_raw_corr, soil_temp, x, soil_moisture, sm_cor_table[x]))

                break

            else:

                # determine the soil moisture value in cb

                sm_per_sm =  (sm_out_table[x] - sm_out_table[x - 1]) / (sm_cor_table[x] - sm_cor_table[x - 1])

                sm_offset = (sm_raw_corr - sm_cor_table[x - 1]) * sm_per_sm

                soil_moisture = sm_out_table[x - 1] + sm_offset

                dbg_parse(1, "soil_moisture_raw %s sm_raw_corr %s soil_moisture_temp %s x %s sm %s x-1 %s, x %s"

                          % (soil_moisture_raw, sm_raw_corr, soil_temp, x, soil_moisture, sm_cor_table[x - 1], sm_cor_table[x]))

                break

    # Clamp potentional between 0 and 200

    # Normally this not necessary; just a precaution

    soil_moisture = max(0, soil_moisture)

    soil_moisture = min(200, soil_moisture)

    return soil_moisture, sm_raw_corr

    

    =====================================================================================================================

    

[Lucas Heijst]

Version 0.37 of the meteostick driver is released on github.

https://github.com/matthewwall/weewx-meteostick

Change log:

* Changed: calculate_soil_moisture

* Changed: calculate_leaf_wetness

* Changed: tag for solar_power.

* Added: tag for supercap_volt.

* Fixed: formula for supercap voltage raw value (Vue only).

* Fixed: formula for uv raw value; thanks to Jozef Smolders for bringing this up and testing it

* Fixed: typo with light rain

* Fixed: accuracy rain rate formula's

* Fixed: formula for solar cell output raw value (vue only)

* Changed: explanation of unknown ATK message (0xC)

* Changed: the value of not connected soil temperature sensors

* Changed: the value of not connected soil moisture sensors

* Changed: the value of not connected leaf wetness sensors

Luc

[Lucas Heijst]

Hi all,

I runned meteostick with machine readable format for about a day to test the behaviour in respect to the raw format.

My findings:

• inside temp is very close to the inside temp reported by the Vantage Pro2 console (nice job!); the temperatures of the raw format are in the range 33-38 degrees C.
• barometric pressure has several 'spikes' not seen in the raw format
• soil moisture values seem not to be temperature normalized; giving differences upto -20 cb
• not present leaf wetness sensors got a reading with value zero instead of no reading
• for channels 1 and 2 the reported soil moisture temps are also valid for leaf wetness sensors (if present); this has to be solved in the driver
• pct good for machine readable format is typically 50-65% where (in my situation) raw format has values of 85-95%.
• leaf wetness values are close to that of the Vantage Pro2 console values.

Cheers,
Luc

[mwall]

this is regarding the commits for meteostick driver 0.45, which included removal of the use of weewx.units.Converter() for unit conversions and the change to weewx.METRICWX from weewx.US for the unit system of LOOP packets emitted by the meteostick driver.

there have been many exchanges via email regarding the meteostick driver between 0.37 and 0.45.  luc and kobuki have done a fantastic job of reverse engineering the davis rf protocols and the meteostick hardware and firmware implementations.

i'm trying to bring the exchanges back to the forum (again) so the knowledge does not get trapped in our personal mailboxes.

On 01 Oct 2016, at 08:43, nls wrote:
> So you mean after we agreed that using US units works better generally, especially with wind, you reverted all
> my efforts of not using metric output back to using metric output? May I ask, why? Because most of us are in
> Europe with metric standards? Or do you maybe think that the original dev of WeeWx made a mistake by
> using US units? Practically everything in the packets is in US units: wind is mph, temperature is F°, rain
> can be both. Solar values have scientifically established dimensions. Only with the addition of the soil/leaf
> stations does the picture change somewhat. Which is admittably sparse in relation to the rest of the instruments.

context:

my goal was (is) to simplify the implementation yet still ensure data integrity.

i consolidated units and made other changes so that i could make sense of what is actually happening in the code.  0.44 was (and 0.45 still is) rather clunky.

i do not care whether the driver emits data in weewx.US or weewx.METRICWX.  (i am pretty sure that it does not matter, at least not in the context of all the other conversions that happen, i.e., driver to database, database to display, database to uploaders.)

btw, kobuki, your efforts have not been reverted.


the facts:

- the driver must emit units in a single unit system.  that can be METRIC, METRICWX, or US.

- rainfall is counts

- calculate_thermistor_temp is degree C

- for parse_machine:
  - pressure is mbar
  - WT temperature is degree C directly from the device
  - LMO temperature is degree C directly from the device
  - wind_speed is m/s

- for parse_raw:
  - type 8 digital temperature is degree F * 10 directly from the device sensors
  - type 8 analog temperature is from calculate_thermistor_temp (degree C)
  - temperatures for leaf and soil stations are from calculate_thermistor_temp (degree C)
  - pressure is mbar * 100
  - wind_speed is obtained from calc_wind_speed_ec (mph)


analysis:

rain is counts, so there is a unit conversion no matter what.  that is a simple multiplication by either inches or mm or cm, depending on the unit system.  the rain_per_tip constant should have sufficient precision to avoid inaccuracies, and the precision of python float is more than adequate to capture rainfall amounts accurately.

wind speed conversion is a simply multiply as well.  i do not see why the accuracy there is any different than rain counts.

same thing for pressure.

so no matter which unit system you choose, there will be unit conversions.  using one unit system for parse_raw and a different unit system for parse_machine might make sense.

but to get there i had to strip away all of the cruft so we could see what we're working with.


problems with 0.44:

- the implementation was inconsistent in its unit conversions

- unit conversions were applied inconsistently

- raw values and converted values were logged inconsistently (they still are)

- use of units.Converter obfuscates the intent.  weewx.wxformulas.CtoF or FtoC is much easier to understand.  units.Converter is appropriate when the to- and from- units are dynamic.

- logging.  it is not consistent.  there is more logging around the problem areas than there is generally.  that is fine for development, but as the driver matures the logging should mature with it.  inconsistent logging can lead to more support problems, because end-users don't see all the messages they think they should.  or because logging is inconsistent, so logwatch and other log monitoring tools do not work as they should.

- rainRate should not be in the driver at all.  the hardware reports a value that is arbitrary, and the rainRate calculation is yet a different arbitrary calculation.  it is more appropriate to put the rainRate calculation into StdWXCalculate as an alternative calculation to the standard weewx sliding window rainRate calculation.  i left rainRate in the driver for now.

- rf tracking.  the current implementation needs to be completely rewritten.  the existing code is brittle and non-obvious.  i am pretty sure that there is no reason that this driver should also be a service.

- sensor map.  this needs to be converted to output=input pattern.  i have not yet done the conversion - i did not want to combine that commit with the disentanglement of units and logging.


please note that these are all fairly minor issues (other than sensor map - that is a big deal throughout weewx that must be addressed).

you guys have done a wonderful job so far!

maybe the market for meteostick is so tiny that the refinements i am looking at will never matter (the sdr usb sticks do much more than meteostick - if only they had a pressure sensor in them!), but we still need to make the meteostick code smell a bit less :)

m

[nls]

Nice write-up. I generally agree with all of it, only a few comments.

-- I've added only small parts but I fully agree it's bloated and many simplifications are needed.
-- Logging is all around the place and needs to be standardized.

-- I'll try to provide evidence on wind accuracy degradation. I still suspect roundoff errors somewhere. They disappeared in the US-units version of the driver. First we need to accept the possibility of a mistake deeper in system, outside of the driver.
-- "... rainRate should not be in the driver at all.  the hardware reports a value that is arbitrary, and the rainRate calculation is yet a different arbitrary calculation ..." - well, the rf packets provide a raw value, which we convert to rainfall/time period values. What do you mean by arbitrary here? I agree it should be done in a single unified way, though, to be able to provide comparable results across devices.

-- We only fully process the raw radio packet data. I'm still of the opinion that just because the machine readable values come handy sometimes, we shouldn't keep it. It's sparsely reporting certain values and generally use unknown converison algorithms (even if they are accurate and correct, we have no way to tell). I'd say as part of the simplification, we should drop it entirely from the driver.

Not completely related, but is there an SDR driver with comparable functionality? I think it wouldn't be a bad idea to use one as the radio capture part and adopt the rest of the current MS driver for the decoding.

[mwall]

On Saturday, October 1, 2016 at 10:01:37 AM UTC-4, kobuki wrote:

-- "... rainRate should not be in the driver at all.  the hardware reports a value that is arbitrary, and the rainRate calculation is yet a different arbitrary calculation ..." - well, the rf packets provide a raw value, which we convert to rainfall/time period values. What do you mean by arbitrary here? I agree it should be done in a single unified way, though, to be able to provide comparable results across devices.

the algorithm implemented in meteostick.py is simply:

3600.0 / time_between_tips * rain_per_tip

that could be implemented generically in StdWXCalculate.  however, time_between_tips could be a problem; most hardware does not report the time of the last bucket tip (some sensors do not tip).

so for now leave rain_rate in meteostick.py, with the StdWXCalculate options of hardware/software to enable/disable it.
 

-- We only fully process the raw radio packet data. I'm still of the opinion that just because the machine readable values come handy sometimes, we shouldn't keep it. It's sparsely reporting certain values and generally use unknown converison algorithms (even if they are accurate and correct, we have no way to tell). I'd say as part of the simplification, we should drop it entirely from the driver.

agreed - drop it at some point.
 
 

Not completely related, but is there an SDR driver with comparable functionality? I think it wouldn't be a bad idea to use one as the radio capture part and adopt the rest of the current MS driver for the decoding.

the weewx-sdr driver parses output from the rtl_433 program.  however, it looks like rtl_433 does not yet recognize output from davis sensors.

you might want to pick up a 20$US sdr dongle then send a pr to the rtl-433 project with the davis decodings.

someone should make a cheap usb pressure sensor, then the pressure sensor plus the sdr dongle would be an inexpensive replacement/backup for just about any weather station console.

m

[nls]

Thanks for the clarifications.

Just a quick note, since it's off-topic: one can build a receiver using an arduino clone and several cheap parts under $10 total (including baro and T/H sensors for indoors) that's compatible with the driver. So SDR is more work and less functionality, even if it's very interesting. It would be even more interesting if it was capable of receiving the transmitters without hopping using a sufficiently wide band.

[Manfred Maier]

Hi Lucas,

I‘ve seen that you have connected your solar sensor to the anemometer transmitter.

I‘m planning to do the same. Is there anything I would have to consider when doing this?

Are there any setting I have to change for the Meteostick or in the Meteostick driver for weewx? Any changes to the database structure that I have to consider?

Thanks!

Manfred