Fallback reading possible?

[michael.k...@gmx.at]

Let's say I have two sensors, "outHumidity" and "extraHumid1".

"outHumidity" ist a sensor that delivers data very reliable but not very precise measurements, it's the stations built-in sensor.  "extraHumid1" delivers data a little less reliable, but very precise values. It's a custom built ESP8266 using a SHT35, sending data over MQTT.

Is it possible to prefer extraHumid1 over outHumidity, if there are extraHumid1 values in the archive Interval? If not, use outHumidity?

Something like

[[Corrections]]

outHumidity = if extraHumid1 == 'None' outHumidity else extraHumid1

[gjr80]

Hi,

You want something like:

    outHumidity = outHumidity if extraHumid1 is None else extraHumid1

My only question is what behaviour do you see when extraHumid1 is not provided; is extraHumid1 set to None (in which case the above will work as you want) or is extraHumid1 omitted from the packet (in which case the above will produce an error and have no effect on outHumidity (which arguably is what you want)).

Gary

[michael.k...@gmx.at]

Good Question. Since I am currently only thinking about how everything could work out, I can't answer this at the moment. But I guess my question is answered. If there would be an error, how would one handle it? Or is a "nasty" log an no effect on "outHumidity" all?

[michael.k...@gmx.at]

But thinking further I have another question: will [[Corrections]] work with every loop package or every archive interval or both? If on every loop package is archive value computed from the loop data a mix of both sensors, since the sensors won't deliver their readings with the same loop package? Or won't outHumidity be affected at all, because the never is a extraHumid1 value available?

[gjr80]

Actually, looking at the StdCalibrate code if extraHumid1 is not present in the packet the error generated by the above correction expression will be ignored. So the expression above will effectively cover both cases.

Gary

[gjr80]

If extraHumid1 and outHumidity are not in that same loop packet then a correction expression will have no effect on the loop packet. If you are using hardware record generation then the correction will be applied to the archive record as well so that may do what you want for archive records. If you are using software record generation then no corrections are applied to archive records.

Gary

[michael.k...@gmx.at]

Thank you Gary. I use "prefer_hardware", so I am very confident the expression will do what I want to achieve.

[gjr80]

Ok, but I am not sure what ‘prefer_hardware’ has to do with things; it is not the same as hardware record generation nor does ‘prefer_hardware’ have anything to do with corrections. ‘prefer_hardware’ is used with the StdCalculate service, corrections are used with the StdCalibrate service and hardware or software record generation is used with the StdArchive service.

Gary

[Greg Troxel]

You may also want to think about calibration. Besides absolute
calibration there is going to be some offset or other more complicated
relationship between your two sensors. Given a "prefer precise if
available" this is going to cause some flipping betweeen them. I had a
little trouble following this thread, and perhaps StdCalibrate runs
before the choice.

But if not, and maybe you want to do this anyway, basically
cross-correlated the data from both over a wide range, calculate a
mapping function, and put that in the ESP8266 code so that it emits
values that are consistent with your other sensor.

I did this with an ESP8266 that is measuring a 12V lead-acid battery.
While I can calculate expected values from the divider resistors and the
data sheet, I ended up just measuring the battery and looking at the raw
values and figuring out a divisor, which I stored in a calibration file.

[michael.k...@gmx.at]

The the ESP just reads the SHT35 in a given interval and publishes the reading, nothing else. No calibration necessary, the sensor is really as good as advertised in the data sheet. And only the outHumidity values of the station are off when humidity > 80%rh, other values are pretty much accurate and even outHumidity is very accurate below 80%rh. Since I don't attach too much importance on historical humidity readings, mixing up different source from time to time is no big deal for me. We're talking about >99% availability with the less reliable sensor. But the current dewpoint and windchill are interesting for me, which both require some realistic humidity values.

"Ok, but I am not sure what ‘prefer_hardware’ has to do with things; it is not the same as hardware record generation nor does ‘prefer_hardware’ have anything to do with corrections. ‘prefer_hardware’ is used with the StdCalculate service, corrections are used with the StdCalibrate service and hardware or software record generation is used with the StdArchive service."

Thanks for pointing out.

[p q]

I have a secondary temperature sensor that runs on solar and a battery. This time a year in the northern hemisphere there's not enough sunlight to charge the battery sufficiently to make it through the night. I handled the failover to the main thermometer in my customized driver. Details here: https://hackaday.io/project/101680-solar-powered-wifi-temperature-sensor-for-weewx/details

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/1d8b9f9d-1aed-49db-8090-40ce627c1819n%40googlegroups.com.

--

Peter Quinn
(415)794-2264

[michael.k...@gmx.at]

@Peter Quinn, thank you. I'll check the weewx part!

Just my 2 off-topic cents: using the breadboard the voltage regulator and the UART, powered by 5V ist a very inefficient way to run this sensor. The simplest approach to get more time from the solar/battery combo would be to use a 3V solar panel (which should exceed a maximum idle voltage of 3.6V) and directly charge a LiFe battery with it. If you ensure that the ESP powers off (deep sleep) at the ESP lower limit (datasheet says 2.5V, my experience says ~2.3V) the LiFe lower discharge limit (2,0V) will probably never be reached. Even with your LiPo this should be possible this way, but you can't use all capacity, since you can't charge it over 3.6V, because this would kill the esp.

In brief:

- don't use such a breadboard or remove the voltage regulator and the UARTs supply

- connect a LiFe directly to 3V3 and GND

- connect a 3V solar panel directly to 3V3 and GND

- ensure to power off the ESP when it reaches it's minimum required voltage (maybe it's never reached), either with a protection circuit (preferred) or the ESPs internal voltage measurement ability (not preferred, because ESPs sometime run into a state where they aren't really running but drawing current from the battery until the battery is flat and damaged.

This will extend your runtime dramatically.

Currently I'm running my sensor on a 2500mAh LTO Battery, one sample a minute with deep sleep in between gives me more than a week runtime. 

[michael.k...@gmx.at]

correction: the panel shoulndn't exceed 3.6V...

[michael.k...@gmx.at]

Back to topic:
I'm using https://github.com/bellrichm/WeeWX-MQTTSubscribe/ for getting the extra readings into weewx. It's configured as a service, so every loop interval the MQTT messages in the queue will be integrated into the loop package, when I'm correct. When it comes to loop data, depending on the loop interval and the MQTT sensors interval, there are loop packages that don't contain messages from the ESP/SHT because the queue is empty. This leads to a mix of readings.

The other thing is, that the archive value also contains this mix as an average of all the values during the archive interval, because ws28xx driver doesn't support hardware record generation, I guess.

I probably have to ensure that the MQTT interval is set to a value that at least one message is in the queue every loop interval or I could do this every archive interval using something like this: https://github.com/mKainzbauer/weewx_extensions/blob/master/fronius.py, just in a more simple way that only extraHumid1 overwrites outHumidity when not None.

[Greg Troxel]

"michael.k...@gmx.at" <michael.k...@gmx.at> writes:

> I'm using https://github.com/bellrichm/WeeWX-MQTTSubscribe/ for getting the
> extra readings into weewx. It's configured as a service, so every loop
> interval the MQTT messages in the queue will be integrated into the loop
> package, when I'm correct. When it comes to loop data, depending on the
> loop interval and the MQTT sensors interval, there are loop packages that
> don't contain messages from the ESP/SHT because the queue is empty. This
> leads to a mix of readings.
> The other thing is, that the archive value also contains this mix as an
> average of all the values during the archive interval, because ws28xx
> driver doesn't support hardware record generation, I guess.
> I probably have to ensure that the MQTT interval is set to a value that at
> least one message is in the queue every loop interval or I could do this
> every archive interval using something like
> this: https://github.com/mKainzbauer/weewx_extensions/blob/master/fronius.py,
> just in a more simple way that only extraHumid1 overwrites outHumidity when
> not None.

I think this needs more sophisticated handling. The archive/loop is
as I understand it due to the Davis design, and other things are
adjusted to that.

As a side comment, I use MQTT sensors that report at 1 minute intervals.
I don't see any value in more frequent measurements, usually, and I
picked 1 min because I track them with Home Assistant not weewx and I
don't want every-second readings in my database.

Overall, I would suggest that it's an architectural bug to have to set
the MQTT update interval to exceed a loop interval. THe real question
is what you or some default says is stale vs fresh for an MQTT reading.

So I think MQTTSubscribe should perhaps be changed to have a stale
interval, keep the most recent value for each input, and if its arrival
time is within the stale interval, use it, and otherwise not. That way,
that sensor will reliably work at update rates that someone might
reasonably choose.

Another way to think about this, probably better, is that it's ok for
some loop packets to be missing an observation. With that view, then
there should be processing of both sensors independently and them some
fused sensor, an that fused sensor should have this stale notion, but
instead of MQTT arrival, have presence in loop instead. I think this
is where you are headed with only using the preferred sensor for archive
generation if it had any reports in the interval.

[michael.k...@gmx.at]

> So I think MQTTSubscribe should perhaps be changed to have a stale
> interval, keep the most recent value for each input, and if its arrival
> time is within the stale interval, use it, and otherwise not. 

That was my first thought as well, when I observed what was happening. But for the moment, things are implemented that way.

So this is my current approach: https://github.com/mKainzbauer/weewx_extensions/blob/master/usePreferred.py

[bell...@gmail.com]

Just making sure I understand this. Currently sometimes a loop packet has an extraHumid1 value and sometimes it doesn't. As long as extrahumid1 is in at least one loop packet, the archive record has what you want, the average of all the extaHumid1 loop packet values. The problem is when no loop packets have a value for extaHumid1.

If so, I think the expires_after option might be what you want. See, https://github.com/bellrichm/WeeWX-MQTTSubscribe/wiki/Configuring---additional-options#expires_after

rich

[michael.k...@gmx.at]

Is this also the case with loop values? The MQTT sensor doesn't send a value each loop interval, but several times an archive interval.   

[bell...@gmail.com]

No, the cache is only for archive records.  

So, if it is In at least one loop packet, it should be in the archive record. So are trying to 'fill in' loop packets?

[michael.k...@gmx.at]

I tried. Since it didn't work out that way I wrote my extension and for viewing "live" values in the browser I chose to view "extraHumid1" and don't care if there are no values for some reason.