Substitute readings from additional sensors for those from a weather station's sensors

[Peter Fletcher]

My (Vantage Pro 2) weather station console is located in my office/computer room, which is typically the warmest place in my house. It consequently returns an indoor temperature which is higher than it 'should' be and an indoor (relative) humidity that is lower than it 'should' be. I do have sensors in the main part of my house, whose readings are monitored and stored on the Raspberry Pi on which I also run weewx, and correspond more closely to the inside temperature and humidity for the house as a whole. I know that I can fairly easily add the readings returned by the central sensor to weewx's records and reports, but what I would really like to do is to substitute them for the misleading values coming from the weather station console. Is this possible? I am a reasonably competent Python programmer.

[p q]

I've done this for outtemp. Should be possible with inside temp. I modified the driver to do it, but there may be an easier way. https://hackaday.io/project/101680-solar-powered-wifi-temperature-sensor-for-weewx

On Thu, Jun 4, 2020 at 11:09 AM 'Peter Fletcher' via weewx-user <weewx...@googlegroups.com> wrote:

My (Vantage Pro 2) weather station console is located in my office/computer room, which is typically the warmest place in my house. It consequently returns an indoor temperature which is higher than it 'should' be and an indoor (relative) humidity that is lower than it 'should' be. I do have sensors in the main part of my house, whose readings are monitored and stored on the Raspberry Pi on which I also run weewx, and correspond more closely to the inside temperature and humidity for the house as a whole. I know that I can fairly easily add the readings returned by the central sensor to weewx's records and reports, but what I would really like to do is to substitute them for the misleading values coming from the weather station console. Is this possible? I am a reasonably competent Python programmer.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7800658a-f675-415c-b1f8-eae80f49429bo%40googlegroups.com.

--

Peter Quinn
(415)794-2264

[John Kline]

My thoughts.  You’ll want to hear others.

I would collect indoor temp on another thread and make the substitution in a function bound to NEW_LOOP_PACKET.  I typically do this for adding (e.g., pm2_5), but replacing should be no different.  You’ll want the separate thread to save off the current indoor temp and a time stamp.  When you go to insert it in NEW_LOOP_PACKET, if it is too old (by your definition), instead of replacing, remove the indoor temp from the packet. 

On Jun 4, 2020, at 11:34 AM, p q <peterq...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAA1SM22-EGrdP1CwEfh01R1bZ0mSKGJQF0ndS84_FXDiJGGHtw%40mail.gmail.com.

[Peter Fletcher]

Thanks! The sensor data is already being collected and stored by another systemd-controlled process, so I would just need to modify that slightly to communicate it to the weewx function - probably via a file in /var/tmp. I think that binding to NEW_LOOP_PACKET does what I want better than just archiving the corrected data and modifying a skin for display. I may or may not save the console values in one of the 'extra' slots.

On Thursday, June 4, 2020 at 3:08:05 PM UTC-4, John Kline wrote:

My thoughts.  You’ll want to hear others.

I would collect indoor temp on another thread and make the substitution in a function bound to NEW_LOOP_PACKET.  I typically do this for adding (e.g., pm2_5), but replacing should be no different.  You’ll want the separate thread to save off the current indoor temp and a time stamp.  When you go to insert it in NEW_LOOP_PACKET, if it is too old (by your definition), instead of replacing, remove the indoor temp from the packet. 

On Jun 4, 2020, at 11:34 AM, p q <peterq...@gmail.com> wrote:



I've done this for outtemp. Should be possible with inside temp. I modified the driver to do it, but there may be an easier way. https://hackaday.io/project/101680-solar-powered-wifi-temperature-sensor-for-weewx

On Thu, Jun 4, 2020 at 11:09 AM 'Peter Fletcher' via weewx-user <weewx...@googlegroups.com> wrote:

My (Vantage Pro 2) weather station console is located in my office/computer room, which is typically the warmest place in my house. It consequently returns an indoor temperature which is higher than it 'should' be and an indoor (relative) humidity that is lower than it 'should' be. I do have sensors in the main part of my house, whose readings are monitored and stored on the Raspberry Pi on which I also run weewx, and correspond more closely to the inside temperature and humidity for the house as a whole. I know that I can fairly easily add the readings returned by the central sensor to weewx's records and reports, but what I would really like to do is to substitute them for the misleading values coming from the weather station console. Is this possible? I am a reasonably competent Python programmer.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7800658a-f675-415c-b1f8-eae80f49429bo%40googlegroups.com.

--

Peter Quinn
(415)794-2264

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Graham Eddy]

if the readings are inside weewx, just use [calibrate] section to overwrite the variable.

if they are outside, you will need to write a simple data service, see one of the examples. the packet is just a dictionary keyed on obs_type so there is no programming difference between adding a value and replacing a value

____________

Graham Eddy

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAA1SM22-EGrdP1CwEfh01R1bZ0mSKGJQF0ndS84_FXDiJGGHtw%40mail.gmail.com.

[Peter Fletcher]

The modifications are now up and running, and I took the opportunity (uneventfully!) to reinstall weewx to run under Python3, since all my other scripts use this Python version. The service code is below. It is modeled on 'p q's code from the hackaday article, with a number of modifications. The only change required for Python3 was a minor modification to the Try Except syntax. Most of the modifications to the Loop packet service were to reduce the load on the system for something that (for the Vantage Pro 2 driver) gets called once every couple of seconds or so. The service only actually reads the external data every minute or so, returning saved values for the intervening calls.

import syslog
import weewx
import time
import os

from weewx.wxengine import StdService

class ArchTHService(StdService):
    def __init__(self, engine, config_dict):
        super(ArchTHService, self).__init__(engine, config_dict)
        d = config_dict.get('ArchTHService', {})
        self.filename = d.get('filename', '/var/tmp/THNow.txt')
        syslog.syslog(syslog.LOG_INFO, "ArchTH: using %s" % self.filename)
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.read_file)

    def read_file(self, event):
# save the console values in 'extra' slots
        event.record['extraTemp1'] = event.record['inTemp']
        event.record['extraHumid1'] = event.record['inHumidity']
        try:
            #skip it if it's stale. Console values will be used instead
            if time.time() - os.path.getmtime("/var/tmp/THNow.txt") < 600: #10 minutes
                with open(self.filename) as f:
                    line = f.read() # contains temp & humidity, comma-separated
                values=line.split(',')
                syslog.syslog(syslog.LOG_DEBUG, "ArchTH: found value of %s" % line)
                event.record['inTemp'] = float(values[0])
                event.record['inHumidity'] = float(values[1])
        except Exception as e:
            syslog.syslog(syslog.LOG_ERR, "ArchTH: cannot interpret value: %s" % e)

class LoopTHService(StdService):
    loopcount= 0
    lastT = 0
    lastH =0

    def __init__(self, engine, config_dict):
        super(LoopTHService, self).__init__(engine, config_dict)
        d = config_dict.get('LoopTHService', {})
        self.filename = d.get('filename', '/var/tmp/THNow.txt')
        syslog.syslog(syslog.LOG_INFO, "LoopTH: using %s" % self.filename)
        self.bind(weewx.NEW_LOOP_PACKET, self.read_file)

    def read_file(self, event):
        if self.loopcount == 0:
            try:
                #skip it if it's stale. Vantage Console values will be used instead
                if time.time() - os.path.getmtime("/var/tmp/THNow.txt") < 300: #5 minutes
                    with open(self.filename) as f:
                        line = f.read() # contains temp & humidity, comma-separated
                    values=line.split(',')
                    syslog.syslog(syslog.LOG_DEBUG, "LoopTH: found value of %s" % line)
                    event.packet['inTemp'] = float(values[0])
                    event.packet['inHumidity'] = float(values[1])
                    self.lastT=float(values[0])
                    self.lastH=float(values[1])
                    self.loopcount += 1
            except Exception as e:
                syslog.syslog(syslog.LOG_ERR, "LoopTH: cannot interpret value: %s" % e)  
        else:
            self.loopcount += 1
            if self.loopcount >= 30:
                self.loopcount = 0
            event.packet['inTemp'] = self.lastT
            event.packet['inHumidity'] = self.lastH
 

[John Kline]

You just have to be careful how you write the file (the external process).  You want something along the lines of writing to a temp file, syncing to flush buffers and then moving that file to the filename that will be read by weewx.

On Jun 5, 2020, at 8:41 AM, 'Peter Fletcher' via weewx-user <weewx...@googlegroups.com> wrote:



The copy and paste for the code in my previous post didn't work properly. The last two lines of code should be as shown below.

            event.packet['inTemp'] = self.lastT
            event.packet['inHumidity'] = self.lastH

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/4f181486-590e-4977-a740-46614d9da27do%40googlegroups.com.

[Peter Fletcher]

On Friday, June 5, 2020 at 12:09:47 PM UTC-4, John Kline wrote:

You just have to be careful how you write the file (the external process).  You want something along the lines of writing to a temp file, syncing to flush buffers and then moving that file to the filename that will be read by weewx.

 

I do not see why this is necessary, or (if you are trying to avoid weewx reading a partially written file) why your approach is necessarily better than my much simpler one. In the writing script, I am using Python's 'with open(....' syntax, so the (very small) file is opened, written to, and then immediately flushed and closed. I am not sure that it is even possible for the file to be in a state that would cause a problem when the weewx service tries to read it.

[John Kline]

> I am not sure that it is even possible for the file to be in a state that would cause a problem when the weewx service tries to read it.

If you are saying that you don’t think it’s possible for the file to be read in an inconsistent state, let’s just agree to disagree on that one.

If you are saying that it doesn’t matter if the file is read in an inconsistent state; that’s a deeper thought experiment.

In any event, I’m happy that you got this running.

On Jun 5, 2020, at 9:32 AM, 'Peter Fletcher' via weewx-user <weewx...@googlegroups.com> wrote:



On Friday, June 5, 2020 at 12:09:47 PM UTC-4, John Kline wrote:

You just have to be careful how you write the file (the external process).  You want something along the lines of writing to a temp file, syncing to flush buffers and then moving that file to the filename that will be read by weewx.

 

I do not see why this is necessary, or (if you are trying to avoid weewx reading a partially written file) why your approach is necessarily better than my much simpler one. In the writing script, I am using Python's 'with open(....' syntax, so the (very small) file is opened, written to, and then immediately flushed and closed. I am not sure that it is even possible for the file to be in a state that would cause a problem when the weewx service tries to read it.

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b6aef2bf-f95a-4b17-9677-807bb9a53fbeo%40googlegroups.com.

[Peter Fletcher]

On Friday, June 5, 2020 at 12:44:11 PM UTC-4, John Kline wrote:

> I am not sure that it is even possible for the file to be in a state that would cause a problem when the weewx service tries to read it.

If you are saying that you don’t think it’s possible for the file to be read in an inconsistent state, let’s just agree to disagree on that one.

If you are saying that it doesn’t matter if the file is read in an inconsistent state; that’s a deeper thought experiment.

In any event, I’m happy that you got this running.

I am saying that, in the very specific circumstances we are discussing, it is (at the least) very improbable indeed that the file will be in an inconsistent or problematic state when the weewx service tries to read it. In the more general case, and particularly with longer files, and/or when the 'write' side is appending to a file, rather than overwriting an existing file, there are a lot more ways in which things can go wrong, and one's coding has to take account of the possibilities.

On the question of whether it matters if the file is very occasionally in an inconsistent state when it is read: again, in these very specific circumstances, it probably doesn't - at worst, one set of data may be lost, but. in fact, I believe that the desired data substitution would not take place for that packet or record.

In any event, I also still don't see, in these circumstances, why you believe that your suggested approach would be any better. Deciding on that would require very much more detailed knowledge of the internals of the file system's operations than I would claim to have.