Add additional rain gauge via second data source

[engolling]

Hello community,

I would like to add an additional rain gauge as additional source described in the customization guide.

Can you give me any hints how to do this the easiest way, to get the daily, weekly and monthly percipation.

Can I use any modules or calculations that are already done inside the system or is this normally done by the corresponding driver, so I have to handle all the signals precalculated.

I hope you understand what I mean.

Best Regards,

engolling

[Andrew Milner]

weewx requires rain during archive interval for storing in the database archive records.  A driver may have to convert whatever it receives (eg running total) so as to obtain the value for the interval.  Daily is accumulated by weewx from the archive interval values and weekly and monthly are derived form the daily totals.  This is possibly an over simplification - but should give the idea.

so if the second source gives a value for rainfall in interval it should be enough for weewx to derive the remainder.

[engolling]

Hello,

I tried to implement a driver providing the rainfall in intervall but until now it does not work as expected.

I'am able to handle the correct data to WeeWx with this command:

                        syslog.syslog(syslog.LOG_DEBUG, "WeatherDuino: " + str(names[n+1]) + ": " + str(deltarain))
                        event.record[str(names[n+1])] = float(deltarain)

The debug log output says:

Mar  3 11:56:16 WeatherDuinoPi weewx[1366]: WeatherDuino: Rain_RG11: 0.17716535433

But in the database there is a "0" logged.

If i change the code hardcoding the rain intervall to 10 it works fine.

                        syslog.syslog(syslog.LOG_DEBUG, "WeatherDuino: " + str(names[n+1]) + ": " + str(10))
                        event.record[str(names[n+1])] = 10

So I think my driver is executed more often then my one minute logging intervall and so the value of the event.record is overwritten with a zero again - because the driver thinks the value is already in the WeeWx database.

You can find my code here:

https://github.com/menachers/WeatherDuino/tree/master/WeeWx_Plugin

It is embedded in the weewx.conf:

#   This section configures the internal weewx engine.

[Engine]
   
    [[Services]]
        # This section specifies the services that should be run. They are
        # grouped by type, and the order of services within each group
        # determines the order in which the services will be run.
        prep_services = weewx.engine.StdTimeSynch
        data_services = user.WeeWx_WeatherDuino_Logger_plugin.WeeWxService,
        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate
        archive_services = weewx.engine.StdArchive
        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS
        report_services = weewx.engine.StdPrint, weewx.engine.StdReport

Regards,

engolling

[engolling]

Short update:

I have found out that my plugin is executed via data_services each time a loop package is generated and this is approximatly every 3 seconds.

An archive dataset is written every 60 seconds, leading to chance of 1/20 that the raindelta is saved correctly - thats too less :-D

So I see two possible solutions:

1. The data service is only executed once before the archiv dataset is written. Has anybody a idea how this could be done?

2. I have access to some kind of software flag saying that a archive record has been written.

I had also a look into the vantage driver and I think this piece of code does the magic of calculating the data:

        # Because the Davis stations do not offer bucket tips in LOOP data, we
        # must calculate it by looking for changes in rain totals. This won't
        # work for the very first rain packet.
        if self.save_monthRain is None:
            delta = None
        else:
            delta = loop_packet['monthRain'] - self.save_monthRain
            # If the difference is negative, we're at the beginning of a month.
            if delta < 0: delta = None
        loop_packet['rain'] = delta
        self.save_monthRain = loop_packet['monthRain']

        return loop_packet

But I do not understand how overwriting the delta is prevented here.

Hoping for some replys.

Best wishes,

engolling

[gjr80]

Hi,

When using a data service to add data to WeeWX you have two choices, you either add your data to the archive records or you add your data to the loop packets. Either way will work. If you decide that your data service is to augment archive records with your rainfall data then your data service must (1) bind to the event NEW_ARCHIVE_RECORD and (2) add the total rainfall seen by your sensor during that archive period (in your case that is the last 1 minute) to the archive record. If you decide your data service is to augment the loop packets then your service must (1) bind to the NEW_LOOP_PACKET event and (2) add the total rainfall seen since the last loop packet you augmented to the loop packet concerned. Note that you do not need to add your data to every loop packet, you can add it to whichever loop packets you wish, but the value you add must be the total rainfall since the last loop packet you augmented. The WeeWX internal machinery will then take care of accumulating the loop data and your accumulated data will appear in the archive record generated by WeeWX.

Which way you decide to go is up to you. If you decide to augment the archive record then your service would need to (in your case) obtain the total rainfall for each 1 minute archive period, this may or may not be easy depending on your hardware. The key here is you need to stick to the archive period. If you decide to augment loop packets you can augment whenever you want, you can skip loop packets if you want as long as you stick to the rule of providing total rainfall since you last augmented a loop packet. The loop packet approach is probably simpler, you just add data when ever you can/want; you are not forced to rigidly stick to a fixed interval as you are with the archive approach.

In terms of saving data to the archive there are two conditions that must be met. Firstly, the field/data you wish to save must appear in the WeeWX generated archive record (you can see the WeeWX generated loop packets and archive records by running weeWX directly). Secondly, your archive table schema must contain a field with the same name as the field of interest in the archive record. So if your service added a field named 'rain2' to the archive records (or loop packets) and the archive table your WeeWX database had a field named 'rain2', then WeeWX would automatically save your rain2 data to the archive. The steps involved in changing your schema are covered in the Customization Guide under Adding a new type to the database. Note there are other approaches to saving new data to archive (eg creating a second database) but these approaches are usually more complex or more involved so I have ignored them in this case.

You might also find that posting your code may help as that way we can see exactly what you are/are not doing.

Gary

[engolling]

Hi Gary,

thanks for your very interesting reply. I tried to augment my extra data to the NEW_LOOP_PACKET, whichs works in priciple for all my data except the rain delta.

As I mentioned it seems that the rain delta values are getting sorted out somehow.

You can find the code here:

https://github.com/menachers/WeatherDuino/tree/master/WeeWx_Plugin

and an excerpt of my weewx.conf in an earlier post.

So if I hardcode a deltarain of 0.1[in] it sums up correctly with each archive record - so after 5 minutes I get a total rain of 0.5[in]. If it is augmented to one loop packet (beeing emitted aproximatly every 3 seconds) i can also see it in the live display of the loop packet, but it can not be found in the archive record. So it seems that is is not accumulated internaly.

Best regards,

engolling

[gjr80]

Hi,

I had a look through your code and I think you have made things overly complex for yourself. As I understand your setup you have a file, /home/pi/WeatherDuino/WeeWx_Exp.txt,  that contains your WeatherDuino data, there are three rows of semicolon separated data, the rows being labels/names, units and values. One of those fields is a cumulative rain value. You also use the file /home/pi/WeatherDuino/Rain_tmp.txt that you use in an elaborate arrangement to read and save the last cumulative rain value and you obtain the delta rain value by taking the difference between the current cumulative rain value and the value read from /home/pi/WeatherDuino/Rain_tmp.txt.

The logic appears to be sound but you are going about it in a very complex manner and I suspect that is tripping you up. The need to determine delta rain from a stream of cumulative rain values is common in many WeeWX drivers. the approach taken in theses drivers is to create a property, let's call it last_rain that is initialised to None in the drivers __init__(). Then when the driver obtains a cumulative rain value, let's call it cumulative_rain, there is a simple check whether last_rain is None, if it is then delta_rain is set to None, if last_rain is not None the delta_rain  is set to cumulative_rain - last_rain. Finally last_rain is set to cumulative_rain. In code it might look like this:

class SomeDriver():

    def __init__():
        ....
        self.last_rain = None
        ....

    def get_delta_rain(cumulative_rain):

        if self.last_rain is not None:
            delta_rain = cumulative_rain - self.last_rain
        else:
            delta_rain =None
        self.last_rain = cumulative_rain
        return delta_rain

I have left a lot of things out there but the point was to illustrate the overall logic and structure. By using the last_rain property there is no need to write temporary values to file; your code is a lot neater, simpler to understand and faster. The example I have used is for a driver, your service, whilst not a driver, operates in the same manner in many respects so you should be able to apply a similar construct in your service. You don' need a separate get_delta_rain method if you don't want, you can put it all in your read_file method.

As for data not appearing in your database there are two conditions that need to be met (1) your data needs ot appear in the archive record generated by WeeWX and (2) the archive field name containing your data needs to be included in the archive table schema. In your case your service should take care of (1). At the moment I am not convinced that it is operating as intended, but if you simplify/restructure your code as suggested we will get there. As for (2) I am not convinced that your code for setting the modified schema is doing as you think. Have you actually looked in your database to see what schema has been implemented? You can easily check the schema of a SQLite database table using the sqlite3 utility (note you may need to install it on your machine using $ sudo apt-get install sqlite3). To check the schema (assuming your database is in /home/weewx/archive - it may be in /var/lib/weewx):

$ sqlite3 /home/weewx/archive/weewx.sdb
sqlite> .schema archive
<schema will be listed here>
sqlite> .q

What does the above show you, is field Rain_RG11 in the schema?

Gary

[engolling]

Hello Gary,

 

thanks fot your efforts. You are right with your description how my code should work.

Let's say my main problem is that I know how I construct software in principle, but my python knowledgde is not very good, so in order to safe the last cumulative rain value my only solution was to write it into a file. Otherwise I tought the value might get lost between executing the data service

 

I will try to 'buffer' my last rain value in a variable as you describe it and give you feedback.

But I still think WeeWx does not accumulate correctly in my case, because im nearly 100% sure, that I have checked that my (indeed strange) code emits the values correctly.

My schema looks like this - I think it is fine.

sqlite> .schema archive
CREATE TABLE archive (`dateTime` INTEGER NOT NULL UNIQUE PRIMARY KEY, `usUnits` INTEGER NOT NULL, `interval` INTEGER NOT NULL, `barometer` REAL, `pressure` REAL, `altimeter` REAL, `inTemp` REAL, `outTemp` REAL, `inHumidity` REAL, `outHumidity` REAL, `windSpeed` REAL, `windDir` REAL, `windGust` REAL, `windGustDir` REAL, `rainRate` REAL, `rain` REAL, `dewpoint` REAL, `windchill` REAL, `heatindex` REAL, `ET` REAL, `radiation` REAL, `UV` REAL, `extraTemp1` REAL, `extraTemp2` REAL, `extraTemp3` REAL, `soilTemp1` REAL, `soilTemp2` REAL, `soilTemp3` REAL, `soilTemp4` REAL, `leafTemp1` REAL, `leafTemp2` REAL, `extraHumid1` REAL, `extraHumid2` REAL, `soilMoist1` REAL, `soilMoist2` REAL, `soilMoist3` REAL, `soilMoist4` REAL, `leafWet1` REAL, `leafWet2` REAL, `rxCheckPercent` REAL, `txBatteryStatus` REAL, `consBatteryVoltage` REAL, `hail` REAL, `hailRate` REAL, `heatingTemp` REAL, `heatingVoltage` REAL, `supplyVoltage` REAL, `referenceVoltage` REAL, `windBatteryStatus` REAL, `rainBatteryStatus` REAL, `outTempBatteryStatus` REAL, `inTempBatteryStatus` REAL, `AVG_Rcv_RX0` REAL, `AVG_Rcv_RX1` REAL, `AVG_Rcv_RX2` REAL, `BatVolt_0` REAL, `SysTemp_0` REAL, `Rsfan_0` REAL, `PacketsSentPerHour_0` REAL, `Snow_Height` REAL, `BatVolt_1` REAL, `SysTemp_1` REAL, `Rsfan_1` REAL, `PacketsSentPerHour_1` REAL, `BatVolt_2` REAL, `SysTemp_2` REAL, `Rsfan_2` REAL, `PacketsSentPerHour_2` REAL, `Soil_Temp_Full1` REAL, `Soil_Temp_Full2` REAL, `Soil_Temp_Full3` REAL, `Soil_Temp_Full4` REAL, `AQI_PM1_0` REAL, `AQI_PM2_5` REAL, `AQI_PM10_0` REAL, `AQI_Index` REAL, `AQI_Temp` REAL, `AQI_Hum` REAL, `CO2` REAL, `GAS_2` REAL, `WiFi_T0` REAL, `WiFi_H0` REAL, `Signal_Quality_TX0` REAL, `Signal_Quality_TX1` REAL, `Signal_Quality_TX2` REAL, `Rain_RG11` REAL, `Rain_Rate_RG11` REAL, `Rain_TX2` REAL, `Rain_Rate_TX2` REAL);

Thank you,

engolling

[gjr80]

OK, your archive schema appears fine, that rules out one variable. WeeWX services are loaded at WeeWX startup and are retained as long as WeeWX continues to run so any properties you add are retained. Have a go at simplifying your service then I suggest you run WeeWX directly so you can see the loop packets and archive records that are generated. It should be clear then what is/is not going on.

Gary

[engolling]

Hi Gary,

maybe I have found my problem after looking at my code and your advices again.
Because it seems that I have bound my additional signals to the archive record and not to the LOOP packet

class WeeWxService(StdService):
    def __init__(self, engine, config_dict):
        super(WeeWxService, self).__init__(engine, config_dict)      
        d = config_dict.get('WeatherDuino_logger_service', {})
        self.filename = d.get('filename', '/home/pi/WeatherDuino/WeeWx_Exp.txt')
        syslog.syslog(syslog.LOG_INFO, "WeatherDuino: using %s" % self.filename)
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.read_file)
        self.last_rain = []

But the logfile tells me my plugin is executed in a frequency like the loop packets:
Mar 17 23:22:19 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:23 WeatherDuinoPi weewx[7972]: manager: Added record 2019-03-17 23:21:00 CET (1552861260) to database 'weewx.sdb'
Mar 17 23:22:23 WeatherDuinoPi weewx[7972]: manager: Added record 2019-03-17 23:21:00 CET (1552861260) to daily summary in 'weewx.sdb'
Mar 17 23:22:24 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:27 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:17:00 CET (1552861020) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 17 23:22:27 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:30 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:18:00 CET (1552861080) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 17 23:22:30 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:33 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:19:00 CET (1552861140) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 17 23:22:33 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found

According to your answers this should be the problem because in this case I have to take care that the rain is only augmented once each minute, right?

Should I change the binding to

        self.bind(weewx.NEW_LOOP_PACKET, self.read_file)

?

Moreover I try to simplify my code as you have proposed, so I added the last line in the init function (is it a function ?) of the class.
My idea was to make a empty list, which I can expand as I want because there can be up to 4 rain values - is this a reasonable solution or do I have to make a list with 4 entries at the beginning?

Thanks for all your help.

Best regards,
engolling

[gjr80]

On Monday, 18 March 2019 08:36:24 UTC+10, engolling wrote:

Hi Gary,

maybe I have found my problem after looking at my code and your advices again.
Because it seems that I have bound my additional signals to the archive record and not to the LOOP packet

There is nothing wrong with augmenting archive records. As I pointed out in my first post there are two ways to add additional sensor data to WeeWX. One approach is to augment the archive record and to do this your would bind to NEW_ARCHIVE_RECORD. If you augment archive records you will not see any of your data in loop packets, this is normal expected behaviour. The other approach is to augment loop packets, to do this you must bind to NEW_LOOP_PACKET. If you augment loop packets your should see your data in loop packets and archive records. Note that if augmenting loop packets you may not necessarily see your data in every loop packet, it depends on how you implement your service. Neither approach is right or wrong, the approach you use is what best suits you or your setup.

 

class WeeWxService(StdService):
    def __init__(self, engine, config_dict):
        super(WeeWxService, self).__init__(engine, config_dict)      
        d = config_dict.get('WeatherDuino_logger_service', {})
        self.filename = d.get('filename', '/home/pi/WeatherDuino/WeeWx_Exp.txt')
        syslog.syslog(syslog.LOG_INFO, "WeatherDuino: using %s" % self.filename)
        self.bind(weewx.NEW_ARCHIVE_RECORD, self.read_file)
        self.last_rain = []

But the logfile tells me my plugin is executed in a frequency like the loop packets:
Mar 17 23:22:19 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:23 WeatherDuinoPi weewx[7972]: manager: Added record 2019-03-17 23:21:00 CET (1552861260) to database 'weewx.sdb'
Mar 17 23:22:23 WeatherDuinoPi weewx[7972]: manager: Added record 2019-03-17 23:21:00 CET (1552861260) to daily summary in 'weewx.sdb'
Mar 17 23:22:24 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:27 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:17:00 CET (1552861020) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 17 23:22:27 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:30 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:18:00 CET (1552861080) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 17 23:22:30 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:33 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:19:00 CET (1552861140) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 17 23:22:33 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found

According to your answers this should be the problem because in this case I have to take care that the rain is only augmented once each minute, right?

I am not sure what is going on with your system but the log extract above shows that WeeWX is attempting to save an archive record every few seconds - refer to the following entries:

Mar 17 23:22:27 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:17:00 CET (1552861020) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

Mar 17 23:22:30 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:18:00 CET (1552861080) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

Mar 17 23:22:33 WeatherDuinoPi weewx[7972]: manager: Unable to add record 2019-03-17 23:19:00 CET (1552861140) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

This is not normal operation. Because something it triggering an archive record to be saved every few seconds the NEW_ARCHIVE_RECORD event is also triggered which causes your service to fire as well. The issue here is not your service but whatever is causing this archive record to be written every few seconds. I suggest we step back a bit and get a clear picture of how your system is configured and what it is running. I suggest you:

1. stop WeeWX

2. run the wee_debug utility and post a copy of the wee_debug output. Make sure any sensitive info is obscured before posting the wee_debug output, wee_debug will obscure most sensitive info but it is not perfect.

3. post a copy of your service file (bin/user/WeeWx_WeatherDuino_Logger_plugin.py ?) from your RPi, it is important we know exactly what software you are running rather than looking at a github repo that may or may not be the same as on your machine.

 

Should I change the binding to

        self.bind(weewx.NEW_LOOP_PACKET, self.read_file)

?

Moreover I try to simplify my code as you have proposed, so I added the last line in the init function (is it a function ?) of the class.
My idea was to make a empty list, which I can expand as I want because there can be up to 4 rain values - is this a reasonable solution or do I have to make a list with 4 entries at the beginning?

I would suggest not doing too much else as until the issue with the frequent archive records is solved your service will not operate correctly irrespective.

Gary

[engolling]

Hi Gary,

thanks for your patience.

This is not normal operation. Because something it triggering an archive record to be saved every few seconds the NEW_ARCHIVE_RECORD event is also triggered which causes your service to fire as well. The issue here is not your service but whatever is causing this archive record to be written every few seconds. I suggest we step back a bit and get a clear picture of how your system is configured and what it is running.

I was wondering where these errors where coming from and I already tried to get more information about it. I found somewhere a statement that is not too bad and might happen if a very narrow saving interval is used. But now I increasingly understand...

Now I have attached the output of wee_debug --info an excerpt of the syslog with debug option disabled and the lastest version of my WeatherDuino_Logger_plugin.

Regards,

engolling

[engolling]

One more thing - I do not own a original davis station - I'm using a WeatherDuino, emulating a davis station for import of "standard" data.

https://www.meteocercal.info/forum/index.php

My plan is to get some extra data out of my hardware wich can not be transmitted via the LOOP packets e.g. the readings of a second rain gauge like the RG11.

engolling

[gjr80]

On Tuesday, 19 March 2019 07:57:06 UTC+10, engolling wrote:

Hi Gary,

thanks for your patience.

This is not normal operation. Because something it triggering an archive record to be saved every few seconds the NEW_ARCHIVE_RECORD event is also triggered which causes your service to fire as well. The issue here is not your service but whatever is causing this archive record to be written every few seconds. I suggest we step back a bit and get a clear picture of how your system is configured and what it is running.

I was wondering where these errors where coming from and I already tried to get more information about it. I found somewhere a statement that is not too bad and might happen if a very narrow saving interval is used. But now I increasingly understand...

Errors involving 'Unable to add record' are not good and are a sign something is wrong, at the very least you are wasting processor time, something you probably should avoid with such a short archive interval. A short archive interval should not be the cause in itself

On Tuesday, 19 March 2019 08:21:12 UTC+10, engolling wrote:

One more thing - I do not own a original davis station - I'm using a WeatherDuino, emulating a davis station for import of "standard" data.

https://www.meteocercal.info/forum/index.php

OK, not saying that is the issue but it is another complicating factor. Your weewx.conf looks fine and I don't see anything in your service that would be causing the archive record problem. Let's step right back to basics and disable your service so that you have WeeWX running just the vantage driver to talk the WeatherDuino. To do this:

1. edit weewx.conf, and change:

[Engine]
   
    [[Services]]
        # This section specifies the services that should be run. They are
        # grouped by type, and the order of services within each group
        # determines the order in which the services will be run.
        prep_services = weewx.engine.StdTimeSynch
        data_services = user.WeeWx_WeatherDuino_Logger_plugin.WeeWxService,
        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate

to

[Engine]
   
    [[Services]]
        # This section specifies the services that should be run. They are
        # grouped by type, and the order of services within each group
        # determines the order in which the services will be run.
        prep_services = weewx.engine.StdTimeSynch

        data_services = , #user.WeeWx_WeatherDuino_Logger_plugin.WeeWxService,

        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate

2. save weewx.conf

3. restart WeeWX and let it run for 10 minutes or so then take a copy of the log from when WeeWX was restarted and post it here, make sure you capture the full WeeWX startup

This should show us whether the WeatherDuino/vantage driver is causing the problem or if it is your service.

Gary

[engolling]

Hi Gary,

I disabled my data service and the errors still were there. See the logfile in attachment.

So I assumed that there is a problem with the vantage driver in combination with my hardware.

So I changed in weewx.conf to

    # If possible, new archive records are downloaded from the station
    # hardware. If the hardware does not support this, then new archive
    # records will be generated in software.
    # Set the following to "software" to force software record generation.
    record_generation = software

this solves the problem with the logging errors.

With the "unable to add records" problem solved I finished my data service - also in attachment - which seems to be working now :).

Back to the issue with the archive records - I read in the customization guide, that in case of the record generation is set hardware, the archive records are directly genereted by the hardware. So I assume, that my hardware emitts an archive record to often.

As I have seen in the vantage protocol descriptin the archive data can be requested via the DMPAFT command and sending the timestamp of the last received archive record.

I think somewhere is a slight different behaviour between my WeatherDuino and an original Davis device - because the developper of the WeatherDuino says the emulation is compatible with the original Davis software and nearly each other weather software. I can confirm this.

Maybe it is necessary to use some kind of port sniffer to check the communication or do you know if there are additional debug outputs of the driver available.

Best regards,

engolling

[gjr80]

It seems that the WeatherDuino emulation of the Davis logger may still have some issues. It's possible that the WeeWX vantage driver has some issues but the driver is mature and has been in use for many years communicating with actual Davis hardware without issue. One thing I have noticed with some of the other PWS software packages is that they seem to operate using the vantage loop packets only, whereas WeeWX has the option of operating with loop packets only (record_generation=software) or a combination of loop packets and hardware archive records (record_generation=hardware). That may explain why no one else sees this issue. I am certain I have seen another WeeWX user who was using the WeatherDuino as a Davis emulator and there were some clear issues with the emulator. Unfortunately I cannot find that thread.

You can obtain extra debug info in the log by setting debug=1 in weewx.conf, though for the vantage driver I doubt this will give you too much more information. The vantage driver does not have the debug capabilities that some other driver have so that you can see individual commands and responses.

Gary

[engolling]

Hi Gary,

I checked the emulation of the WeatherDuino in the code and from my point of view it seems to be implemented as in the protocol description.

Nevertheless it might differ in any behaviour which is not exactly defined in the description.

It's very hard for me to understand the python code of the vantage driver since I am not familiar with the language.

I estimate that the vantage driver make the weewx engine to save an archive record each time it gets one from the hardware.

According to my observations und the upper preconditions it seems that archive records are received in the same frequency as the LOOP packets.

This would mean that the vantage driver polls with a DMPAFT command each view seconds.

Do you have any knowledge about this matter and do you know how it should work correctly?

To get this issue solved I think it would be necessary to get a serial sniffer in place to be able to see the communications between both devices.

Can I simply add some logs in the source of the vantage driver, like I did in my data service?

Regards,

engolling

[gjr80]

The vantage driver is somewhat complex. How the vantage driver operates depends on whether hardware or software record generation is in use. If software record generation is in use then the vantage driver obtains loop packets from the console (using the LOOP command) and the loop packets are then passed to WeeWX. This goes on continuously. WeeWX accumulates the loop packets and at the end of each archive interval WeeWX synthesises an archive record from the accumulated loop packets. This archive record is then further processed by various services, saved to database and used as necessary in report generation.

If hardware record generation is in use then the same loop packet processes occur as for software record generation (ie the driver passes loop packets to WeeWX and WeeWX accumulates them) but at the end of the archive interval instead of WeeWX synthesising an archive record from the accumulated loop packets, WeeWX asks the driver to obtain an archive record from the console. The driver uses the DMPAFT command to obtain the archive record and this archive record is passed back to WeeWX. This hardware archive record is further processed by various services, saved to database and used as necessary in report generation.

As far as I know the DMPAFT command is only used at the end of the archive interval (eg every 5 minutes) and not very few seconds. I think to dig any further into the issue would take some detailed analysis of how the WeatherDuino responds to various commands. I suspect that somewhere the behaviour when the WeatherDuino receives the DMPAFT command is slightly different to the Davis console/logger. You should be able to instrument the vantage driver to log anything from raw data from the console/logger through to the decoded observations and packets/records. The good thing is the Davis protocols are documented and in the public domain so it should be easy to compare actual and expected responses.

Gary

[engolling]

Hi Gary,

I tried to debug the system and therefore I built a "serial sniffer" in form of an Arduino Mega passing through the data between serial0 and serial1 and printing all messages out on serial2.

I used HTERM to display it and I will provide some screenshots for a better visibility.

Here we see a typical archive request:

1. DMPAFT\n - request of WeeWx

2. WeatherDuino acknowledge with 0x06

3. WeeWx sends Date, Time and Checksum

4. WeatherDuino acknowledge with 0x06

5. WeatherDuino sends number of pages and checksum

6. WeeWx acknowledge

7. WeatherDuino sends page and overhead - in sum 267 bytes as expected.

8. WeeWx sends LineFeed, but should acknowledge, send a CRC failure or skip - that is strange...

See documentation line 35 and 36

https://www.davisinstruments.com/support/weather/download/VantageSerialProtocolDocs_v261.pdf

1 Minute and 360ms later the next DMPAFT command is sent.

This example is starting today at 19:05:16 local time - this is written into the logfile.

Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added record 2019-03-26 19:04:00 CET (1553623440) to database 'weewx.sdb'
Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added record 2019-03-26 19:04:00 CET (1553623440) to daily summary in 'weewx.sdb'
Mar 26 19:05:19 WeatherDuinoPi weewx[17760]: manager: Unable to add record 2019-03-26 19:00:00 CET (1553623200) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 26 19:05:21 WeatherDuinoPi weewx[17760]: manager: Unable to add record 2019-03-26 19:01:00 CET (1553623260) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 26 19:05:31 WeatherDuinoPi weewx[17760]: cheetahgenerator: Generated 9 files for report StandardReport in 10.25 seconds
Mar 26 19:06:19 WeatherDuinoPi weewx[17760]: manager: Added record 2019-03-26 19:05:00 CET (1553623500) to database 'weewx.sdb'
Mar 26 19:06:20 WeatherDuinoPi weewx[17760]: manager: Added record 2019-03-26 19:05:00 CET (1553623500) to daily summary in 'weewx.sdb'
Mar 26 19:06:23 WeatherDuinoPi weewx[17760]: manager: Unable to add record 2019-03-26 19:01:00 CET (1553623260) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

In my opinion the WeatherDuino is acting as described in the protocol description. It is also a bit strange that WeeWx tries to add the "older" records now to the database which are also saved in the same page of the flash. But as it does not acknowledge the received page something strange is going on at this point.

Maybe you or somebody else owning an original Davis station has an idea.

Regards,

engolling

[gjr80]

Sorry for being a little tardy in replying but I wanted to sit down and work my way through the Davis protocol. It seems everything is being followed, I do not know if the lack of a final acknowledgement is critical or not but what I would say is that the Davis driver works with real Davis hardware without issue.

One thing I did wonder, the DMPAFT command downloads all archive records after a given timestamp. Archive records are downloaded a page at a time and a page contains 5 archive records. When we look at the earlier log extracts you have provided there seems to be groups of 5 archive records being processed (1 page?) but of course all that are already in the archive are rejected, for example:

Mar 18 22:45:18 WeatherDuinoPi weewx[12220]: manager: Unable to add record 2019-03-18 22:43:00 CET (1552945380) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added record 2019-03-18 22:44:00 CET (1552945440) to database 'weewx.sdb'
Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added record 2019-03-18 22:44:00 CET (1552945440) to daily summary in 'weewx.sdb'
Mar 18 22:45:22 WeatherDuinoPi weewx[12220]: manager: Unable to add record 2019-03-18 22:40:00 CET (1552945200) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 18 22:45:24 WeatherDuinoPi weewx[12220]: manager: Unable to add record 2019-03-18 22:41:00 CET (1552945260) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
Mar 18 22:45:25 WeatherDuinoPi weewx[12220]: manager: Unable to add record 2019-03-18 22:42:00 CET (1552945320) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime

I did not see anywhere where you ran WeeWX directly and provided a few minutes of console output. Maybe that may shed more light on the problem. Failing that you may need to put some logging in the vantage driver to see exactly what the driver is reading and then sending to WeeWX.

Gary

[engolling]

Hi Gary,

again thanks for your extended reply.

Meanwhile I added a lot of debugging code in the vantage driver and the WeatherDuino code and was able to find out the reason what happened.

Their is a buffer on the flash chip which is written before storing the data in the flash page. And this buffer always had the outdated data of the page before in the last entries.

This lead to this behaviour that the actual written page contained wrong data, but the page before and the page after was pretty fine...

It took me quite some time to find the system behind this error.

I'm in touch with the developper of the WeatherDuino and this issue will be fixed in the next firmware release so there will be full compatibility with WeeWx.

Now that erverything is working I have a last question - do you, or anybody else, think it is possible to draw a bar graph having the rain bars of two sensors right beside?

I did not find anything in the manual and if I define the diagramm like in the line / dot graphs the two bars overlay, so that the bar of the second rain sensor is not visible, if it is a bit smaller.

Thank you for all your replies.

Regards,

engolling

[gjr80]

Good that you have tracked down the issue. Running hardware record generation on a Davis station has it advantages, though maybe not so advantageous on emulated hardware.

Regards plotting rain data from two sensors. The current WeeWX plot engine will certainly allow you to plot two obs in a bar graph plot, though I expect the outcome will be one bar plotted over the top of the other. Not really what you are seeking. There is no option to plot the bars adjacent to each other. It may be possible to modify the plot engine though I do not expect it would be a simple modification, the plot engine is fairly rudimentary. You may need to use an external charting package which should easily do what you want, though this too will not be a turn key solution as you will need to generate the necessary data for the package as well as integrate it into your web page(s).

Gary

[engolling]

Hello together,

it's me again with hopefully a last question...

As you know I'm augmenting some extra data from my service to the WeeWx archive records. But if archive records are loaded of the stations logger this data (which is in the future) is also augmented.

Is it possible to read the timestamp of the archive record beeing saved at the moment and then decide if data should be augmented or not?

Im thinking of a code like this:

if dateTime_AugmentedData - event.record[dateTime] < 300:
   --> augment_data

[Thomas Keffer]

Yes. That's certainly possible. 

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/94baea9a-ed38-4e52-9ea1-6ccdb5689960%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[engolling]

Hello,

me again :).

Thank you for your last answer Thomas. Works fine.

Now I have a new "thing" I do not understand.

As discussed above I'm augmenting different data - also rain data to the archive records as described in the customization guide.

I also augment temperature signals and sometimes they are converted to metric variables and sometimes not. The problem is the signal is already in °C and should be in °C, but if it is converted it looks like this:

The database is mertric. If I get for example a temperature of 22.61°C which should be imported into the database directly and it is most time. But sometimes I get -5.21°C (which is the result of the Farenheit Celsius conversion of 22.61°C).

My Raspberry Pi has a bad WIFI internet connection and FTP often fails and it seems to be related to this.

Could this be a reason and do you know how to handle the conversion?

Regards,

engolling

<span style="font-family:arial,san

[gjr80]

Hi,

Perhaps you could give us some more details as to how the extra temperature data is being obtained/provided/added to WeeWX, happy to see code/data formats etc - code usually gives an unambiguous picture, descriptions are often open to interpretation. I am having a little difficulty understanding how the numeric data is always being received but not the units, but perhaps that will become clear once you provide some more detail.

Gary

This would mean that the vantage driver polls with a <font s

[engolling]

Hello,

of course I can give more details :-)

So as you know I'm augmenting some extra data to my archive records which I get from emulated davis loop packaged.

My file whichs contains always the latest data looks like this:

https://github.com/menachers/WeatherDuino/blob/master/WeeWx_Plugin/WeeWx_Exp.txt

So there is written a signal name, a unit group and the actual data as csv.

The data is then augmented with the following script:

https://github.com/menachers/WeatherDuino/blob/master/WeeWx_Plugin/WeeWx_WeatherDuino_Logger_plugin.py

called as data service in the weewx.conf

record_generation is set to software, target_unit is set to metric.

The augmented data is in °C and most of the time it is saved to the database without being converted but sometimes a few datasets are double converted.

When record_generation is set to hardware the data is always converted, so I'm exporting the data in US units and weewx converts it back. This is no problem either.

But it is hard to handle when weewx jumps between no conversion and conversion.

Hopefully it is more clear now what I mean.

Regards,

engolling

[engolling]

Hallo,

think I tracked the issue down.

First it starts with record generation software.

So each time a archive dataset is retreivedn form the hardware logger my data is augmented and then it runs apparently through the unit converter, converting it to metric units (in case of my augmented data coming in metric this is bad).

This happens whenn WeeWx was off for some time.

Each time an archive paket is gained from the LOOP data it was seemingly already converted my data is augmented and it is saved correctly to the database.

So my question is simplified. Is it possible to determine if a archive record is archived of hardware logger data or LOOP data or can I influence the order of augmenting and converting?

Best regards,

engolling

[Andrew Milner]

The script you posted SHOULD only be augmenting  REC packets, and not be affected by any LOOP packets.

Maybe you should set record generation to either use hardware or use software and not prefer hardware or prefer software.  Your augmented data should be use software I would have thought

Mar 26 19:06:23 WeatherDuinoPi weewx[17760<span style=

[engolling]

Hello Andrew,

indeed it actually is only augmenting.

But in case the archive data is gained from a DMPAFT packet it seems to be converted complete after augmenting. If the archive data is gained from a LOOP packet it converted before.

So the following things would probably solve the issue:

 - Only using record generation hardare

 - Database in US units

My intention is that my plugin works with any configuration, so I would like to work out an solution for this. So my idea would be to find an indicator of the archive data is gained from an LOOP packet or a DMPAFT packet - so conversion could be done if necessary.

Regards,

engolling

Mar<span style="color:rgb(0,

[gjr80]

The number one rule when augmenting archive records or loop packets with data is that you must check the unit system used in the record or packet (ie check the usUnits field) you are adding data to and make sure that any data you do add is in the same units as used by that unit system. If you fail to do this you may be fortunate to have something that works, but it will be good luck rather than good planning and the resulting system will be fragile with any future changes likely to result in unit conversion inconsistencies. For example, say you have a service that is adding a rain value to an archive record and your service produces a variable rainfall that contains the rain data to be added in mm. The archive record field you wish to create is 'our_new_field'. You might use some code in your service like the following to actually augment the archive record (untested):

import weewx

if event.record['usUnits'] == weewx.US:
    # archive record uses US unit system so we need rainfall in inches
    event.record['our_new_field'] = rainfall/25.4 if rainfall is not None else None
elif event.record['usUnits'] == weewx.METRIC:
    # archive record uses the Metric unit system so we need rainfall in cm
    event.record['our_new_field'] = rainfall/10.0 if rainfall is not None else None
elif event.record['usUnits'] == weewx.METRICWX:
    # archive record uses the MetricWX unit system so we need rainfall in mm - leave as is
    event.record['our_new_field'] = rainfall if rainfall is not None else None

or if you want to be a bit more sophisticated you can use the internal WeeWX unit conversion routines to do the unit conversion for you (much more compact code and useful for more complex/less well known conversions) (again untested):

import weewx.units

# express our rainfall value as a ValueTuple
rainfall_vt = weewx.units.ValueTuple(rainfall, 'mm', 'group_rain')
# now augment the archive record with the appropriately converted value
event.record['our_new_field'] = weewx.units.convertStd(rainfall_vt, event.record['usUnits'])

The above examples augment an archive record but they apply equally to loop packets. In the cases above we knew the incoming data was always in mm, if that is not the case and you need to handle the incoming data being in any one of a number of units then you will need some extra code to identify the incoming units and ensure the corect conversions are applied.

Gary

[engolling]

Hi Gary,

sometimes I can't see the forest because of all of those trees anymore... :)

Thank you for your great answer again.

So I started off with the noob variant...

https://github.com/menachers/WeatherDuino/tree/master/WeeWx_Plugin

I will change it to the sophisticated procedure you proposed.

As I got you right

# express our rainfall value as a ValueTuple
rainfall_vt = weewx.units.ValueTuple(rainfall, 'mm', 'group_rain')

I have to generate a tuple with the variable holding the actual value, followed by the unit of the signal as it can be found in the units.py dict and ending with the unit group which it belongs to.

best regards,

engolling

[gjr80]

On Tuesday, 28 May 2019 06:52:22 UTC+10, engolling wrote:

So I started off with the noob variant...

https://github.com/menachers/WeatherDuino/tree/master/WeeWx_Plugin

That looks like it will work, but be aware that if the record you are augmenting is in anything other than US customary units no conversion will be applied (this may be fine given your current setup but who knows how it may change in the future). Nothing to worry about if you are going to rewrite the code anyway.

 

I will change it to the sophisticated procedure you proposed.

As I got you right

# express our rainfall value as a ValueTuple
rainfall_vt = weewx.units.ValueTuple(rainfall, 'mm', 'group_rain')

I have to generate a tuple with the variable holding the actual value, followed by the unit of the signal as it can be found in the units.py dict and ending with the unit group which it belongs to.

Correct. The ValueTuple is the basis of the WeeWX system for unit conversion; it brings together the value, the units used and the unit group to which it belongs. When WeeWX needs to convert the value to some other units or to the units used in a particular unit system (US, Metric or MetricWX) the ValueTuple has the core information used to determine how to do the conversion. You might want to look at the class ValueTuple in bin/weewx/units.py. the other good thing about ValueTuple based conversion is that it will handle the case where the data value is None - note how in the simple approach I outlined we had to take care of the case where the data value may be None.

Gary

[engolling]

Hello to all,

I got another question. The import of additional rain works fine now.

So if a add the following tag to my template:

$day.Rain_RG11.sum

I get the correct value in my preselected unit. (Database is metric and my default unit for group_rain is "mm")

I'm also generating two graphs with the following code:

        [[[dayrain]]]
            # Make sure the y-axis increment is at least 0.02 for the rain plot
            yscale = None, None, 0.02
            plot_type = bar
            [[[[rain]]]]
                aggregate_type = sum
                aggregate_interval = 3600
                label = Niederschlag (Stundenwerte)
        [[[dayrain_RG11]]]
            # Make sure the y-axis increment is at least 0.02 for the rain plot
            yscale = None, None, 0.02
            plot_type = bar
            [[[[rain_RG11]]]]
                aggregate_type = sum
                aggregate_interval = 3600
                label = Niederschlag (Stundenwerte)

Here I get the following images (do not look at the actual bar values - they are not representative and result due to testing):

My self added rain source generates correct values but it does not display the unit and the values which are plotted are seeming to be in the native unit [cm] as it is stored in the database.

Does anybody has an idea what could be the matter here? Tags are working fine, but not the image generation.

Thanks in advance for your answers.

Regards,

engolling

[engolling]

Hello,

I checked my image generation settings again and I also tried to check the imagegenerator.py file but I did not find any reason why the conversion to "mm" is not applied or at least the native database unit "cm" is not applies.

I would be glad, if someone can give me a hint where this could come from and what is the best way to debug.

Regards,

engolling

[engolling]

Hello,

I have finally found the problem.

Seemingly the image generator is not case sensitive with the signal names.

The input signal was named 'rain_RG11'.

The unit group system IS case sensitive because in my plugin I linked

weewx.units.obs_group_dict['Rain_RG11'] = 'group_rain'

And so I got a graph but with no units on the y-axis and no conversion.

Regards,

engolling