--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/58b42f79-b051-433c-a2ce-670300f75e9bn%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/9760a446-9e20-4f5e-bb03-98446729383an%40googlegroups.com.
I didn't much like that the UDP-based driver doesn't fetch data if it's down.
The problem for me is that you cannot fetch historical data, for example if the system has been down for a period. But that's where the REST API comes in, at least that's the plan :)
> * if you're asking if there's some kind of hybrid UDP 'and/or' REST weewx driver...
I'm not asking that, but the latter is what I'm trying to create. Well, it's working though very much a WIP. However I have wondered as well if I should drop the UDP altogether and end up with a REST-only-driver. I haven't really decided on that yet. Maybe I'll make it configurable in an all-in-one driver.
On Wednesday, January 20, 2021 at 11:10:35 PM UTC-8 Jan-Jaap van der Geer wrote:> * if you're asking if there's some kind of hybrid UDP 'and/or' REST weewx driver...I'm not asking that, but the latter is what I'm trying to create. Well, it's working though very much a WIP. However I have wondered as well if I should drop the UDP altogether and end up with a REST-only-driver. I haven't really decided on that yet. Maybe I'll make it configurable in an all-in-one driver.Nobody's going to turn down having more drivers available :-)
I guess I could see a few options:
- a REST-only driver
- a driver that is configurable UDP 'or' REST
- a driver that is smart enough to use REST to fill in UDP gaps
- or maybe even something like wunderfixer as a standalone utility to fill in weewx gaps by comparing the db with the WF server data
Power loss risk is easier. Buy a UPS. Put your Hub on it.
Old posts on the WF Forums from their engineering said that the old Air+Sky sensors could store 90-120 minutes if they can't reach their Hub. The Hub stores 8 days of data if it can't reach WF servers. So if you're concerned re: short outages then I'd UPS the Hub and ideally the weewx box if you could. The new Tempest stations are similar enough those numbers are probably reasonable for the current gear.
Personally I kinda like the idea of (1) and (4) at least initially, and maybe (2) as a stretch goal.
I've just installed your driver. Whats the status now? Right now it neither seems to backfill data nor store current data from UDP into the database.
Other Question:What purpose does this have and why is it hardcoded to 60? Shouldn't it read and return@propertydef archive_interval(self):return 60[StdArchive]# If the station hardware supports data logging then the archive interval# will be downloaded from the station. Otherwise, specify it (in seconds).archive_interval = 300
from weewx.conf?
If you implement function genArchiveRecords(), then you should also implement archive_interval as either an attribute, or as a property function. It should return the archive interval in seconds.
To clarify, the archive interval coming from the driver or weewx.conf is quite acceptable.If a driver has the ability to emit archive records then it MAY also control the archive period (by the same token it may not). Consequently, if you are using driver emitted archive records (ie hardware record generation) WeeWX will look to the driver for the archive interval during startup. If the driver provides the archive interval then WeeWX will use that instead of the weewx.conf archive_interval setting. If the driver does not provide the archive interval WeeWX will fallback to the weewx.conf value.
Irrespective of the archive interval, what happens at the end of an archive interval depends on whether hardware or software record generation is being used. If hardware record generation is being used then WeeWX will ask the driver for for all archive records since the last. If one or more archive records are provided WeeWX accepts and uses them, if the driver cannot provide archive records (for example it has no genArchiveRecords() method) then WeeWX falls back to software record generation. If software record generation is selected then at the end of the archive interval WeeWX (not the driver) generates an archive records based on accumulated loop data seen by WeeWX over the archive interval. After this processing by WeeWX for both hardware and software record generation is essentially the same.
One other point. I notice some mention of the daily summaries (archive_day_xxxxxx). I would be very careful dealing with the daily summaries. I am not sure a driver should be delving into the daily summaries. The driver should really have no need to read the daily summaries and it certainly should not be writing to them. The driver should just emit archive records (be it current or historical) and loop packets and let the rest of the WeeWX machinery take care of dealing with (day based) aggregates in the daily summaries.
On Tuesday, 9 February 2021 at 10:23:40 UTC+1 gjr80 wrote:To clarify, the archive interval coming from the driver or weewx.conf is quite acceptable.If a driver has the ability to emit archive records then it MAY also control the archive period (by the same token it may not). Consequently, if you are using driver emitted archive records (ie hardware record generation) WeeWX will look to the driver for the archive interval during startup. If the driver provides the archive interval then WeeWX will use that instead of the weewx.conf archive_interval setting. If the driver does not provide the archive interval WeeWX will fallback to the weewx.conf value.Hm, OK, interesting. That's not how I read the documentation. I interpreted it to say that if you implement genArchiveRecords() you should (interpreted as 'must') provide the archive_interval. You seem to imply that that is not necessarily the correct interpretation. I will investigate this.
Thanks Gary.So if the driver "wants" to control the archive interval, it is overriden? For my personal reality, 1 Minute is too short.
> If one or more archive records are provided WeeWX accepts and uses themHow does this work in detail? does weewx take care of min/max if there are more hardware archive records than weewx archive records?
That being said there is no requirement for that to be so, hence why you see words like ‘if’ and not ‘must’.
Very simply, when your driver is emitting loop packets your time resolution is the time between loop packet timestamps, once your driver is emitting archive records your time resolution decreases and is the time between archive record timestamps. Similar but slightly different with actual values, loop packets give you the finest granularity of the observation values, once you move to archive records you get the average of the loop values (well actually you get whatever function is used to obtain an archive record value from a group of loop packet values, this is usually the average but it could be the sum or the last or the first value, it depends).
Thank you :)so when I said:"For backfilling missing data would use an approach like: If there is data available in weatherflow REST service, an if it is available on a time base which is more frequent than archive_interval, I would try to backfill like this: for every top of an weewx archive_interval that is missing, I'd get all data available for this interval, calculate averages (take care with vector data windSpeed/windDir!), check if minimum/maximum values of each REST entry, if it is lower/higher than the min/max value in archive_day_[archiveColumn], handle it."I'd now say:For a weatherflow driver I wouldn't set an archive_interval in the driver itself. For backfilling, get the all records from the REST within the current archive_interval to backfill, create a single record for that interval, with the values being the average values of all the REST records (with the right treatment for vector data). Losing precision of min/max values and their timestamps is acceptable (at least for reasonable short archive_interval settings)
That's the good thing about WeeWX, everyone can contribute. If you believe the documentation can be improved propose an amendment, preferably via a pull request, though I expect for something simple Tom would consider a request with suggested text in a post.
In brief my observations with your driver:
- udp-packets are being picked up (and published with mqtt in my case, because I've installed the extension)
- missing database values are NOT being backfilled
- current archive records from loop data are NOT being generate
I'll give i a try the next few days. API-Token was already configured, it was probably something with the sensor map I indeed have in my config.
How is "For most configurations the driver will figure it out itself" to be understood?
Well, if you have one Tempest, or one Air or one Skye or one set of Sky & Air, it should Just Work (tm). However if you have multiple units or unknown units then you would need to do things to the sensor map.
I'm confident I'll get along with this. Looking forward to get this "production ready": https://www.kainzbauer.net/weather/Rif-Tempest/live.html
seems to work!
I made two local changes:
1.) I removed the@propertydef archive_interval(self):return 60And changed the logging of "Import from UDP" to debug level. The driver works now for me, the last thing on the wishlist would bei to let genStartupRecords only generate one entry per archive_interval. I might do that myself if I find some time.
Good work! And thank you for the driver enhancements!
> You're welcome. Did you find out why the REST part didn't work initially? From your earlier explanation, I don't really see why it wouldn't work for you?It was devices = ST-000xxxxx
> A bit surprised you didn't take up an issue that I had expected you'd mention: The skin you use seems to actively use the wind packages that come much more often in the UDP messages, but which I have removed,> at least when it comes to the default sensor maps. And if you want to use these nonetheless, by using an explicit sensor map, you will get problems when using the REST interface as these are not really compatible as it is.> I thought we didn't loose too much functionality by doing that in the driver, but I now understand that there actually is a use case for that stuff as well.Well, here is why:[[sensor_map]]outTemp = air_temperature.ST-000xxxxx.obs_stoutHumidity = relative_humidity.ST-000xxxxx.obs_stpressure = station_pressure.ST-000xxxxx.obs_stlightning_strike_count = lightning_strike_count.ST-000xxxxx.obs_stlightning_distance = lightning_strike_avg_distance.ST-000xxxxx .obs_stsupplyVoltage = battery.ST-000xxxxx.obs_stwindSpeed = wind_avg.ST-000xxxxx.obs_stwindDir = wind_direction.ST-000xxxxx.obs_stcurrentWindSpeed = wind_speed.ST-000xxxxx.rapid_windcurrentWindDir = wind_direction.ST-000xxxxx.rapid_windwindGust = wind_gust.ST-000xxxxx.obs_stluminosity = illuminance.ST-000xxxxx.obs_stUV = uv.ST-000xxxxx.obs_strain = rain_accumulated.ST-000xxxxx.obs_stradiation = solar_radiation.ST-000xxxxx.obs_stI map the rapid_wind data to nonexistent (database) keys ("currentWindSpeed", "currentWindDir"). These values get into the loop and are published to the mqtt server to feed the live gauges and only the live gauges of the skin I use. You can see what happens, if you debug the javascript of my skin and take a closer look to the "jPayload"-variable on "site.js". Every three seconds there will be a packet containing "currentWindSpeed" and "currentWindDir", updating the "Wind", "Winböen" (gust) and "Windrichtung" (direction) - gauges. Yes, all three of them, even without having "windGust" in the package. If the currentWindSpeed is higher than the gust-gauges value, the new gust value is currentWindSpeed. It's reset with the once-a-minute package cointaining "windGust_mps", which is mapped to windGust in the above stanza.The other values are stored in the database and they also feed the live charts. There is no problem at all, I use all values in their special way, I am fully compatible. There is absolutely no benefit in storing rapid_wind data when the device calculates the avg values on a one-minute-base, and my interval is on a 5-minute-base.
> Hm, why would that not work? I just tried that but it worked for me. (I suppose you don't mean x but the actual number, right?)I didn't have this value in my config at all
> If weewx does the accumulation you also get a windGustDirThat's a good point.By now, you are using the mapping, if specified, for both I guess?
> I'm thinking of having some kind of suffix ('.udp' and '.rest' for example)[[sensor_map]]#this is the sensor mapping used for UDP and REST...windSpeed = wind_avg.ST-000xxxxx.rapid_windwindDir = wind_direction.ST-000xxxxx.rapid_windwindGust = wind_gust.ST-000xxxxx.obs_st...[[[REST]]] #if configured, use this mapping when getting data from REST, values are inherited from abovewindSpeed = wind_avg.ST-000xxxxx.obs_st #overwrites above mappingwindDir = wind_direction.ST-000xxxxx.obs_st #overwrites above mappingOr like this?
I still have one thought: let's assume your WIFI has an outage. In this case, a friend told me, the station is holding back all the data. When the WIFI is back, it burst all the missing UDP values, the timestamps are all there. How to deal with that? Weewx is still online, because, in my example it's on a wired device. What does the driver do with that data?
And then a similar case: a power outage: the power comes back, WIFI comes back, the station burst all the values. Weewx comes back just right after that, tries to fetch missing data, which not yet is available via REST. Weewx starts up, doesn't get any values from REST, it never tries to get them again.
In both cases I know what I would do, because I save my weewx database every hour a day. overwriting the backup after 24 hours, but keep the 0:00 copy for a month before overwriting. So I would go back to a database backup before the outage and start weewx when all values are available via REST.
@Jan-Jaap: a friend of mine made a couple of things mentioned above and filed a pull request.