Polling interval confusion

[Jan-Jaap van der Geer]

I'm new to python and new to weewx drivers, but I've had a weewx installation for several years. I recently acquired a WeatherFlow Tempest and I created a new weewx installation with the based on the https://github.com/captain-coredump/weatherflow-udp driver. I later switched to the Tom Keffer fork.

I didn't much like that the UDP-based driver doesn't fetch data if it's down. Since there's a REST interface I thought I'd implement genArchiveRecords() based on the REST API. This seems to work.

The REST interface has a record for every minute (if everything is working). So I set the archive_interval to 1. However the old installation accumulated for 300 seconds and since that works well, I don't see a reason to change that. So archive_interval is set to 300. I've tried record_generation to 'software' but when fetching data from the REST API I see it records every record, i.e. every minute instead of a record for every 5 minutes.

Do I need to do the accumulation myself in the driver? Or is this something in the settings? Or is this a scenario that is not supported? I'm confused! :)

Regards,

Jan-Jaap

[Tom Keffer]

I'm confused by what you're doing.

So, you are using UDP data for the LOOP data, but the REST data for the archive data? I suppose that could work, but you're going to have to decide whether to honor the 1 minute archive interval of the REST data, or the 5 minutes you hope for. 

If you really want to do the latter, then as you say, you will have to accumulate the 5 records and aggregate them into a single record somehow. This record is what would get emitted by genArchiveRecords().

Frankly, I gave up on my Tempest and sent it back last August. The UDP interface was so bad, that I couldn't get it to work reliably. 

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/58b42f79-b051-433c-a2ce-670300f75e9bn%40googlegroups.com.

[Jan-Jaap van der Geer]

Yes, that's the idea. Use REST for archive data and UDP for the LOOP data. OK, so I understand that this is not a supported scenario. I'll have to decide if I want to bother doing something with that or not. The wind vector graph looks rather different for  1 minute and 5 minute intervals :) Thanks for the info.

What is so bad about the UDP interface of the Tempest? I don't have much experience with either weather stations or UDP interfaces for them. It seems to do the job. The problem for me is that you cannot fetch historical data, for example if the system has been down for a period. But that's where the REST API comes in, at least that's the plan :)

[Tom Keffer]

Here's the discussion: https://community.weatherflow.com/t/backdated-udp-obs-st-messages/7873

Maybe they've made some changes since then and my comments are no longer relevant. I don't know.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/9760a446-9e20-4f5e-bb03-98446729383an%40googlegroups.com.

[Vince Skahan]

On Wednesday, January 20, 2021 at 2:13:40 PM UTC-8 Jan-Jaap van der Geer wrote:

I didn't much like that the UDP-based driver doesn't fetch data if it's down.  

 

The problem for me is that you cannot fetch historical data, for example if the system has been down for a period. But that's where the REST API comes in, at least that's the plan :)

 If 'what' is down ?   The weewx box ?  The WF servers ?   The network in-between ?   The Hub ?

The weewx UDP driver is designed/intended to work completely LAN-only with no need for the WF servers.  FWIW, I found the driver 'very' stable in over 24 months of running it.

But it's a little unclear which box you think will be occasionally down....

• if you're asking if there's a weewx REST API driver for WeatherFlow gear that hits the WF servers for data, the answer currently is noi
• f you're asking if there's some kind of hybrid UDP 'and/or' REST weewx driver, then the answer is 'really' no, as that would be even 'more' complicated.

If you wanted to write a weewx driver that used 'only' their REST interface and that didn't listen for UDP at all, I'd guess there would be some folks who would be interested in that.  That would let folks run a small inexpensive Internet-only weewx site that just hits the WF servers for the data and that requires nothing weewx on the home LAN.   That might be pretty cool to be able to do.

[Jan-Jaap van der Geer]

On Thursday, 21 January 2021, 01:30:19 CET, Vince Skahan wrote:

> If 'what' is down ?   The weewx box ?  The WF servers ?   The network in-between ?   The Hub ?

The weewx-box. That's under my control and I suppose I won't manage to have anywhere near the up time any of the others. And it should be possible to use the WF servers as a backup then. If the WF servers are down, there's really no difference from today's situation. If the hub is down I suppose absolutely nothing is working anyway.

> The weewx UDP driver is designed/intended to work completely LAN-only with no need for the WF

> servers.  FWIW, I found the driver 'very' stable in over 24 months of running it.

Yes, that's my experience as well. But the box it runs on needs updates and things happen :)

Electricity may go. In that case my hub has a battery backup, so things will eventually go to the WF servers. But the UDP-only driver would get a gap.

> * if you're asking if there's a weewx REST API driver for WeatherFlow gear that hits the WF servers

> for data, the answer currently is noi

> * f you're asking if there's some kind of hybrid UDP 'and/or' REST weewx driver, then the answer is

> 'really' no, as that would be even 'more' complicated.

I'm not asking that, but the latter is what I'm trying to create. Well, it's working though very much a WIP. However I have wondered as well if I should drop the UDP altogether and end up with a REST-only-driver. I haven't really decided on that yet. Maybe I'll make it configurable in an all-in-one driver.

Cheers,

Jan-Jaap

[Jan-Jaap van der Geer]

Interesting. I haven't noticed this, but I probably have not that much traffic on the frequencies that WF is using... I'll investigate this. How does weewx cope with reporting out-of-order times for its LOOP data?

[Vince Skahan]

On Wednesday, January 20, 2021 at 11:10:35 PM UTC-8 Jan-Jaap van der Geer wrote:

> * if you're asking if there's some kind of hybrid UDP 'and/or' REST weewx driver...

I'm not asking that, but the latter is what I'm trying to create. Well, it's working though very much a WIP. However I have wondered as well if I should drop the UDP altogether and end up with a REST-only-driver. I haven't really decided on that yet. Maybe I'll make it configurable in an all-in-one driver.

Nobody's going to turn down having more drivers available :-)

I guess I could see a few options:

1. a REST-only driver
   
2. a driver that is configurable UDP 'or' REST
   
3. a driver that is smart enough to use REST to fill in UDP gaps
   
4. or maybe even something like wunderfixer as a standalone utility to fill in weewx gaps by comparing the db with the WF server data

Power loss risk is easier.  Buy a UPS.  Put your Hub on it.

Old posts on the WF Forums from their engineering said that the old Air+Sky sensors could store 90-120 minutes if they can't reach their Hub.  The Hub stores 8 days of data if it can't reach WF servers.   So if you're concerned re: short outages then I'd UPS the Hub and ideally the weewx box if you could.    The new Tempest stations are similar enough those numbers are probably reasonable for the current gear.

Personally I kinda like the idea of (1) and (4) at least initially, and maybe (2) as a stretch goal.

[Jan-Jaap van der Geer]

On Thursday, January 21, 2021 at 6:53:11 PM UTC+1 vince...@gmail.com wrote:

On Wednesday, January 20, 2021 at 11:10:35 PM UTC-8 Jan-Jaap van der Geer wrote:

> * if you're asking if there's some kind of hybrid UDP 'and/or' REST weewx driver...

I'm not asking that, but the latter is what I'm trying to create. Well, it's working though very much a WIP. However I have wondered as well if I should drop the UDP altogether and end up with a REST-only-driver. I haven't really decided on that yet. Maybe I'll make it configurable in an all-in-one driver.

Nobody's going to turn down having more drivers available :-)

No? :) 

I guess I could see a few options:

1. a REST-only driver
   
2. a driver that is configurable UDP 'or' REST
   
3. a driver that is smart enough to use REST to fill in UDP gaps
   
4. or maybe even something like wunderfixer as a standalone utility to fill in weewx gaps by comparing the db with the WF server data

Never looked into wunderfixer but heard the name, sounds interesting. I suppose it isn't something general that works with the current drivers?

Anyway, the goal is nr 3 although it will only look at the gaps since the last measurement, so not really gaps.

 

Power loss risk is easier.  Buy a UPS.  Put your Hub on it.

My hub is on an old 10.000 mAh battery. No idea how long of a power loss it would survive (but then that's not really a big issue usually here anyway). Hm, did I introduce a fire risc?

Old posts on the WF Forums from their engineering said that the old Air+Sky sensors could store 90-120 minutes if they can't reach their Hub.  The Hub stores 8 days of data if it can't reach WF servers.   So if you're concerned re: short outages then I'd UPS the Hub and ideally the weewx box if you could.    The new Tempest stations are similar enough those numbers are probably reasonable for the current gear.

Ah, that's a lot more than I thought. I thought I'd seen something like 1h for the hub, and I was assuming nothing for the sensor. Sounds good! 

Personally I kinda like the idea of (1) and (4) at least initially, and maybe (2) as a stretch goal.

Well, aiming for (3) for the time being... :-P

Cheers,
Jan-Jaap

[michael.k...@gmx.at]

Hi Jan-Jaap,

I've just installed your driver. Whats the status now? Right now it neither seems to backfill data nor store current data from UDP into the database.

Other Question:

    @property

    def archive_interval(self):

        return 60

What purpose does this have and why is it hardcoded to 60? Shouldn't it read and return 

[StdArchive]

    

    # If the station hardware supports data logging then the archive interval

    # will be downloaded from the station. Otherwise, specify it (in seconds).

    archive_interval = 300

from weewx.conf?

[Jan-Jaap van der Geer]

On Sunday, February 7, 2021 at 12:31:33 AM UTC+1 michael.k...@gmx.at wrote:

I've just installed your driver. Whats the status now? Right now it neither seems to backfill data nor store current data from UDP into the database.

Yes, I've recently noticed that UDP data is not getting filled into the database. I'm not sure why, as it's virtually using the same methods as the backfill (which BTW is working fine here). It took some time before I noticed the UDP not working as the REST data fills it up anyway. But I'm not sure why it's not working. I've been looking at it today, but it seems that somehow the fact that I have the genArchiveRecords() method causes the UDP/LOOP packages to be ignored by weewx. As far as I can see I'm still returning them to weewx and they seem fine to me, but it seems weewx is ignoring them.

 

Other Question:

    @property

    def archive_interval(self):

        return 60

What purpose does this have and why is it hardcoded to 60? Shouldn't it read and return 

[StdArchive]

    

    # If the station hardware supports data logging then the archive interval

    # will be downloaded from the station. Otherwise, specify it (in seconds).

    archive_interval = 300

 

from weewx.conf?

In the documentation at http://weewx.com/docs/customizing.htm#genArchiveRecords() it says:

archive_interval

If you implement function genArchiveRecords(), then you should also implement archive_interval as either an attribute, or as a property function. It should return the archive interval in seconds.

This too me means that as the WeatherFlow REST service has a record for every minute, the archive_interval should be set to 60 seconds.

Cheers,

Jan-Jaap

[michael.k...@gmx.at]

Hi Jan-Jaap. 

In brief my observations with your driver:

• udp-packets are being picked up (and published with mqtt in my case, because I've installed the extension)
• missing database values are NOT being backfilled
• current archive records from loop data are NOT being generated

For the archive interval:

I don't know exactly how this all works, hopefully the "staff" here can clarify, but: there must be either something wrong with setting the archive interval in your driver or how this value is being used. Leaving the hard coded 60 seconds in your code this leads weewx to generate reports every 60 seconds, which, as far as I understand how things are intended to work, overrides the archive_interval in weewx.conf, which I do not consider as desired behavior. The sentence in the documentation also starts with an "if", so maybe this is only necessary under certain circumstances.

For backfilling missing data would use an approach like: If there is data available in weatherflow REST service, an if it is available on a time base which is more frequent than archive_interval, I would try to backfill like this: for every top of an weewx archive_interval that is missing, I'd get all data available for this interval, calculate averages (take care with vector data windSpeed/windDir!), check if minimum/maximum values of each REST entry, if it is lower/higher than the min/max value in archive_day_[archiveColumn], handle it.

[gjr80]

To clarify, the archive interval coming from the driver or weewx.conf is quite acceptable.

If a driver has the ability to emit archive records then it MAY also control the archive period (by the same token it may not). Consequently, if you are using driver emitted archive records (ie hardware record generation) WeeWX will look to the driver for the archive interval during startup. If the driver provides the archive interval then WeeWX will use that instead of the weewx.conf archive_interval setting. If the driver does not provide the archive interval WeeWX will fallback to the weewx.conf value. 

Irrespective of the archive interval, what happens at the end of an archive interval depends on whether hardware or software record generation is being used. If hardware record generation is being used then WeeWX will ask the driver for for all archive records since the last. If one or more archive records are provided WeeWX accepts and uses them, if the driver cannot provide archive records (for example it has no genArchiveRecords() method) then WeeWX falls back to software record generation. If software record generation is selected then at the end of the archive interval WeeWX (not the driver) generates an archive records based on accumulated loop data seen by WeeWX over the archive interval. After this processing by WeeWX for  both hardware and software record generation is essentially the same.

One other point. I notice some mention of the daily summaries (archive_day_xxxxxx). I would be very careful dealing with the daily summaries. I am not sure a driver should be delving into the daily summaries. The driver should really have no need to read the daily summaries and it certainly should not be writing to them. The driver should just emit archive records (be it current or historical) and loop packets and let the rest of the WeeWX machinery take care of dealing with (day based) aggregates in the daily summaries. 

Gary

[Jan-Jaap van der Geer]

On Tuesday, 9 February 2021 at 10:23:40 UTC+1 gjr80 wrote:

To clarify, the archive interval coming from the driver or weewx.conf is quite acceptable.

If a driver has the ability to emit archive records then it MAY also control the archive period (by the same token it may not). Consequently, if you are using driver emitted archive records (ie hardware record generation) WeeWX will look to the driver for the archive interval during startup. If the driver provides the archive interval then WeeWX will use that instead of the weewx.conf archive_interval setting. If the driver does not provide the archive interval WeeWX will fallback to the weewx.conf value. 

Hm, OK, interesting. That's not how I read the documentation. I interpreted it to say that if you implement genArchiveRecords() you should (interpreted as 'must') provide the archive_interval. You seem to imply that that is not necessarily the correct interpretation. I will investigate this.

Irrespective of the archive interval, what happens at the end of an archive interval depends on whether hardware or software record generation is being used. If hardware record generation is being used then WeeWX will ask the driver for for all archive records since the last. If one or more archive records are provided WeeWX accepts and uses them, if the driver cannot provide archive records (for example it has no genArchiveRecords() method) then WeeWX falls back to software record generation. If software record generation is selected then at the end of the archive interval WeeWX (not the driver) generates an archive records based on accumulated loop data seen by WeeWX over the archive interval. After this processing by WeeWX for  both hardware and software record generation is essentially the same.

Interesting, thanks for the info. 

 

One other point. I notice some mention of the daily summaries (archive_day_xxxxxx). I would be very careful dealing with the daily summaries. I am not sure a driver should be delving into the daily summaries. The driver should really have no need to read the daily summaries and it certainly should not be writing to them. The driver should just emit archive records (be it current or historical) and loop packets and let the rest of the WeeWX machinery take care of dealing with (day based) aggregates in the daily summaries. 

Agreed! I was not planning on anything to this, personally :)

Cheers,

Jan-Jaap

[michael.k...@gmx.at]

Thanks Gary.  

So if the driver "wants" to control the archive interval, it is overriden? For my personal reality, 1 Minute is too short.

>  If one or more archive records are provided WeeWX accepts and uses them 

How does this work in detail? does weewx take care of min/max if there are more hardware archive records than weewx archive records?

[gjr80]

On Tuesday, 9 February 2021 at 19:40:43 UTC+10 Jan-Jaap van der Geer wrote:

On Tuesday, 9 February 2021 at 10:23:40 UTC+1 gjr80 wrote:

To clarify, the archive interval coming from the driver or weewx.conf is quite acceptable.

If a driver has the ability to emit archive records then it MAY also control the archive period (by the same token it may not). Consequently, if you are using driver emitted archive records (ie hardware record generation) WeeWX will look to the driver for the archive interval during startup. If the driver provides the archive interval then WeeWX will use that instead of the weewx.conf archive_interval setting. If the driver does not provide the archive interval WeeWX will fallback to the weewx.conf value. 

Hm, OK, interesting. That's not how I read the documentation. I interpreted it to say that if you implement genArchiveRecords() you should (interpreted as 'must') provide the archive_interval. You seem to imply that that is not necessarily the correct interpretation. I will investigate this.

You can implement genArchiveRecords() and not have the driver provide archive_interval. The different hardware for most drivers where I have seen genArchiveRecords() implemented have some form of logger/memory that stores archive record type data at some hardware defined/set interval, in such cases it makes sense for the driver to override archive interval in weewx.conf. That being said there is no requirement for that to be so, hence why you see words like ‘if’ and not ‘must’.

Gary

[gjr80]

On Tuesday, 9 February 2021 at 19:43:50 UTC+10 michael.k...@gmx.at wrote:

Thanks Gary.  

So if the driver "wants" to control the archive interval, it is overriden? For my personal reality, 1 Minute is too short.

 

Simply put, if WeeWX is using hardware record generation and the driver has an archive_interval property then the driver value is used.

The hardware supported by most drivers I have seen that have an archive_interval property has the ability to store the archive interval in the hardware, the driver reads this value on startup and that is what is provided by the driver via the archive_interval property. Ultimately such hardware gives the user some ability to set this ‘hardware’ archive interval separately, maybe through a front panel setting or using separate software such as the wee_device utility or some hardware specific software. For example, the Davis stations can have the archive interval set via wee_device —set-interval or via front panel buttons.

>  If one or more archive records are provided WeeWX accepts and uses them 

How does this work in detail? does weewx take care of min/max if there are more hardware archive records than weewx archive records?

The WeeWX StdArchive service is responsible for taking loop packets and archive records from the driver and (1) saving archive records to database and (2) updating the daily summary tables as necessary. This is greatly simplified but essentially what happens. Not sure what you mean by “does weewx take care of min/max if there are more hardware archive records than weewx archive records?” If you mean does WeeWX determine if any incoming archive record has (for example) a new daily max for some field and then save that data, then yes that is what StdArchive does.

Gary

[michael.k...@gmx.at]

> If you mean does WeeWX determine if any incoming archive record has (for example) a new daily max for some field and then save that data, then yes that is what StdArchive does.  

So if a hardware sends a loop packet with all the readings in a one-minute-interval, and stores these exact values in its own storage in a separate entry, one entry per minute, the weewx database values are exactly the same, regardless if they were received as loop data as if they were backfilled? Including timestamps and Values for min/max in archive_day_xxxx, which may differ from the min/max value found in all weewx archive records of that particular day, with timestamps between possible timestamps of archve values?

[gjr80]

If your one loop packet was the only loop packet received by WeeWX in that archive interval then the archive record that WeeWX would synthesise from that solitary loop packet would be the same as that loop packet. If as you said that loop packet was stored by hardware and if the driver's genArchiveRecord() method obtains that same data when it generates an archive record, then yes they will be the same. Highs and lows in the daily summaries would be the same but it would not surprise me if some timestamps were different. Loop packets are generally timestamped when the data is read, whereas archive records are typically timestamped at the end of the period they cover. So for an archive interval that covers say 11:05:00am to 11:06:00am, the loop packet may have been created at 11:05:25am and timestamped accordingly. If WeeWX is set to obtain highs and lows from loop packets (ie loop_hilo is True) then if a new max/min is found in that loop packet the relevant timestamp recorded in the daily summary will be the loop packet timestamp. If you now consider the archive record that is read during a backfill, it will be timestamped 11:06:00am, again any new highs or lows will be recorded (so the actual aggregate value will be the same) but in this case the timestamp recorded for the high or low will be the archive record timestamp 11:06:00am. If you happen to have more than one loop packet in the archive period then you may loose some fidelity of the actual observation high/low value as well.

This is the same effect you see if you rebuild your daily summaries with wee_database --rebuild-daily, if you have loop_hilo = True then your daily summary tables will likely have loop packet timestamps for many highs and lows but after you rebuild the daily summaries you loose that fine detail and your high/low timestamps will align with archive record timestamps. If you have multiple loop packets in an archive interval then your high/low values may be one of the values in those loop packets, after rebuilding though you end up with archive values being used to generate the highs and lows which are typically averages of loop values. For example, if you have four loop packets in an archive interval and each has an outTemp value of 8.0, 9.0, 12.0 and 7.0 degrees C your daily summary may record 12.0 as the max. The corresponding archive record outTemp value would (8.0+9.0+12.0+7.0)/4=9.0 so your max might be 9.0.

Gary

[michael.k...@gmx.at]

For the loop packets this is what I already thought to know.

What I wanted to know if my weew archive interval is 5 mins, and the hardware stored at 1min:

Lets assume the archive_interval 11:05:00am to 11:10:00am is to be backfilled. The driver gets the stored archive values from the hardware, where the maxvalue was at 11:07:00am.

Is the maxvalue in the archive_day_xxx table for that day timestamped with 11:07:00am or with 11:10:00am?

[gjr80]

OK, let's work through it. The hardware stored data at one minute intervals, so between 11:05:00am to 11:10:00am the hardware will have five stored records, the driver then creates a single archive record covering 11:05:00am to 11:10:00am, the observation that has the max value at 11:07:00am will appear as the average of the five values stored between 11:05:00am and 11:10:00am (so chances are that will not be the same as the value (the maximum) seen at 11:07:00am) and that is what will be stored as the new daily max in the daily summary (if in fact it is a new maximum). The timestamp of this backfilled record will be 11:10:00am and that is what will be recorded as the max timestamp in the daily summary. So in this case not only will you likely miss the maximum value that occurred at 11:07:00am but you will miss the actual timestamp of the maximum value as well, it will be moved to an archive interval boundary.

Gary

[gjr80]

Very simply, when your driver is emitting loop packets your time resolution is the time between loop packet timestamps, once your driver is emitting archive records your time resolution decreases and is the time between archive record timestamps. Similar but slightly different with actual values, loop packets give you the finest granularity of the observation values, once you move to archive records you get the average of the loop values (well actually you get whatever function is used to obtain an archive record value from a group of loop packet values, this is usually the average but it could be the sum or the last or the first value, it depends).

Gary

[michael.k...@gmx.at]

Thank you :)

so when I said:

"For backfilling missing data would use an approach like: If there is data available in weatherflow REST service, an if it is available on a time base which is more frequent than archive_interval, I would try to backfill like this: for every top of an weewx archive_interval that is missing, I'd get all data available for this interval, calculate averages (take care with vector data windSpeed/windDir!), check if minimum/maximum values of each REST entry, if it is lower/higher than the min/max value in archive_day_[archiveColumn], handle it."

I'd now say:

For a weatherflow driver I wouldn't set an archive_interval in the driver itself. For backfilling, get the all records from the REST within the current archive_interval to backfill, create a single record for that interval, with the values being the average values of all the REST records (with the right treatment for vector data). Losing precision of min/max values and their timestamps is acceptable (at least for reasonable short archive_interval settings)

[Jan-Jaap van der Geer]

On Tuesday, February 9, 2021 at 10:56:34 AM UTC+1 gjr80 wrote:

That being said there is no requirement for that to be so, hence why you see words like ‘if’ and not ‘must’.

The 'if' concerns the if we're implementing the genArchiveRecords(), so that's a clear yes. Other than that it says 'should'. Agreed, that's not a 'must', but it is very close. Maybe that documentation could use a bit more info there assuming you're right that it's totally OK and usual not to implement the archive_interval property.

Cheers,

Jan-Jaap

[gjr80]

Sounds fine, though what function you use to obtain archive record field values from the REST records depends on the observation; most will be ‘average’ but if rain is involved I expect it would be ‘sum’ not ‘average’, also if any status fields are involved they might be ‘last’ rather than ‘average’.

Thinking again regards archive interval. The driver needs to know the archive interval if it is to publish archive records either by backfill or normal generation. When the driver has an archive_interval property this knowledge is inherent, if this is no longer the case the driver needs to pick up the archive interval from the config dict and it needs to be able to handle the user changing the value.

Gary

[Jan-Jaap van der Geer]

On Tuesday, February 9, 2021 at 1:48:18 PM UTC+1 gjr80 wrote:

Very simply, when your driver is emitting loop packets your time resolution is the time between loop packet timestamps, once your driver is emitting archive records your time resolution decreases and is the time between archive record timestamps. Similar but slightly different with actual values, loop packets give you the finest granularity of the observation values, once you move to archive records you get the average of the loop values (well actually you get whatever function is used to obtain an archive record value from a group of loop packet values, this is usually the average but it could be the sum or the last or the first value, it depends).

When is the start of an interval? If the interval chosen would be a silly number, like for example 7, when is the start of it? Is that every multiple of 7 minutes since unixtime 0, I assume?

Cheers,

Jan-Jaap 

[Jan-Jaap van der Geer]

On Tuesday, February 9, 2021 at 2:03:52 PM UTC+1 michael.k...@gmx.at wrote:

Thank you :)

so when I said:

"For backfilling missing data would use an approach like: If there is data available in weatherflow REST service, an if it is available on a time base which is more frequent than archive_interval, I would try to backfill like this: for every top of an weewx archive_interval that is missing, I'd get all data available for this interval, calculate averages (take care with vector data windSpeed/windDir!), check if minimum/maximum values of each REST entry, if it is lower/higher than the min/max value in archive_day_[archiveColumn], handle it."

I'd now say:

For a weatherflow driver I wouldn't set an archive_interval in the driver itself. For backfilling, get the all records from the REST within the current archive_interval to backfill, create a single record for that interval, with the values being the average values of all the REST records (with the right treatment for vector data). Losing precision of min/max values and their timestamps is acceptable (at least for reasonable short archive_interval settings)

I hope to look into it, but it is not at the top of the priorities list at the moment :)

Cheers,

Jan-Jaap 

[gjr80]

Correct. The start of an interval depends on the current timestamp. The start of the interval right now may well be different to the start of the interval if it were five seconds later. Consider all time since epoch timestamp=0 split into intervals archive_interval seconds long. Then the start of the interval is the timestamp at the start of the current archive_interval interval if that makes sense, more easily explained mathematically, mathematically it is defined as:

archive_interval_start_ts = int(ts/archive_interval) * archive_interval

where:

archive_interval_start_ts is the epoch timestamp of the start of the archive interval

ts is the current epoch timestamp

archive_interval is length of the archive interval in seconds

Within WeeWX it is calculated by calling the weeutil.weeutil.startOfInterval() function.

Gary

[gjr80]

That's the good thing about WeeWX, everyone can contribute. If you believe the documentation can be improved propose an amendment, preferably via a pull request, though I expect for something simple Tom would consider a request with suggested text in a post.

Gary

[Jan-Jaap van der Geer]

On Wednesday, February 10, 2021 at 6:39:42 AM UTC+1 gjr80 wrote:

That's the good thing about WeeWX, everyone can contribute. If you believe the documentation can be improved propose an amendment, preferably via a pull request, though I expect for something simple Tom would consider a request with suggested text in a post.

I appreciate that but I'm afraid my understanding of this stuff is nowhere near good enough to come up with a good alternative text. I'm not even sure you agree with my conclusion... :) 

Cheers,

Jan-Jaap

[Jan-Jaap van der Geer]

On Tuesday, February 9, 2021 at 7:29:55 AM UTC+1 michael.k...@gmx.at wrote:

In brief my observations with your driver:

• udp-packets are being picked up (and published with mqtt in my case, because I've installed the extension)
• missing database values are NOT being backfilled
• current archive records from loop data are NOT being generate

 

Getting back to this post... Please update the driver to the current one and try again.

If I understand correctly the first point wasn't a problem (not familiar with the mqtt stuff, don't even know what it is, but I suppose it's some kind of interfacing to other applications listening to published data in some way?).

Your third point is probably fixed with the latest version.

The second point has probably not changed, as I haven't done anything with this. But having said that, it does seem to be working fine here.

What did you do with the weewx.conf file? At a minimum you need to add a API-token, as described in the documentation. Otherwise it won't have access to the API and will skip using the REST interface. The log should in this case contain a line like this:

"Skipped fetching from REST API"

Do you have a sensor map in the config file? If you do and if you didn't change it you probably want to read what I wrote for the sensor map. The simplest solution is to simply delete it. For most configurations the driver will figure it out itself. But if you decide to keep it and if it is more or less the same as the one used by the original driver, then you really should read the section on the sensor map closely because you either need to make changes or accept that some fields aren't getting imported when using the REST packages.

Cheers,

Jan-Jaap

[michael.k...@gmx.at]

I'll give i a try the next few days. API-Token was already configured, it was probably something with the sensor map I indeed have in my config. How is "For most configurations the driver will figure it out itself" to be understood?

[Jan-Jaap van der Geer]

On Friday, 12 February 2021 at 07:01:44 UTC+1 michael.k...@gmx.at wrote:

I'll give i a try the next few days. API-Token was already configured, it was probably something with the sensor map I indeed have in my config.

Well, if it is your sensor map I think the only consequence would be missing values for the values fetched through the REST interface. Unless absolutely no fields get a value, then there would be no records created for the REST packages, but I doubt you can have a realistic sensor map and still end up in that situation so I suppose the problem must be something else.

  

How is "For most configurations the driver will figure it out itself" to be understood?

Well, if you have one Tempest, or one Air or one Skye or one set of Sky & Air, it should Just Work (tm). However if you have multiple units or unknown units then you would need to do things to the sensor map.

(Or if you're not satisfied with the mapping provided by the driver, of course)

Cheers,

Jan-Jaap

[michael.k...@gmx.at]

I'm confident I'll get along with this. Looking forward to get this "production ready": https://www.kainzbauer.net/weather/Rif-Tempest/live.html

[Jan-Jaap van der Geer]

On Friday, 12 February 2021 at 07:19:56 UTC+1 Jan-Jaap van der Geer wrote:

Well, if you have one Tempest, or one Air or one Skye or one set of Sky & Air, it should Just Work (tm). However if you have multiple units or unknown units then you would need to do things to the sensor map.

 

BTW note that the driver has command line options to emit a sensor map for you to look at or copy/paste into the weewx.conf. You just need to provide the same values as you have set in the weewx.conf file. The values actually used by the driver are also emitted in the logfile (this was already the case before I started working on the driver), although that's in a different format.

Cheers,

Jan-Jaap

[Jan-Jaap van der Geer]

On Friday, 12 February 2021 at 07:23:23 UTC+1 michael.k...@gmx.at wrote:

I'm confident I'll get along with this. Looking forward to get this "production ready": https://www.kainzbauer.net/weather/Rif-Tempest/live.html

Nice site. Is that skin or whatever it is publicly available?

How does your Tempest register precipitation even though it's freezing cold? Mine certainly doesn't register snow...

Cheers,

Jan-Jaap

[michael.k...@gmx.at]

The skin is Nick's bootstrap skin, he and me are currently working on it to show live data (did you notice the 3s updates at the wind gauges) more Info: https://groups.google.com/g/weewx-user/c/3BCDn6lNEPw/m/nXAosYWEAwAJ here ist the current bleeding edge version: https://github.com/brewster76/fuzzy-archer/tree/liveCharts

The Tempest is mounted on a 4m Mast on top of the roof of my house ( https://www.kainzbauer.net/weather/Rif/news.html ) and despite of implementing a Scruton Helix, it still is shaking in the wind, causing the haptic sensor to register precipitation. Still work to do, I need to experiment with weights on in the upper part of the mast to lower the oscillating frequency, hope, his will help.

The more realistic values are still with my ws28xx: https://www.kainzbauer.net/weather/Rif/live.html

[michael.k...@gmx.at]

Hi Jan-Jaap,

seems to work! I made two local changes: 1.) I removed the 

    @property

    def archive_interval(self):

        return 60

And changed the logging of "Import from UDP" to debug level. The driver works now for me, the last thing on the wishlist would bei to let genStartupRecords only generate one entry per archive_interval. I might do that myself if I find some time.

Good work! And thank you for the driver enhancements!

[Jan-Jaap van der Geer]

On Sunday, February 14, 2021 at 8:10:44 PM UTC+1 michael.k...@gmx.at wrote:

seems to work!

Great news! Thanks for letting me know! :)

 

I made two local changes: 

1.) I removed the 

    @property

    def archive_interval(self):

        return 60

And changed the logging of "Import from UDP" to debug level. The driver works now for me, the last thing on the wishlist would bei to let genStartupRecords only generate one entry per archive_interval. I might do that myself if I find some time.

I'll incorporate these changes yes :)

 

Good work! And thank you for the driver enhancements!

You're welcome. Did you find out why the REST part didn't work initially? From your earlier explanation, I don't really see why it wouldn't work for you?

A bit surprised you didn't take up an issue that I had expected you'd mention: The skin you use seems to actively use the wind packages that come much more often in the UDP messages, but which I have removed, at least when it comes to the default sensor maps. And if you want to use these nonetheless, by using an explicit sensor map, you will get problems when using the REST interface as these are not really compatible as it is.

I thought we didn't loose too much functionality by doing that in the driver, but I now understand that there actually is a use case for that stuff as well.

I have some ideas how to address that for a future version.

Cheers,

Jan-Jaap

[michael.k...@gmx.at]

>  You're welcome. Did you find out why the REST part didn't work initially? From your earlier explanation, I don't really see why it wouldn't work for you? 

It was  devices = ST-000xxxxx

> A bit surprised you didn't take up an issue that I had expected you'd mention: The skin you use seems to actively use the wind packages that come much more often in the UDP messages, but which I have removed,

> at least when it comes to the default sensor maps. And if you want to use these nonetheless, by using an explicit sensor map, you will get problems when using the REST interface as these are not really compatible as it is.

> I thought we didn't loose too much functionality by doing that in the driver, but I now understand that there actually is a use case for that stuff as well.

Well, here is why:

    [[sensor_map]]

        outTemp = air_temperature.ST-000xxxxx.obs_st

        outHumidity = relative_humidity.ST-000xxxxx.obs_st

        pressure = station_pressure.ST-000xxxxx.obs_st

        lightning_strike_count =  lightning_strike_count.ST-000xxxxx.obs_st

        lightning_distance =  lightning_strike_avg_distance.ST-000xxxxx  .obs_st

        supplyVoltage = battery.ST-000xxxxx.obs_st

        windSpeed = wind_avg.ST-000xxxxx.obs_st

        windDir = wind_direction.ST-000xxxxx.obs_st

        currentWindSpeed = wind_speed.ST-000xxxxx.rapid_wind

        currentWindDir = wind_direction.ST-000xxxxx.rapid_wind

        windGust = wind_gust.ST-000xxxxx.obs_st

        luminosity  = illuminance.ST-000xxxxx.obs_st

        UV = uv.ST-000xxxxx.obs_st

        rain = rain_accumulated.ST-000xxxxx.obs_st

        radiation = solar_radiation.ST-000xxxxx.obs_st

I map the rapid_wind data to nonexistent (database) keys ("currentWindSpeed", "currentWindDir"). These values get into the loop and are published to the mqtt server to feed the live gauges and only the live gauges of the skin I use. You can see what happens, if you debug the javascript of my skin and take a closer look to the "jPayload"-variable on "site.js". Every three seconds there will be a packet containing "currentWindSpeed" and "currentWindDir", updating the "Wind", "Winböen" (gust) and "Windrichtung" (direction) - gauges. Yes, all three of them, even without having "windGust" in the package. If the currentWindSpeed is higher than the gust-gauges value, the new gust value is currentWindSpeed. It's reset with the once-a-minute package cointaining "windGust_mps", which is mapped to windGust in the above stanza.

 The other values are stored in the database and they also feed the live charts. There is no problem at all, I use all values in their special way, I am fully compatible. There is absolutely no benefit in storing rapid_wind data when the device calculates the avg values on a one-minute-base, and my interval is on a 5-minute-base.

[Jan-Jaap van der Geer]

On Monday, February 15, 2021 at 10:26:25 PM UTC+1 michael.k...@gmx.at wrote:

>  You're welcome. Did you find out why the REST part didn't work initially? From your earlier explanation, I don't really see why it wouldn't work for you? 

It was  devices = ST-000xxxxx

Hm, why would that not work? I just tried that  but it worked for me. (I suppose you don't mean x but the actual number, right?)

> A bit surprised you didn't take up an issue that I had expected you'd mention: The skin you use seems to actively use the wind packages that come much more often in the UDP messages, but which I have removed,

> at least when it comes to the default sensor maps. And if you want to use these nonetheless, by using an explicit sensor map, you will get problems when using the REST interface as these are not really compatible as it is.

> I thought we didn't loose too much functionality by doing that in the driver, but I now understand that there actually is a use case for that stuff as well.

Well, here is why:

    [[sensor_map]]

        outTemp = air_temperature.ST-000xxxxx.obs_st

        outHumidity = relative_humidity.ST-000xxxxx.obs_st

        pressure = station_pressure.ST-000xxxxx.obs_st

        lightning_strike_count =  lightning_strike_count.ST-000xxxxx.obs_st

        lightning_distance =  lightning_strike_avg_distance.ST-000xxxxx  .obs_st

        supplyVoltage = battery.ST-000xxxxx.obs_st

        windSpeed = wind_avg.ST-000xxxxx.obs_st

        windDir = wind_direction.ST-000xxxxx.obs_st

        currentWindSpeed = wind_speed.ST-000xxxxx.rapid_wind

        currentWindDir = wind_direction.ST-000xxxxx.rapid_wind

        windGust = wind_gust.ST-000xxxxx.obs_st

        luminosity  = illuminance.ST-000xxxxx.obs_st

        UV = uv.ST-000xxxxx.obs_st

        rain = rain_accumulated.ST-000xxxxx.obs_st

        radiation = solar_radiation.ST-000xxxxx.obs_st

I map the rapid_wind data to nonexistent (database) keys ("currentWindSpeed", "currentWindDir"). These values get into the loop and are published to the mqtt server to feed the live gauges and only the live gauges of the skin I use. You can see what happens, if you debug the javascript of my skin and take a closer look to the "jPayload"-variable on "site.js". Every three seconds there will be a packet containing "currentWindSpeed" and "currentWindDir", updating the "Wind", "Winböen" (gust) and "Windrichtung" (direction) - gauges. Yes, all three of them, even without having "windGust" in the package. If the currentWindSpeed is higher than the gust-gauges value, the new gust value is currentWindSpeed. It's reset with the once-a-minute package cointaining "windGust_mps", which is mapped to windGust in the above stanza.

 The other values are stored in the database and they also feed the live charts. There is no problem at all, I use all values in their special way, I am fully compatible. There is absolutely no benefit in storing rapid_wind data when the device calculates the avg values on a one-minute-base, and my interval is on a 5-minute-base.

Ah, smart! I understand.

I'm thinking of having some kind of suffix ('.udp' and '.rest' for example)  which could then be used to differentiate between the two maps.

There might be a tiny benefit by letting weewx accumulate the rapid_wind instead of using the accumulation of the Tempest. If weewx does the accumulation you also get a windGustDir, which you won't get by letting Tempest do the accumulation. Not sure if there's much value in that datapoint, but it is a difference...

Cheers,

Jan-Jaap

[michael.k...@gmx.at]

> Hm, why would that not work? I just tried that  but it worked for me. (I suppose you don't mean x but the actual number, right?)

I didn't have this value in my config at all

> If weewx does the accumulation you also get a windGustDir

That's a good point.

By now, you are using the mapping, if specified, for both I guess?

> I'm thinking of having some kind of suffix ('.udp' and '.rest' for example)   

    [[sensor_map]]

        #this is the sensor mapping used for UDP and REST

        ...

        windSpeed = wind_avg.ST-000xxxxx.rapid_wind

        windDir = wind_direction.ST-000xxxxx.rapid_wind

        windGust = wind_gust.ST-000xxxxx.obs_st

        ...

        [[[REST]]] #if configured, use this mapping when getting data from REST, values are inherited from above

            windSpeed = wind_avg.ST-000xxxxx.obs_st #overwrites above mapping

            windDir = wind_direction.ST-000xxxxx.obs_st #overwrites above mapping

Or like this?

I still have one thought: let's assume your WIFI has an outage. In this case, a friend told me, the station is holding back all the data. When the WIFI is back, it burst all the missing UDP values, the timestamps are all there. How to deal with that? Weewx is still online, because, in my example it's on a wired device. What does the driver do with that data?

And then a similar case: a power outage: the power comes back, WIFI comes back, the station burst all the values. Weewx comes back just right after that, tries to fetch missing data, which not yet is available via REST. Weewx starts up, doesn't get any values from REST, it never tries to get them again.

In both cases I know what I would do, because I save my weewx database every hour a day. overwriting the backup after 24 hours, but keep the 0:00 copy for a month before overwriting. So I would go back to a database backup before the outage and start weewx when all values are available via REST.

[michael.k...@gmx.at]

@Jan-Jaap: a friend of mine made a couple of things mentioned above and filed a pull request.

[Jan-Jaap van der Geer]

On Tuesday, February 16, 2021 at 5:52:06 AM UTC+1 michael.k...@gmx.at wrote:

> Hm, why would that not work? I just tried that  but it worked for me. (I suppose you don't mean x but the actual number, right?)

I didn't have this value in my config at all

Hm, that shouldn't matter. As long as you have one tempest, it should just find out about that... 

What does this report? :

PYTHONPATH=./bin python3 ./bin/user/weatherflowudp.py --token=<insert your token here> --create-sensor-map

> If weewx does the accumulation you also get a windGustDir

That's a good point.

By now, you are using the mapping, if specified, for both I guess?

Yes, that is correct.

> I'm thinking of having some kind of suffix ('.udp' and '.rest' for example)   

    [[sensor_map]]

        #this is the sensor mapping used for UDP and REST

        ...

        windSpeed = wind_avg.ST-000xxxxx.rapid_wind

        windDir = wind_direction.ST-000xxxxx.rapid_wind

        windGust = wind_gust.ST-000xxxxx.obs_st

        ...

        [[[REST]]] #if configured, use this mapping when getting data from REST, values are inherited from above

            windSpeed = wind_avg.ST-000xxxxx.obs_st #overwrites above mapping

            windDir = wind_direction.ST-000xxxxx.obs_st #overwrites above mapping

Or like this?

Not sure I like that. I would rather have one map that can be generated by the software if omitted. The above model makes that a bit more complicated IMHO.

I still have one thought: let's assume your WIFI has an outage. In this case, a friend told me, the station is holding back all the data. When the WIFI is back, it burst all the missing UDP values, the timestamps are all there. How to deal with that? Weewx is still online, because, in my example it's on a wired device. What does the driver do with that data?

I don't think that's a driver question. The question is more: What does weewx do then? The driver supplies a timestamp so in theory the driver should just pick up the UDP values (I don't know if they'd come in the correct order. Tom Keffer has indicated that they sometimes come out of order, and this may well be such a case. I don't know. I also don't know if weewx would cope with that (maybe not since Tom was not at all happy about it?)

 

And then a similar case: a power outage: the power comes back, WIFI comes back, the station burst all the values. Weewx comes back just right after that, tries to fetch missing data, which not yet is available via REST. Weewx starts up, doesn't get any values from REST, it never tries to get them again.

There really is no other way I suppose. How is the driver supposed to know that it should wait and how long? If you're concerned about this maybe you should add a pause before weewx starts, that might solve this scenario.

In both cases I know what I would do, because I save my weewx database every hour a day. overwriting the backup after 24 hours, but keep the 0:00 copy for a month before overwriting. So I would go back to a database backup before the outage and start weewx when all values are available via REST.

That's another solution, but it's a bit more hassle I suppose...

Cheers,

Jan-Jaap 

[Jan-Jaap van der Geer]

On Tuesday, February 16, 2021 at 4:56:49 PM UTC+1 michael.k...@gmx.at wrote:

@Jan-Jaap: a friend of mine made a couple of things mentioned above and filed a pull request.

Nice! I saw it earlier. I'll take a more thorough look later on. :)

Cheers,

Jan-Jaap

 

[michael.k...@gmx.at]

The current state of the driver seems to meet what I'd expect. Thanks again to everyone.