Belchertown skin line graph gaps

[István Hegedűs]

Hi guys! I'm using the great Belchertown skin but I have gaps on today line graphs. These are 5 or 10 minutes long.

Sometimes (as you can see on the second picture) the data is present but there is no line on the graph.

Weewx 3.9.2 on RPI

Belchertown skin 1.0.1

report_timing = */5 * * * *

archive_interval  = 300 (from hardver)

default skin and graphs config file

Can you help me?

[Pat]

This is due to your gapSize not matching your archive_interval or something else is going on with your weewx not archiving reliably and skipping archives. Your log files will know more. 

For starters double check that gapSize matches your archive_interval. 

[István Hegedűs]

gapsize = 300000

I think the problem is that weewx archives with some secound differences. Attached part of log file. 

[gjr80]

I question the wisdom of specifying a report_timing option setting of */5 * * * * on a system with a five minute archive interval. I am not sure what effect you are trying to achieve but the cron like report_timing option works off the timestamp of the latest archive record, if the difference between successive timestamps is very close to the report_timing value I would not be surprised to see some report cycles being skipped. This would be even more likely to occur if your archive records are consistently coming in less than five minutes apart (as many of yours are).

It may have no affect on your go problem, but it might be worth removing your report_timing setting and restarting WeeWX.

Gary

[István Hegedűs]

Thanks, but unfortunately it didn't help. I think the problem is the differences between archive record timestamps:

Jan  3 21:05:24 raspberrypi weewx[325]: manager: Added record 2020-01-03 21:03:23 CET (1578081803) to database 'weewx.sdb'

Jan  3 21:05:24 raspberrypi weewx[325]: manager: Added record 2020-01-03 21:03:23 CET (1578081803) to daily summary in 'weewx.sdb'

Jan  3 21:10:26 raspberrypi weewx[325]: manager: Added record 2020-01-03 21:08:25 CET (1578082105) to database 'weewx.sdb'

Jan  3 21:10:26 raspberrypi weewx[325]: manager: Added record 2020-01-03 21:08:25 CET (1578082105) to daily summary in 'weewx.sdb'

1578081803 - 1578081803 = 302 sec > 300000 (gapsize). But sometimes is less than 300 sec.

[gjr80]

Guess that is one for Pat.

I think it would help to know how WeeWX is configured and what it is doing. Could you post a wee_debug (http://weewx.com/docs/utilities.htm#wee_debug_utility) report. Make sure you check the wee_debug output for sensitive info before posting, wee_debug is good at obfuscating such info but it’s not perfect. Also post a debug log from startup; restore your StdReport log output, set debug = 1 in weewx.conf, restart WeeWX and let WeeWX run for at least 10 minutes. Then post a log extract from WeeWX startup through the 10 minutes, make sure you include the full WeeWX startup.

Gary

[Pat]

Side note: I'm a little lost on this one. Not sure why weewx isn't saving records consistently every 300s. (Nothing to do with the skin). Maybe something to do with the resources of the Pi being stretched thin?

[gjr80]

I think that is why we need some config and log info.

Gary

[vince]

On Friday, January 3, 2020 at 12:41:08 PM UTC-8, Pat wrote:

Side note: I'm a little lost on this one. Not sure why weewx isn't saving records consistently every 300s. (Nothing to do with the skin). Maybe something to do with the resources of the Pi being stretched thin?

 

What model pi is it ?

What other than weewx is running on it ?

I'll have to check my tiny/weak Seagate Dockstar at home, but my recollection is that weewx not saving exactly on a 300s interval is not that unusual.    Sometimes the skins take longer than at other times to process as well.  Belchertown is a rather intensive skin to run from what I've seen.   My recollection is the processing time for that one can be rather high compared with more minimal skins.

[Pat]

I agree that Belchertown skin has a lot going on with it, but that shouldn't delay weewx from saving the archive since the skin runs after the archive is run. 

The only scenario I can think of where the skin would slow things down is if the skin is running longer than 300s to generate the graph json files, which is causing weewx to delay on the next archive interval. 

This was common on a Pi Zero in early versions of the skin (I haven't seen it being reported in the last ~12 months though since updating the chart json generator)

[rich T]

I'm running Weewx 3.9.2 / Belchertown skin 1.1b3 / MQTT enabled on RPI 3 with the Rtldavis driver , BME280, and AS3935.  I archive data every 300 seconds without any issues.

[gjr80]

I really don't think this has anything to do with when the records may or may not be saved to archive, rather it has to do with the timestamps of the records (there is a difference). A system under load may save records at (slightly) different times but the system should still timestamp archive records in a consistent manner.

Little point in further supposition until we have config and log info.

Gary

[István Hegedűs]

Attached wee_debug output and log. I'm running Weewx on a 1st gen RPI.

[vince]

On Friday, January 3, 2020 at 2:41:53 PM UTC-8, gjr80 wrote:

I really don't think this has anything to do with when the records may or may not be saved to archive, rather it has to do with the timestamps of the records (there is a difference). A system under load may save records at (slightly) different times but the system should still timestamp archive records in a consistent manner.

ok, my very wimpy system is 'very' regular in timestamps it archives, always ending in :00 seconds of the minute.

Jan  3 18:05:19 debian weewx[13833] INFO weewx.manager: Added record 2020-01-03 18:05:00 PST (1578103500) to database 'weewx.sdb'
Jan  3 18:10:19 debian weewx[13833] INFO weewx.manager: Added record 2020-01-03 18:10:00 PST (1578103800) to database 'weewx.sdb'
Jan  3 18:15:19 debian weewx[13833] INFO weewx.manager: Added record 2020-01-03 18:15:00 PST (1578104100) to database 'weewx.sdb'

Jan  3 18:20:16 debian weewx[13833] INFO weewx.manager: Added record 2020-01-03 18:20:00 PST (1578104400) to database 'weewx.sdb'

Syslog reports this happens usually about 19 seconds after the minute, but periodically a few seconds quicker than that as indicated in the snippet above.

Just in case it matters....

[István Hegedűs]

Maybe is it a bug in WS23xx driver?

[gjr80]

The issue of archive record timestamps is somewhat complex, what results are seen (and how that result was derived) vary depending on a couple of factors. First up the type of archive record generation. If the install is using software record generation then WeeWX is controlling the archive record timestamps and you will see nice archive records sitting on top of the minute boundaries eg, 00:05, 00:10 not 00:05:23, 00:10:21. If you are using hardware record generation then you are at the mercy of the station and driver, so the answer here is it depends. If the station/driver is capable of emitting hardware generated archive records then the timestamp you see if what the driver/station emitted. If the driver/station does not support emitting hardware based archive records then WeeWX falls back to software record generation with 'nice' archive record timestamps. OK, maybe not so complex but not so simple either.

I note from the wee_debug output that the OPs install is using hardware record generation (WeeWX default to hardware record generation if nothing is specified for the record_generation setting in [StdArchive]). it might be an interesting test to change that to software record generation (ie set record_generation = software and restart WeeWX) and see what the results are. That should give you 'nice' top of the minute archive record timestamps and it might be interesting to see if that helps with the gap issue. I am not sure how the Belchertown plots are generated, it might take some time before a 'new' history is built up under software record generation. Note that I am not suggesting the change to software record generation is a fix for the gap problem, rather I see it as a simple way to see if the irregular archive record timestamps are the source of the issue and if so that may narrow down the solution in terms of tweaking of the Belchertown settings/operation.

Gary

[István Hegedűs]

Ok, so I changed to record_generation = software and now I have nice archive record timestamps:

Jan  5 11:45:24 raspberrypi weewx[16347]: manager: Added record 2020-01-05 11:45:00 CET (1578221100) to daily summary in 'weewx.sdb'

Jan  5 11:50:18 raspberrypi weewx[16347]: manager: Added record 2020-01-05 11:50:00 CET (1578221400) to database 'weewx.sdb'

Jan  5 11:50:18 raspberrypi weewx[16347]: manager: Added record 2020-01-05 11:50:00 CET (1578221400) to daily summary in 'weewx.sdb'

Jan  5 11:55:21 raspberrypi weewx[16347]: manager: Added record 2020-01-05 11:55:00 CET (1578221700) to database 'weewx.sdb'

Jan  5 11:55:22 raspberrypi weewx[16347]: manager: Added record 2020-01-05 11:55:00 CET (1578221700) to daily summary in 'weewx.sdb'

Jan  5 12:00:20 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:00:00 CET (1578222000) to database 'weewx.sdb'

Jan  5 12:00:20 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:00:00 CET (1578222000) to daily summary in 'weewx.sdb'

Jan  5 12:05:15 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:05:00 CET (1578222300) to database 'weewx.sdb'

Jan  5 12:05:15 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:05:00 CET (1578222300) to daily summary in 'weewx.sdb'

Jan  5 12:10:26 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:10:00 CET (1578222600) to database 'weewx.sdb'

Jan  5 12:10:26 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:10:00 CET (1578222600) to daily summary in 'weewx.sdb'

Jan  5 12:15:17 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:15:00 CET (1578222900) to database 'weewx.sdb'

Jan  5 12:15:17 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:15:00 CET (1578222900) to daily summary in 'weewx.sdb'

Jan  5 12:20:23 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:20:00 CET (1578223200) to database 'weewx.sdb'

Jan  5 12:20:23 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:20:00 CET (1578223200) to daily summary in 'weewx.sdb'

Jan  5 12:25:22 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:25:00 CET (1578223500) to database 'weewx.sdb'

Jan  5 12:25:22 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:25:00 CET (1578223500) to daily summary in 'weewx.sdb'

Jan  5 12:30:17 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:30:00 CET (1578223800) to database 'weewx.sdb'

Jan  5 12:30:17 raspberrypi weewx[16347]: manager: Added record 2020-01-05 12:30:00 CET (1578223800) to daily summary in 'weewx.sdb'

This fixed my graphs so maybe the driver or just the cheap weather station is the problem.

Are there any disadvantages of software record generation?

[gjr80]

OK, so we know the cause. I don't see this as an issue with the driver/station per se (there is no rule specifying the need for equal hardware archive periods), rather an issue between Belchertown and drivers/stations that don't emit archive records with exactly the same length. Maybe the Bechertown gapsize setting can be changed, I have no experience with Belchertown, that is one for Pat.

There is no particular disadvantage to using software record generation, WeeWX takes all of the loop data seen in each archive period and synthesises and archive record using an appropriate aggregate on each observation. Some stations emit additional data/info in hardware archive records and that may be of value to the user and justification to use hardware record generation. Some (many ?) stations/drivers do not support hardware record generation. It really comes down to your needs and the capabilities of your station and the driver.

Gary

[Pat]

Just as an FYI - gapSize isn't a function of Belchertown, it's a highcharts highstock function. Belchertown just passes along that config value to the JavaScript for the charts to show or not show a gap where data is expected to be. I'm not sure how to implement a "buffer" around this setting since it isn't my setting to begin with. 

[gjr80]

Wellllll, maybe so, but it's a setting made in the Belchertown skin somewhere and that is how the user controls the setting.... That aside, and I have only had a cursory glance at gapsize in Highcharts, what happens if it is set to say four minutes on a nominal five minute archive period system? What happens if you go the other way to six minutes? I expect the variable archive period stays pretty close to the nominal value. Seems to me a little tinkering by the OP would be appropriate. No need for any complex code changes just a few config changes.

Gary