Strange entries in database

[Maginos]

Hi guys,

I have a question on some strange entries in the weewx database concerning the date of the entry. I have set polling_interval = 1800 and polling_mode = ADAPTIVE, so weewx asks the base every 48 s for a new entry and every 30 minutes, the base gets new data. So I expect entries at XX:00 and at XX:30. When weewx is stopped, or the rasperry is shutdown, it asks for a new entry at the next start of weewx. But somehow, no :00 or :30 values are added to the database, but :04 or :34 and I cannot explain why this is. And unfortunately, these entries are not displayed in the graphics on the website, for whatever reason. There are no error messages in the logs, and all other services (AWEKAS, PWS, Wunderground, etc.) also get these strange entries.

Can someone help me with this?

Thank you in advance!

Maginos

[gjr80]

Hi,

Would help to see the logs. The timestamp for each record saved to archive is recorded in the log. A log extract showing the full WeeWX startup and a log extract covering the period of concern would help.

Gary

[Maginos]

Hi Gary and thank you for your answer,

unfortunately, the logs don't show anything special, just a normal entry in the database, as it would for every "right" entry. I also checked the entry in the database, there's nothing wrong with it, except the time stamp. I switched on debug mode in weewx.conf and later I can test, if the debug log is helpful. What is also strange is, that the entries in the database in the base of the weather station are at minute 5 and 35 and not at 0 and 30. 

[gjr80]

Didn’t expect any errors in the log, the log unambiguously records the timestamp of all records saved to archive. If you have timestamps appearing in the database that should not be there then the log is the first place to check. The startup portion of the log shows key config info for your setup. Helps add to the picture and help us avoid wasting our time asking 20 questions. Sometimes it’s not about errors, it’s about getting a clear picture.

Gary

[Maginos]

So here's the log.

Before 12 o'clock I stopped weewx, started it, until it has finished and stopped again. At 12:09, I started it again. 
I hope this is helpful.

[gjr80]

OK, that helps paint the picture. If I understand correctly your concern is the timestamp of these records:

Apr 1 12:11:25 maginos weewx[4560]: manager: Added record 2020-04-01 12:05:10 CEST (1585735510) to database 'weewx.sdb'
Apr 1 12:11:25 maginos weewx[4560]: manager: Added record 2020-04-01 12:05:10 CEST (1585735510) to daily summary in 'weewx.sdb'

You expect the timestamps should be xx:00:00 (or xx:30:00) but not xx:05:10. In your case WeeWX is configured to synthesise archive records every 30 minutes from the loop data received during that 30 minute archive interval. This will result in archive records on xx:00:00 and xx:30:00 boundaries and once your station has been running for a while this is what will occur. What you are seeing is a hardware catchup. When WeeWX starts up it asks the driver if your station has any stored archive records that are newer than the last record in the WeeWX archive. This is WeeWX catching up if WeeWX has been disconnected from the station for some time. During a hardware catchup WeeWX does not synthesise the archive records, rather WeeWX takes the archive records (and their timestamps) from the driver/station as is and saves them to archive. In your case the driver is obtaining an archive record from the station that has an unusual timestamp and WeeWX is accepting that record and saving it to archive as is. I am no fine offset expert but I know the fine offset driver ignores the station clock. I suspect the station timestamps the archive records that it stores in memory but these timestamps are in no way synchronised with the WeeWX clock, hence you see records that are stored in the station memory being retrieved with unusual timestamps. It may be the station is saving a record in memory every 30 minutes, just not on the hour and half hour as WeeWX does.

Since restful services such as WOW, CWOP etc are largely archive record based these services are passed the record with the odd timestamp and that is why these servies are using the odd timestamp.

So what can you do. There are a couple of things that spring to mind that might mitigate the situation. Firstly, you could verify that your station hardware is using the same archive interval that you have WeeWX set to use (30 minutes). You can use the wee_device utilitily using with the --info and --set-interval actions to check and set the interval respectively. I suspect that even if the WeeWX and station hardware intervals are the same there will still be odd timestamped records during hardware catchup. You could drop the archive interval used by WeeWX and the station hardware to something like five minutes. Again this will not eliminate the problem but it may make it less noticeable. Finally, you could use the undocumented no_catchup config option in weewx.conf to prevent WeeWX doing a hardware catchup on startup:

[StdArchive]
    ....
    no_catchup = True
    ....

This will eliminate the odd timestamped records you are seeing but it will also prevent WeeWX from automaticlly catching up if it is disconnected from your station for a period of time. Or finally you could leave it as is, the odd timestamped records will not trouble WeeWX. I am not so certain about the external services your are uploading to, some have requirements of uploaders bnu these are usually in regards to frequency of uploads so they may not be a problem.

Then again an experienced fine offset user may come along  too...

Gary

[Maginos]

Ok Gary, if I cannot do anything about the timestamps, I'm completely fine with it, as long as these entries are displayed in the graphs on my public webserver. The main goal of this weather station is, to provide my family, the people in my village and of course me with recent data about the weather. The uploads to WOW, Wunderground, etc. is just bonus. 
The reason, why I noticed these odd timestamps is, that these entries were NOT displayed in the graphs on my webserver and then I thought, this could be connected to the odd timestamps. Small spoiler: It is indeed, but the timestamp is not the problem.
In the skin.conf file there is an option which is called line_gap_fraction = 0.01. As you probably know, this means, that a timespan between two entries in the database >1% of the displayed time (for recent weather 27 hours) is considered to be a gap and therefore not displayed in the graphs (or displayed with dots). Since I have an update interval of 30 minutes, I had to set the line_gap_fraction to 0.02 (27*60*0.02 = 32.4). I think you see, where this is going... When I have entries in the database with timestamps of minute 0 and 35, this entry is considered as a gap and not displayed in the graphs on the webserver. So I just had to set line_gap_fraction = 0.03 (27*60*0.03 = 48.6) and now all the entries with odd timestamps are displayed correctly in the graphs. As long as I don't change the archive interval in the station and in weewx, I can do maintenance on the Pi, as long as I want and all data are displayed correctly in the graphs.

So for me, this topic is finished and I reached my aim, which is doing maintenance on the Pi as long as I want, without being in a hurry all the time. 

Thank you for your help and your time Gary!

[gjr80]

I think your line gap analysis correct and as long as you are happy with the outcome then all is well.

Gary