Database - Data not archived correctly

[Danie Cillie]

Morning,

I use a Raspberry Pi with the GroveWeatherRack and sensors and weewx.

The grovePi driver sends the data directly to weewx.

The LOOP data and the archive data as printed on the terminal when weewxd runs is correct and updates correctly, but  the humidity data is stored correctly the first time in the database and then gets stuck. Sometimes after a long time (hours) it changes to a new value and sticks to that again.

I can not understand why the data is read and displayed correctly but not stored in the database.

The data on this page illustrates what the problem is https://www.pwsweather.com/obs/IBLUERID2.html

Please find the config file and log file attached.

Any suggestions appreciated.

[Andrew Milner]

how do you know it is 'stuck'??  the archived value is the AVERAGE of readings received during the archive period.  What values are displayed that you say are correct - the loop value or the archive record value??  Remember that humidity does not change very quickly.

[Danie Cillie]

It will stay at a certain value for example 51% for a long time even though the loop value has changed drastically. Both the loop and archive record values are correct when displayed on the terminal but that is not what is written to the database.

How does one determine the archive period over which the values are averaged?

[Andrew Milner]

it is defined in weewx.conf [StdArchive] section in seconds, and is usually 300 (5 minutes).

http://weewx.com/docs/usersguide.htm#StdArchive

If the hardware has a different archive interval weewx will use that.

if you stop weewx, and start it again the log when weewx is started will show the archive interval to be used.

[Danie Cillie]

The archive period you refer to, is that the value set in the weewx.conf file? I have reduced that to 240 or 3 minutes. If I understand you correctly, the average is over the 3 minute period until the next write is done to the database.

If I run weewxd directly, (sudo weewxd weewx.conf) the LOOP values are correct.

Then I assume the values read (poll_interval is 2.5 seconds) in the 3 minute period is averaged and written to the database, is that correct?

I'll monitor the readings over a few days to see if I can make any sense out of this.

[Andrew Milner]

that is correct.  BUt you really should not see much difference between polling values if you are polling at 2.5 seconds!!  But the value which is archived will be the average value over the 3 minute archive period.

If you are displaying the LOOP packets - the values written to the database are in the REC packet - which should occur once every 3 minutes - and they are ALL averaged values for values received in the LOOP packets of the interval - pressure, temperature, humidity etc etc

[Danie Cillie]

Since my last message, the humidity was displayed as 48% (last 3 hours) even though we had rain and the direct sensor reading was 94%.

Just out of desperation I stopped the weewx engine and restarted and the humidity updated.

So for some reason it only updates when the weewx engine starts.

[Andrew Milner]

what does the log say from startup until two archive intervals have passed??

the answer usually lies in the log - so please post your log from startup until two intervals later.

Also, as a seperate exercise, can you post the result of running weewx directly and give us the LOOP and REC data for at least two archive intervals.

When you say the humidity updated what exacgly do you mean - in a report, in weewx.sdb or somewhere else.  When you say humidity is displayed as 48% where and how is it displayed?  weewx does not 'display' data anywhere it merely stores archive data in a database, runs a report generator to create html pages and outpts data to other places - there is no 'display' anywhere

[Danie Cillie]

By display I mean the html page in a web browser.

I have since stopped and started weewx a few times and every time it updates the humidity value on the webpage and in the database.

I'll attach log shortly.

How do I capture the LOOP and REC data?

[Danie Cillie]

Here is the log information

[Andrew Milner]

why do you say it is updated in the webpage and the database??  The data on the webpage comes from the database!!

have you tried forcing a browser page reload (shift refresh usually) when things appear 'stuck'??  There is no reason whatsoever why humidity should be any different with regards updates than the other fields?

you can capture the terminal output from running directly by sending the terminal output to a file as a simple way

the log does not appear to show anything strange, and confirms that archive records are being written every 3 minutes

[Danie Cillie]

I think I am starting to understand what is happening.

I'll try and explain what the symptoms are as clearly as possible:

When the weewxd engine is started, all values are stored to the database. However the humidity reading stays at the first level and does not change unless the weewxd engine is restarted.

This morning the humidity reading was still at the level displayed early last night. Only when I restarted the weewxd engine did the value change.

I am almost 100% certain that the problem is in the GrovePi driver.

I thought that the driver was providing the correct values when looking at the LOOP data but I'll have to monitor it over a longer period to see if it actually changes.

The LOOP data file is attached.

Thank you for your help

[Andrew Milner]

From the short bit of file you attached everything looked OK - the LOOP data seemed correct and so did the REC - however you only covered 1 archive period, and the value for that period seemed ok.

If you think you know what is happening then great, as there does not appear anything wrong in the short file you posted!

[gjr80]

Agree it is almost certainly the driver. Can you post the driver code you are using, or a link to the code, so we can have a look. As long as the driver is consistently emitting loop packets WeeWX should take care of the rest.

Gary

[Danie Cillie]

The driver I am currently using has the code added for the Adafruit SHT31D humidity sensor which connects to the I2C bus at address 0x44.

I can not use the official Adafruit driver as it needs python3 to run.

The code I added for the SHT31 is from line 233 to 262 and then from line 478 to 487.

The driver code is attached.

Thank you 

[gjr80]

genLoopPackets() assigns the variable ‘humidity’ to the ‘outHumidity’ field in the loop packet (line 341) but when genLoopPackets() is iterating continuously it calls the sampleAndDisplay() method to (presumably) obtain updated sensor values. As far as I can tell variable ‘humidity’ is not set by sampleAndDisplay(). ‘humidity’ is only set at line 254 when the driver module is first loaded. This probably explains why the value is only correct/updated when WeeWX is started/restarted.

I think you need to revise you driver to ensure that that variable ‘humidity’ is in fact updated on a regular basis.

Gary

[Danie Cillie]

Thanks Gary, I came to the same conclusion but have not been able to figure out how to get sampleAndDisplay() to read the humidity sensor.

Any ideas?

[Andrew Milner]

try copying in the code that reads the sensor used at startup would be a good starting point, and you may need to declare humidity as a global.  

[Danie Cillie]

Thanks, I got it working a short while ago and is busy testing.