Rain dailly total and NOAA rain total wrong value

[lionel...@free.fr]

Hello,

I use a RPI using a USB-RS232 converter with a Lacross WS2350 since December 2014 with Weewx 2.7. This last July, I upgraded the Weewx to 3.5 and since a week ago the 3.6.0 or above version.

I notice with these last versions the rain values include in the NOAA reports are wrong. I checked the data and I noticed these 2 points :

* The values in the NOAA report are x10 higher than in the SQL database in the archive_day_rain table

* The values of the NOAA report compared with other local station are very different. The ratio seem to be around of 0.66 from the measurement I get.

The reference site :

http://www.meteociel.fr/climatologie/climato.php?mois=11&annee=2016

My stattion :

http://meteo.eybens.free.fr/

http://meteo.eybens.free.fr/NOAA/NOAA-2016-11.txt

Is some one have any idea to solve this issue ?

Rds

lionel

PS: In the WS23xx.py driver it's necessary to have a delay in the delay read definition :

This sleep time allows to solve the hazardous or wrong reading data from the station. It's due to communication between the station and the outside sensors in the same time.

"""

    def read_byte(self, timeout=1.0):

        if self.log_mode != 'r':

            self.log_mode = 'r'

            self.log(':')

        result = self.serial_port.read_byte(timeout)

        if not result:

            self.log("--")

        else:

            self.log("%02x" % ord(result))

        time.sleep(0.01) # LSA use to slove synchro rs232 data in

        return result

"""

[mwall]

On Monday, January 2, 2017 at 5:30:56 PM UTC-5, lionel.sylvie wrote:

Hello,

I use a RPI using a USB-RS232 converter with a Lacross WS2350 since December 2014 with Weewx 2.7. This last July, I upgraded the Weewx to 3.5 and since a week ago the 3.6.0 or above version.

I notice with these last versions the rain values include in the NOAA reports are wrong. I checked the data and I noticed these 2 points :

* The values in the NOAA report are x10 higher than in the SQL database in the archive_day_rain table

* The values of the NOAA report compared with other local station are very different. The ratio seem to be around of 0.66 from the measurement I get.

lionel,

there was a bug in the ws23xx driver that resulted in rain values 10x too big.

this was fixed in june 2015 at commit 0bb7db1edc806caebe68357bc315ccf8674eea52

do you have something in your [Corrections] in weewx.conf that was fixing the bad rain values before you upgraded from weewx 2.7?

 

PS: In the WS23xx.py driver it's necessary to have a delay in the delay read definition :

This sleep time allows to solve the hazardous or wrong reading data from the station. It's due to communication between the station and the outside sensors in the same time.

have you been using this delay since weewx 2.7?  are you running it now with weewx 3.6?  what happens when you do not use the delay?

m

[lionel...@free.fr]

Le mardi 3 janvier 2017 00:15:40 UTC+1, mwall a écrit :

Thx for you support.

On Monday, January 2, 2017 at 5:30:56 PM UTC-5, lionel.sylvie wrote:

Hello,

I use a RPI using a USB-RS232 converter with a Lacross WS2350 since December 2014 with Weewx 2.7. This last July, I upgraded the Weewx to 3.5 and since a week ago the 3.6.0 or above version.

I notice with these last versions the rain values include in the NOAA reports are wrong. I checked the data and I noticed these 2 points :

* The values in the NOAA report are x10 higher than in the SQL database in the archive_day_rain table

* The values of the NOAA report compared with other local station are very different. The ratio seem to be around of 0.66 from the measurement I get.

lionel,

there was a bug in the ws23xx driver that resulted in rain values 10x too big.

this was fixed in june 2015 at commit 0bb7db1edc806caebe68357bc315ccf8674eea52

do you have something in your [Corrections] in weewx.conf that was fixing the bad rain values before you upgraded from weewx 2.7?

No I did anything, below the corrections settings from the weewx.conf :

"""

#   This section can adjust data using calibration expressions.

[StdCalibrate]

    

    [[Corrections]]

        # For each type, an arbitrary calibration expression can be given. 

        # It should be in the native units of the weather station hardware.

        # For example:

        outTemp = outTemp - 2

        inTemp = inTemp - 1

        dewpoint = dewpoint - 2

        heatindex = heatindex - 2

        windchill = windchill - 2

        pressure = pressure + 6

        # For each type, an arbitrary calibration expression can be given. 

        # It should be in the units defined in the StdConvert section.

        # Example:

        foo = foo + 0.2

# For each type, an arbitrary calibration expression can be given. 

# It should be in the units defined in the StdConvert section.

# Example:

# foo = foo + 0.2

"""

This is the same corrections in the weewx.conf between versions.

Where I can find the commits?

 

PS: In the WS23xx.py driver it's necessary to have a delay in the delay read definition :

This sleep time allows to solve the hazardous or wrong reading data from the station. It's due to communication between the station and the outside sensors in the same time.

have you been using this delay since weewx 2.7?  are you running it now with weewx 3.6?  what happens when you do not use the delay?

m

Yes I already used this delay in the WS23xx.py in the 2.7 and added it in the 3.5 in june and now in the 3.6. This solves the spikes issues in the temperature measurement and calculation which arrive randomly. I had mentioned this issue in December 2014 in the LaCrosse WS2300 & WS2350 support post. 

rds

[mwall]

On Tuesday, January 3, 2017 at 5:47:04 PM UTC-5, lionel.sylvie wrote:

This is the same corrections in the weewx.conf between versions.

Where I can find the commits?

the change to rain multiplier happened on 11 jun 2015:

https://github.com/weewx/weewx/commit/0bb7db1edc806caebe68357bc315ccf8674eea52

so it has been part of the ws23xx driver for some time now.

the divide-by-ten happens only to data from the logger, so if you are using record_generation=hardware you will definitely see a difference, but if you are using record_generation=software you will see a difference only when weewx has to do catchup.

 

 

PS: In the WS23xx.py driver it's necessary to have a delay in the delay read definition :

This sleep time allows to solve the hazardous or wrong reading data from the station. It's due to communication between the station and the outside sensors in the same time.

have you been using this delay since weewx 2.7?  are you running it now with weewx 3.6?  what happens when you do not use the delay?

Yes I already used this delay in the WS23xx.py in the 2.7 and added it in the 3.5 in june and now in the 3.6. This solves the spikes issues in the temperature measurement and calculation which arrive randomly. I had mentioned this issue in December 2014 in the LaCrosse WS2300 & WS2350 support post. 

adding a slight delay seems like an innocuous change, especially if it reduces or eliminates spikes.  on the other hand, magic numbers for serial communications make for some pretty stinky code.

i added the delay at commit 61b866a

ws23xx driver version is now 0.26rc1

m

[lionel...@free.fr]

Hi,

Le mercredi 4 janvier 2017 00:38:20 UTC+1, mwall a écrit :

On Tuesday, January 3, 2017 at 5:47:04 PM UTC-5, lionel.sylvie wrote:

This is the same corrections in the weewx.conf between versions.

Where I can find the commits?

the change to rain multiplier happened on 11 jun 2015:

https://github.com/weewx/weewx/commit/0bb7db1edc806caebe68357bc315ccf8674eea52

so it has been part of the ws23xx driver for some time now.

the divide-by-ten happens only to data from the logger, so if you are using record_generation=hardware you will definitely see a difference, but if you are using record_generation=software you will see a difference only when weewx has to do catchup.

 I compare all the versions of the WS23xx I used since 2.7, all are similar concerning the formula. The x10 in the commit above concern the wind speed which is included after the 2.7 version.

To come back to my original question, it's concern not the generated data but the values shown in the NOAA files.

In the file attached in this post NOAA of November, when we take the rain value for the 24 November 2016, this is 24.9mm. When we take the value in the database (weewx.sdb) for the same date ("1479942000" : Sat Nov 24 2016 00:00:00 GMT+0100), the value is 2.486. Is it the same units?

From a report of an official station close to mine (4-5kms), the cumul was 11.2mm.

http://www.infoclimat.fr/observations-meteo/archives/24/novembre/2016/saint-martin-d-heres/00014.html

When I do the sum of the rain falls during this same day (data taken in the archive table of weewx.sdb), I get 1.243 which is the half value form the archive_day_rain. Is it normal ?

When I sum all the rain data for November 2016 coming from the archive (= 8.029) and compare to the sum of the archive_day_rain for November (=16.058) and the NOAA of November (=160.6mm), I get a double value for the archive_day_rain and NOAA. Is there any issue during the calculation of the archive  ?

rds

Lio

PS : After these all analysis, the archive_day_rain is in cm and the NOAA is in mm. True? then I get the explanation for the x10 between values of archive_day_rain and NOAA.

[Andrew Milner]

If your database is metric then rain is in cm

If your database is metricwx then rain is mm

Your NOAA report is, according to the title line above the table, in mm - but the units are determined by the contents of the template and so can easily be changed to other units by changing the template appropriately.

http://weewx.com/docs/customizing.htm#units

[lionel...@free.fr]

Hello,

Ok for this part of configuration.

And concerning the sum of of the rain for a day, have you seen my remarks in my last post?

in summary, the SQL file (weewx.sdb), is made of different tables. One is all the stored data (called archive), and others which are statistics calculated by weewx.

When I calculate the rain fallen during a day from the stored data, this result which is closed to the other local weather station, is the half of the value calculated by weewx which is stored in the archive_day_rain table. From my point of view there is a mistake when weewx calculates the rain fallen during a day.

rds

lio

[Andrew Milner]

So you are saying that the sum of the archived data in the archive table for the calendar day does not equal the value in the sum column in the archive_day-rain table?

Does the value in he count column of archive_day_rain contain the correct number of archive records for the day which have been totalled to form the sum value??

[lionel...@free.fr]

Le jeudi 5 janvier 2017 10:49:44 UTC+1, Andrew Milner a écrit :

So you are saying that the sum of the archived data in the archive table for the calendar day does not equal the value in the sum column in the archive_day-rain table?

Yes. 

Does the value in he count column of archive_day_rain contain the correct number of archive records for the day which have been totalled to form the sum value??

 No I get 14 then archive_day_rain gives 28. See in the xls files in attached. 

What is the link between the sum of rain in a day and the number of rain recorded in this day ?

rds

Lio

[Andrew Milner]

Looking at my archive_day_rain table the count is usually = 288 which is the number of archive records per day.  I archive every 5 minutes (= 12/.hr = 12*24/day = 288 per day)  The sum = the total of the data from the archive records for that day which starts at dateTime column

[Andrew Milner]

Something else does not look quite right with your archive_day_rain table.  I would have expected the min to have been 0 unless it really did rain in every archive record (which it didn't according to your spreadsheet.  My table always has a zero for min.

Are you sure that you have only got one instance of weewx running?  It is starting to look as though somewhere,somehow your archive records are being counted and totalled twice.  Personally I have not seen or heard of that issue before.

Sorry - I will have to let Matthew and/or Tom come to your rescue!!

[lionel...@free.fr]

The configuration of my station is base on a record every 15min, this gives 96 records à day.

The archive_day_Rain shows a sum of 28 records for the 11/25,‎ in the xls file I also calculated the number of records in the day (95), the number of 'rainy' records in the day (14). I never find the 28 records.

This is not explain the wrong dayliy Rain sum.‎ Normaly to have the rain quantity for a day, the system just do the sum of Rain quantity of each record included in this day. 

Nb: i am sorry if m'y english is not so correct. I am french.

‎Lio

Envoyé de mon smartphone BlackBerry 10.

De: Andrew Milner Envoyé: jeudi 5 janvier 2017 12:33 À: weewx-user Répondre à: weewx...@googlegroups.com Objet: [weewx-user] Re: Rain dailly total and NOAA rain total wrong value

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/8bF5k9TlSZI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Milner]

I believe that weewx updates the daily tables for each and every archive record processed rather than just doing a calculation once per day.  Tom will correct me if I am wrong!!  The end result should indeed equal the sum of the archive records.  If you look at the day tables during the course of the day you will see that they contain the partial sums up to the point in time that you look at the table.

The number of 'rainy' records should not be affecting the counts - unless your system is behaving differently to mine - possibly because your station emits partial records.  Do the other daily tables have counts of 95 or 96?  Is it only rain which seems to count 'rainy' records rather than total records?  Does your archive contain null when there is no rain during that archive period - or does it contain zero?  This could explain the counting 'issue' if the archive contains null and not 0.

 

On Thursday, 5 January 2017 14:04:20 UTC+2, lionel...@free.fr wrote:

The configuration of my station is base on a record every 15min, this gives 96 records à day.

The archive_day_Rain shows a sum of 28 records for the 11/25,‎ in the xls file I also calculated the number of records in the day (95), the number of 'rainy' records in the day (14). I never find the 28 records.

This is nbecause your weather statot explain the wrong dayliy Rain sum.‎ Normaly to have the rain quantity for a day, the system just do the sum of Rain quantity of each record included in this day. 

[lionel...@free.fr]

The archives doesn't include the value 0 because with the 2.7 version, I add a minimum of limitation in weewx.conf to 0.001 mm due to rain error calculation. I also did the same thing with the wind.

In the xls file, all the empty lignes are fill as 'NULL' in the recorded data and archives.

Normally, I get only one instance of weewx. Weewx is running on a raspberry. I already reboot the system. Do you know how see all programs are running?

[lionel...@free.fr]

All the days have 95 records excepted when the external sensor have no more battery or the link between sensor en the internal station.

I get this issue when I use the 3.6.0 version or above and since I clean the archive_xxxxxx because I had some bad recorded values. When I did it I remove all the statics data from July to now and then send the wee_database weewx.conf --backfill-daily and I also tested by restart the weewx application directly. I always get this error.

 As here we don't have any rain or snow on the mountains since end of November, today I  am not able to say if the error is present during normal operation....

[Andrew Milner]

I'm leaving this to those who know your specific console.  It seems as though the rain is being counted and totalled in two places somehow - perhaps within the driver and also when the archive record is created.  I'll leave it to Matthew and Tom and those more knowledgeable than me.  The counting difference will certainly be being caused by the Nulls - which won't be counted. 

ps -A  I think is the command you are looking for

[Andrew Milner]

Did you delete-daily before you backfilled-daily??

[Thomas Keffer]

these should match:

SELECT sum FROM archive_day_rain WHERE dateTime = 1480114800;

SELECT SUM(rain) FROM archive WHERE dateTime > 1480114800 AND dateTime <= (1480114800 + 86400);

If they don't, then something got corrupted in your daily summaries. Rebuild them using the utility wee_database:

wee_database --drop-daily --backfill-daily

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Andrew Milner]

I knew I should have looked up the specific commands before I posted!!

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Thomas Keffer]

Forgot that you cannot specify more than one "verb" at a time for wee_database. So, to rebuild the daily summaries, you must do two commands:

wee_database --drop-daily

wee_database --backfill-daily

-tk

[lionel...@free.fr]

Hello,

the answer below:

Le jeudi 5 janvier 2017 15:48:32 UTC+1, Tom Keffer a écrit :

these should match:

SELECT sum FROM archive_day_rain WHERE dateTime = 1480114800;

for datetime 1479942000, I get "2.48599999492855"

 

SELECT SUM(rain) FROM archive WHERE dateTime > 1480114800 AND dateTime <= (1480114800 + 86400);

for datetime 1479942000, I get "1.24299999746428"

 

 

If they don't, then something got corrupted in your daily summaries. Rebuild them using the utility wee_database:

wee_database --drop-daily --backfill-daily

I did it with to commands. To rebuild the database took almost 1 hour (almost 9 years of data). Now the 2 lines above gives the same result "1.24299999746428"

Now I will follow the results and If a day it will happen again, I know what to do. 

On another way, What was happen in the archive we don't know. I kept the database if necessary.

Thanks to all for the support. It was an interesting analysis.

rds

Lio

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Thomas Keffer]

If these values are different in the future, there is definitely a problem. Let us know.

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[lionel...@free.fr]

Hello, Ok.

Question : is it possible to rebuild just a part of the database ? This will save time for heavy database.

I think about a possible origin of that issue that I get. I will do it next night and I will give you the result.

rds

[gjr80]

Hi,

No it is not possible, at the moment it is an 'all or nothing' approach. But once Issue #117 (https://github.com/weewx/weewx/issues/117) is addressed yes it will be possible.

Gary