acurite 5 in 1, once or twice a month random amounts of rain (2000mm more or less)

[Marc Ratés]

Hello,

I've been using weewx with acurite 5 in 1 and the big problem is the rain.
Now installed version 3.6.2.

The class or plugin in charge of acurite 5 in 1 inserts randoms inputs of rain, today 2000mm.
It's very annoying because I have to delete by hand the sqlite.db. and I lose the accuracy of the liters.

I've configured the unit in mode 4.


Anyone with same problem?

Thanks.

[mwall]

marc,

look in your log for 'rain counter decrement ignored'

sudo grep "rain counter decrement" /var/log/syslog

do you see one of these messages immediately before each large rainfall amount?

m

[mwall]

On Wednesday, November 30, 2016 at 3:01:36 PM UTC-5, Marc Ratés wrote:

The class or plugin in charge of acurite 5 in 1 inserts randoms inputs of rain, today 2000mm.
It's very annoying because I have to delete by hand the sqlite.db. and I lose the accuracy of the liters.

i forgot to mention the weewx quality control (QC).

in weewx.conf, do something like this in your QC section:

[StdQC]
    [[MinMax]]
        ...
        rain = 0, 1000, mm

this will not fix the root cause (spurious data from the acurite station), but it will prevent obviously bad values from getting into the database.

m

[Chris Swanda]

mwall,

QC has never prevented Rain nor RainRate spikes being written to my database.  I typically will get 30 inches of rain (one record) and 99 inches of rain (four records) written.  

I typically just null them out and restart.  

[Chris Swanda]

I mean to say 

99 inches of RainRate (four records) written

.  

[Andrew Milner]

What limits have you got set in QC for rain and rainrate such that the spikes are not detected and prevented?

[eldruida]

I'll check qc, didn't know

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/qCTkcG_8Nbs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Milner]

Well the world record for rainfall is apparently 38mm in one minute ( = 190mm if a for a 5 minute archive period) and the world record for an hour is 300mm in 42 minutes - so that should let you set some sensible limits!!

[eldruida]

thank you,  now waiting it happens again. I didn't have debug enabled....

[Andrew Milner]

I have just seen that my weewx.conf has rain set as 0 to 60 inch - and I now think that 0 to 6 inch is more appropriate, so I am changing my QC values now!!

[Chris Swanda]

[StdQC]

    [[MinMax]]

      ......

        rainRate = 0, 10, inch_per_hour

        rain = 0, 10, inch

Since we sometimes has massive rain storms here in the midwest, I keep it like this.

Back in September we received almost a whole month's of rain in about 3 hours.  :)

But, randomly about every two weeks, I will get a spike of anywhere between 30 in/hr to 99 in/hr (four records written to the DB every 300 seconds)

As luck would have it, I have a backup copy of my DB, but I had already nulled out the records... but here it is for example.

RainRate:

1478168400|2016-11-03 05:20:00|0.0

1478168700|2016-11-03 05:25:00|

1478169000|2016-11-03 05:30:00|

1478169300|2016-11-03 05:35:00|

1478169600|2016-11-03 05:40:00|

1478169900|2016-11-03 05:45:00|0.0

Rain:

1478168400|2016-11-03 05:20:00|0.0

1478168700|2016-11-03 05:25:00|

1478169000|2016-11-03 05:30:00|0.0

Just like clockwork, four records for RainRate that are erroneous and one record for rain.

Next time it happens, I'll grep for logs and get a copy of the database entries.

[Chris Swanda]

As luck would have it, the mysterious rain storm error popped up today....

sqlite> select dateTime, datetime(dateTime,'unixepoch','localtime'), rain from archive;

1480709100|2016-12-02 14:05:00|0.0

1480709400|2016-12-02 14:10:00|0.0

1480709700|2016-12-02 14:15:00|0.0

1480710000|2016-12-02 14:20:00|0.0

1480710300|2016-12-02 14:25:00|0.159999999836802

1480710600|2016-12-02 14:30:00|0.0

1480710900|2016-12-02 14:35:00|0.0

1480711200|2016-12-02 14:40:00|0.0

1480711500|2016-12-02 14:45:00|0.0

1480711800|2016-12-02 14:50:00|0.0

1480712100|2016-12-02 14:55:00|0.0

select dateTime, datetime(dateTime,'unixepoch','localtime'), rainRate from archive;

1480709100|2016-12-02 14:05:00|0.0

1480709400|2016-12-02 14:10:00|0.0

1480709700|2016-12-02 14:15:00|0.0

1480710000|2016-12-02 14:20:00|0.0

1480710300|2016-12-02 14:25:00|0.274285714005946

1480710600|2016-12-02 14:30:00|0.639999999347208

1480710900|2016-12-02 14:35:00|0.639999999347208

1480711200|2016-12-02 14:40:00|0.365714285341262

1480711500|2016-12-02 14:45:00|0.0

1480711800|2016-12-02 14:50:00|0.0

1480712100|2016-12-02 14:55:00|0.0

This time it wasn't as severe as it normally is.  

I just nulled out the values, dropped daily, backfilled and re-started weewx.  

[mwall]

On Friday, December 2, 2016 at 4:58:48 PM UTC-5, Chris Swanda wrote:

As luck would have it, the mysterious rain storm error popped up today....

chris,

what does the log say from about 5 minutes before until about 5 minutes after the rain event?

as far as data collection goes, the rainRate is not relevant - it is a derived calculation, so if you fix rain, you fix rainRate.

m

[Chris Swanda]

I was in a hurry at the office and forgot about logs.... I should know better than that. 

Dec  2 14:22:11 wxpi weewx[1109]: acurite: rain counter decrement ignored: new: 105.1052 old: 105.5116

This is the only anomaly.  This was in syslog.  Nothing seen in messages.

[Chris Swanda]

more details.

chris@wxpi:/var/log $ cat message* |grep rain

Dec  2 14:22:11 wxpi weewx[1109]: acurite: rain counter decrement ignored: new: 105.1052 old: 105.5116

Nov 22 11:13:08 wxpi weewx[1109]: acurite: rain counter decrement ignored: new: 105.029 old: 106.045

Nov 15 07:24:17 wxpi weewx[22745]: acurite: rain counter decrement ignored: new: 80.518 old: 105.029

Nov 15 07:30:19 wxpi weewx[22745]: engine: 2016-11-15 07:30:00 CST (1479216600) Archive value 'rainRate' 38.5999999606 outside limits (0.0, 10.0)

Nov 15 07:35:26 wxpi weewx[22745]: engine: 2016-11-15 07:35:00 CST (1479216900) Archive value 'rainRate' 38.5999999606 outside limits (0.0, 10.0)

Nov 15 07:40:15 wxpi weewx[22745]: engine: 2016-11-15 07:40:00 CST (1479217200) Archive value 'rainRate' 36.7619047244 outside limits (0.0, 10.0)

Nov 18 20:53:10 wxpi weewx[1109]: engine: 2016-11-18 20:53:10 CST (1479523990) LOOP value 'rain' 32.2799999671 outside limits (0.0, 10.0)

Nov 18 20:53:46 wxpi weewx[1109]: acurite: rain counter decrement ignored: new: 105.029 old: 187.0202

Nov 19 11:38:17 wxpi weewx[1109]: acurite: rain counter decrement ignored: new: 1.2192 old: 105.029

Nov 19 11:38:53 wxpi weewx[1109]: engine: 2016-11-19 11:38:53 CST (1479577133) LOOP value 'rain' 40.8699999583 outside limits (0.0, 10.0)

chris@wxpi:/var/log $

[mwall]

chris,

each 'rain counter decrement ignored' is a spurious value for rain.  the acurite console sends a value that looks like a counter wraparound.  the next value ends up looking like rainfall.

it is not clear whether the root cause is a bogus 5n1 unit sending the wrong value, the console borking a value it gets from the 5n1, or something else.  it could even be interference in the RF or USB comms.

this is not uncommon on low-end hardware like acurite.

there is no fix for this - the acurite firmware cannot be updated.  but here are some workarounds:

1) use MinMax in StdQC to eliminate large values.  unfortunately, this will not eliminate spurious small values, as you have seen.  and we have existence proof that the spurious values are not only large.

2) figure out whether the decrement is a real decrement (wraparound) or a spurious value.  you could set debug=1 and [Acurite].debug_raw=1 to capture all of the raw data.  when this happens, record the raw R1 packets.  capture a bunch of them then see if there is a pattern to the spurious data that we could recognize.

3) if we see a counter decrement, retain the value of the decrement.  at the next rain increment, ignore the next rain increment if it is equal to the decrement.  this approach could work, as long as the bogus values are consistent, i.e., each spurious decrement is followed at some point by a spurious increment of the same value.  to verify that this approach would work, we need a list of the 'rain counter decrement' log entries, plus the actual values for 'rain' in the database for the same time period.

unfortunately, (or perhaps fortunately?) the acurite station i have for testing (a 01036) has only ever produced spurious rain values when i've been abusing its usb interface, and that is far from repeatable.  so any log/database correlations that you or anyone else seeing spurious rain on acurite hardware can collect would help.

if we can verify that any spurious decrement is followed by a spurious increment of the same size, then we can fix the driver to recognize that pattern.

m

[Chris Swanda]

Makes sense.  I am well aware that this Acurite is low end hardware, and honestly, I am pleasantly surprised how well it has worked over the past year with Weewx.  I figured I would have more issues than this.  I guess it is great software development and community involvement ... :)  I have some torrids laying in a junk box somewhere that I could use to lessen the chances of RFI on my USB cable.  When I'm down in my basement, maybe I'll poke around for them.  But, it is probably just the Acurite console spitting out some bogus values every once in a while.  I would like to see if it is random bogeys or some repeatable pattern.  

If you would like for me to capture raw data to try to see if there is a spurious pattern, I'd be glad to do so to further help the Weewx project.  

I've applied debug logging along with [Acurite].debug_raw=1 and if it pops up again (two weeks?) we'll go from there.  

Now, to monitor logs today and adjust my logrotate just in case my logs fill up with extra logging turned on. 

Let me know if there is something else you would like me to do.

Thanks.

[mwall]

On Saturday, December 3, 2016 at 7:41:41 AM UTC-5, Chris Swanda wrote:

If you would like for me to capture raw data to try to see if there is a spurious pattern, I'd be glad to do so to further help the Weewx project.  

I've applied debug logging along with [Acurite].debug_raw=1 and if it pops up again (two weeks?) we'll go from there.  

thanks chris.  just to be clear, [Acurite].debug_raw = 1 is my shorthand for:

[Acurite]
    ...
    debug_raw = 1

also, if you are running on an sd card you should log to a reasonably sized tmpfs - debug_raw will cause a lot of disk writes.

i think the most helpful info will be to see if the next increment after a decrement is the same size as the decrement.  having the corresponding R1 data for those events will let us see if there is a lower-level pattern to the behavior.

m

[Chris Swanda]

 just to be clear, [Acurite].debug_raw = 1 is my shorthand for:

[Acurite]
    ...
    debug_raw = 1

Thanks for the clarification.  I was wondering where to put this.  I made the change and bounced weewx.

I'll monitor my logs to make sure I don't fill up my SD in the middle to day today.  If nothing else, I'll change log rotate to be size based versus time based and adjust a few things.

[Chris Swanda]

Just an update...

First I took this as an opportunity to run everything on my RPi off of RAM (tmpfs)...

Secondly, I have yet to run into my mysterious rain spike since then, whereas it would generally pop up every two weeks on average.  

Just updating here, so I can find this post again... I might turn off debugging and revert things back to normal.

[epe...@gmail.com]

I've been seeing this on my AcuRite as well.  The values aren't displayed on the console either.  I'd just been marking it up to either deer or dogs tripping over the guy wires stabilizing my mounting pole.

Is the QC setting is applied to each reading and rain total is emitted every 36 seconds, so the mixmax threshold should be what I feel would be reasonable for rainfall in a 36 second window? Or is this applied in the 5 minute loop so must allow for larger values that could accumulate in 5 minutes?

Looking at my archive data I ran:

select min(rain), max(rain), avg(rain) from archive where rain > 0 and rain < .5

returning

min(rain) max(rain) avg(rain)

0.00999999999999907 0.47 0.0251541850220265

Granted we've not had spring rains yet to compare with, but I'm thinking a upper boundary of .1 rain in a 36 second window.

 

Thanks

Chad