Using Temporal Analysis for Temperature QC?

[Jared Marquis]

I've noticed a few issues with the occasional bad temperature/dew point measurement.  Specifically, this week, my high temperature is: 101.0°F at 01:42:00 AM (Monday).  While unusual, not impossible.  That said, it happened overnight, and was surrounded by temperatures in the 50s.  This is obviously an errant reading.

I am currently recording information every 5min.  How can I look at a time series of, say 5-10 temperature readings, to remove or flag suspect data?  Also, I'd like to be able to restrict tendencies to do the same (e.g., +/-25* change in 5min is suspect and needs to examined with time series when more data comes in).

Is there any flag variable in the observation objects or would this need to be added?  Is it possible to examine other times with a qc function?

Thanks!

[Thomas Keffer]

Hello, Jared

Unfortunately, StdQC can only reject values that are outside of a [min, max] range.

However, this would not be a very hard service to write. Something like (NOT TESTED):

import time

from weewx.engine import StdService

class MyQC(StdService):
    """Check for time and temperature out of bounds """
   
    def __init__(self, engine, config_dict):
        super(MyQC, self).__init__(engine, config_dict)

        self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)

    def new_archive_record(self, event):
        """Check for a high temperature at night"""

        # Make sure there

        if 'outTemp' in event.record and event.record['outTemp'] is not None:
            time_tuple = time.localtime(event.record['dateTime'])

            hour = time_tuple.tm_hour

            # This assumes US Customary units. You probably want to check the unit system first!!

            if (hour < 5 or hour > 20) and event.record['outTemp'] > 100:

                event.record['outTemp'] = None
        
This example uses a pretty stupid time / temperature check. You'd probably want to substitute something more sophisticated.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b9af5ef9-66e7-41a6-8892-60f580f80b72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gjr80]

We're not going to see the mythical spike detector resurrected are we? (also https://github.com/weewx/weewx/pull/228)

Gary

On Monday, 24 June 2019 07:14:49 UTC+10, Thomas Keffer wrote:

Hello, Jared

Unfortunately, StdQC can only reject values that are outside of a [min, max] range.

However, this would not be a very hard service to write. Something like (NOT TESTED):

import time

from weewx.engine import StdService

class MyQC(StdService):
    """Check for time and temperature out of bounds """
   
    def __init__(self, engine, config_dict):
        super(MyQC, self).__init__(engine, config_dict)

        self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)

    def new_archive_record(self, event):
        """Check for a high temperature at night"""

        # Make sure there

        if 'outTemp' in event.record and event.record['outTemp'] is not None:
            time_tuple = time.localtime(event.record['dateTime'])

            hour = time_tuple.tm_hour

            # This assumes US Customary units. You probably want to check the unit system first!!

            if (hour < 5 or hour > 20) and event.record['outTemp'] > 100:

                event.record['outTemp'] = None
        
This example uses a pretty stupid time / temperature check. You'd probably want to substitute something more sophisticated.

-tk

On Sun, Jun 23, 2019 at 2:00 PM Jared Marquis <marqui...@gmail.com> wrote:

I've noticed a few issues with the occasional bad temperature/dew point measurement.  Specifically, this week, my high temperature is: 101.0°F at 01:42:00 AM (Monday).  While unusual, not impossible.  That said, it happened overnight, and was surrounded by temperatures in the 50s.  This is obviously an errant reading.

I am currently recording information every 5min.  How can I look at a time series of, say 5-10 temperature readings, to remove or flag suspect data?  Also, I'd like to be able to restrict tendencies to do the same (e.g., +/-25* change in 5min is suspect and needs to examined with time series when more data comes in).

Is there any flag variable in the observation objects or would this need to be added?  Is it possible to examine other times with a qc function?

Thanks!

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Jared Marquis]

Could this not be applied in a semi-similar method to calc_rainRate's previous record look-up?  E.g., psuedocoding, 

Grab last archive value (T,Td,etc)

If abs(last archive value - current value)>some_value:

  add suspect_flag

Then go back and remove suspect_flags based on whether a spike occurred?

archive_vals=[val for val in records[-n:-5]]

if any(archive_vales.suspect_flag):

   inds=where(suspect_flag)

   if abs(archive_vals[inds]-archive_vales[inds+1])<some_value:

      suspect_flag=false

Obviously this requires a suspect_flag added to the sdb.  I have no sql experience, but this should be rather trivial I would think.  Also, this doesn't help issues with consecutive erroneous values, but adding that should be trivial also.

On Sunday, June 23, 2019 at 4:14:49 PM UTC-5, Thomas Keffer wrote:

Hello, Jared

Unfortunately, StdQC can only reject values that are outside of a [min, max] range.

However, this would not be a very hard service to write. Something like (NOT TESTED):

import time

from weewx.engine import StdService

class MyQC(StdService):
    """Check for time and temperature out of bounds """
   
    def __init__(self, engine, config_dict):
        super(MyQC, self).__init__(engine, config_dict)

        self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)

    def new_archive_record(self, event):
        """Check for a high temperature at night"""

        # Make sure there

        if 'outTemp' in event.record and event.record['outTemp'] is not None:
            time_tuple = time.localtime(event.record['dateTime'])

            hour = time_tuple.tm_hour

            # This assumes US Customary units. You probably want to check the unit system first!!

            if (hour < 5 or hour > 20) and event.record['outTemp'] > 100:

                event.record['outTemp'] = None
        
This example uses a pretty stupid time / temperature check. You'd probably want to substitute something more sophisticated.

-tk

On Sun, Jun 23, 2019 at 2:00 PM Jared Marquis <marqui...@gmail.com> wrote:

I've noticed a few issues with the occasional bad temperature/dew point measurement.  Specifically, this week, my high temperature is: 101.0°F at 01:42:00 AM (Monday).  While unusual, not impossible.  That said, it happened overnight, and was surrounded by temperatures in the 50s.  This is obviously an errant reading.

I am currently recording information every 5min.  How can I look at a time series of, say 5-10 temperature readings, to remove or flag suspect data?  Also, I'd like to be able to restrict tendencies to do the same (e.g., +/-25* change in 5min is suspect and needs to examined with time series when more data comes in).

Is there any flag variable in the observation objects or would this need to be added?  Is it possible to examine other times with a qc function?

Thanks!

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Graham Eddy]

could do something like a highly simplified 1st pass filter i.e. have a running mean value and reject items that deviate more than a tolerance in a single interval - this is really QCing something like the first differential (dx/dt) to filter spikes, even if the spikes are within the [min,max] range of the variable

____________

Graham Eddy