NOAA data question

[Charles Ross]

Hi all,

Yesterday two pieces of bad temperature and dewpoint data made it's way into the weewx.sdb archive database.   I removed the bad data but my NOAA September and yearly files are still corrupt this morning.

Can someone please guide me on how to fix the reports so I can have correct NOAA reports?

Thank you for taking the time to help,

Chuck Ross 

[Thomas Keffer]

Did you follow the directions in the Wiki entry, Cleaning up old 'bad' data?

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Philip Kutzenco]

I'll let those with more experience and knowledge weigh in, if this is more to this. But have you followed the steps in this link https://github.com/weewx/weewx/wiki/Cleaning-up-old-'bad'-data? It worked for me. You shouldn't need to delete the existing NOAA reports as the latest monthly and yearly files should be re-created automatically (I had fixed older data and deleted the relevant monthly report to force re-creation). I think that dropping and rebuilding the old daily-summary tables is required.

Phil

[Charles Ross]

Yes I did, I followed that web page exactly yesterday.

[Philip Kutzenco]

What skin are you using? I am using the belchertown skin. It copies the NOAA reports and displays them from the new location. I needed to delete the copy it made to force it to re-copy it before it would show the updated report. Again, my corrected data was from an older monthly NOAA report. I would go and check the actual NOAA report that the Standard report creates in /home/weewx/public_html/NOAA. If that's wrong, I'm sure Thomas will have a better suggestion.

Phil

[Thomas Keffer]

OK. Try this.

Check the date & time on the NOAA HTML pages. How old are they? Make sure that they have, in fact, been recently generated. If they are old, delete them, and WeeWX will regenerate. On the other hand, if they have been recently generated, then the problem lies deeper.

But, to do that will require more information, such as what values you are trying to fix and their timestamps. We will need to check both the archive and daily summaries for the values.

-tk

[Charles Ross]

Thanks all for the help.

I started from scratch this morning.   I stop weewx and run  'wee_database weewx.conf --drop-daily'  I am getting the following feedback right at the start:

Traceback (most recent call last):
  File "/usr/bin/wee_database", line 793, in <module>
    main()
  File "/usr/bin/wee_database", line 133, in main
    config_path, config_dict = weecfg.read_config(options.config_path, args)
  File "/usr/share/weewx/weecfg/__init__.py", line 223, in read_config
    config_dict = configobj.ConfigObj(config_path, file_error=True)
  File "/usr/lib/python2.7/dist-packages/configobj.py", line 1230, in __init__
    self._load(infile, configspec)
  File "/usr/lib/python2.7/dist-packages/configobj.py", line 1320, in _load
    raise error
configobj.ConfigObjError: Parsing failed with several errors.
First error at line 1.

-Chuck

[Thomas Keffer]

You have an error in your weewx.conf file. The error message suggests it's in the very first line.

-tk

[Charles Ross]

Weird, I have no other issues, weewx stops, starts, performs fine.  I see no error messages in the logs, only when I run the wee_database command do I have problems.

I'll keep digging.

CR

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/--ofTjRbzII/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Thomas Keffer]

Then perhaps you are using a different weewx.conf when running wee_database, versus weewxd. Try an explicit path. For example (assuming a setup.py install):

wee_database /home/weewx/weewx.conf

-tk

[Charles Ross]

I ran wee_database with the  --update token and that cleaned things up.   Thanks for all the help and suggestions.  Good learning experience.

Chuck

[Thomas Keffer]

That is odd. I don't see why that would have anything to do with anything. Could you include the log from the run?

-tk

[Charles Ross]

Sep 26 10:57:27 raspberrypi wee_database[3358]: Preparing Interval
Weighting Fix...
Sep 26 10:57:27 raspberrypi wee_database[3358]: manager: Daily summary
version is 1.0
Sep 26 10:57:27 raspberrypi wee_database[3358]: manager: Daily summary
version is 1.0
Sep 26 10:57:27 raspberrypi wee_database[3358]: Daily summary tables
are at version 1.0
Sep 26 10:57:27 raspberrypi wee_database[3358]: Calculating interval weights...
Sep 26 10:57:27 raspberrypi wee_database[3358]: intervalweighting:
Using database binding 'wx_binding', which is bound to database
'weewx.sdb'.
Sep 26 10:57:27 raspberrypi wee_database[3358]: intervalweighting:
Database transactions will use 100 days of data.
Sep 26 10:57:27 raspberrypi wee_database[3358]: intervalweighting:
Checking table 'archive' for multiple 'interval' values per day...
Sep 26 10:57:31 raspberrypi wee_database[3358]: intervalweighting:
Multiple distinct 'interval' values found for at least one archive
day.
Sep 26 10:57:31 raspberrypi wee_database[3358]: intervalweighting:
Interval Weighting Fix will be applied by dropping and rebuilding
daily summaries.
Sep 26 10:57:31 raspberrypi wee_database[3358]: manager: Dropping
daily summary tables from 'weewx.sdb' ...
Sep 26 10:57:31 raspberrypi wee_database[3358]: manager: Drop
summaries failed for database 'weewx.sdb': attempt to write a readonly
database
Sep 26 10:57:31 raspberrypi wee_database[3358]: manager: Daily summary
version is 1.0
Sep 26 10:57:31 raspberrypi wee_database[3358]: manager: Starting
backfill of daily summaries
Sep 26 10:57:31 raspberrypi wee_database[3358]: Interval Weighting Fix
completed in 3.45 seconds.
Sep 26 10:57:31 raspberrypi wee_database[3358]: Preparing Maximum
windSpeed Fix...
Sep 26 10:57:31 raspberrypi wee_database[3358]: manager: Daily summary
version is 1.0
Sep 26 10:57:31 raspberrypi wee_database[3358]: maxwindspeed: Using
database binding 'wx_binding', which is bound to database 'weewx.sdb'.
Sep 26 10:57:31 raspberrypi wee_database[3358]: maxwindspeed: Database
transactions will use 100 days of data.
Sep 26 10:57:31 raspberrypi wee_database[3358]: maxwindspeed: Applying
Maximum windSpeed Fix...
Sep 26 10:57:51 raspberrypi wee_database[3360]: Preparing Interval
Weighting Fix...
Sep 26 10:57:51 raspberrypi wee_database[3360]: manager: Daily summary
version is 1.0
Sep 26 10:57:51 raspberrypi wee_database[3360]: manager: Daily summary
version is 1.0
Sep 26 10:57:51 raspberrypi wee_database[3360]: Daily summary tables
are at version 1.0
Sep 26 10:57:51 raspberrypi wee_database[3360]: Calculating interval weights...
Sep 26 10:57:51 raspberrypi wee_database[3360]: intervalweighting:
Using database binding 'wx_binding', which is bound to database
'weewx.sdb'.
Sep 26 10:57:51 raspberrypi wee_database[3360]: intervalweighting:
Database transactions will use 100 days of data.
Sep 26 10:57:51 raspberrypi wee_database[3360]: intervalweighting:
Checking table 'archive' for multiple 'interval' values per day...
Sep 26 10:57:54 raspberrypi wee_database[3360]: intervalweighting:
Multiple distinct 'interval' values found for at least one archive
day.
Sep 26 10:57:54 raspberrypi wee_database[3360]: intervalweighting:
Interval Weighting Fix will be applied by dropping and rebuilding
daily summaries.
Sep 26 10:57:54 raspberrypi wee_database[3360]: manager: Dropping
daily summary tables from 'weewx.sdb' ...
Sep 26 10:58:04 raspberrypi wee_database[3360]: manager: Dropped daily
summary tables from database 'weewx.sdb'
Sep 26 10:58:04 raspberrypi wee_database[3360]: manager: Created daily
summary tables
Sep 26 10:58:04 raspberrypi wee_database[3360]: manager: Daily summary
version is 2.0
Sep 26 10:58:04 raspberrypi wee_database[3360]: manager: Starting
backfill of daily summaries
Sep 26 11:17:01 raspberrypi /USR/SBIN/CRON[3365]: (root) CMD (   cd /
&& run-parts --report /etc/cron.hourly)
Sep 26 11:19:37 raspberrypi wee_database[3360]: manager: Processed
644068 records to backfill 2244 day summaries in 1292.06 seconds
Sep 26 11:19:37 raspberrypi wee_database[3360]: Interval Weighting Fix
completed in 1305.68 seconds.
Sep 26 11:19:37 raspberrypi wee_database[3360]: Preparing Maximum
windSpeed Fix...
Sep 26 11:19:37 raspberrypi wee_database[3360]: manager: Daily summary
version is 2.0
Sep 26 11:19:37 raspberrypi wee_database[3360]: maxwindspeed: Using
database binding 'wx_binding', which is bound to database 'weewx.sdb'.
Sep 26 11:19:37 raspberrypi wee_database[3360]: maxwindspeed: Database
transactions will use 100 days of data.
Sep 26 11:19:37 raspberrypi wee_database[3360]: maxwindspeed: Applying
Maximum windSpeed Fix...
Sep 26 11:19:45 raspberrypi wee_database[3360]: maxwindspeed: Maximum
windSpeed calculated for 2244 days in 8.50 seconds.
Sep 26 11:19:45 raspberrypi wee_database[3360]: Maximum windSpeed Fix
completed in 8.52 seconds

[Thomas Keffer]

I suspect that you did not rebuild the daily summaries before. If you had, the summaries would be at Version 2, not Version 1 (as shown in the entry at 10:57:51).

In any case, they have now been rebuilt, so your bad data has been removed from the summaries.

-tk

[Charles Ross]

Yup, it’s been a while since I’ve worked on this, it’s very stable. Shows what a great program Weewx is!!