Pse, how to erase bad data on weewx.sdb

[f5...@free.fr]

Hello

I just installed today my new weather station : Vantage Pro 2.

It works well and I connected it with my raspberry-pi + weewx with new .conf file.

I was using a WMR200 before.

This evening, I have bad reception of the ISS outside and this created some bad datas : T°C, wind-chill and dew point very low.

http://f5phw.free.fr/weewx/week.html

I would like to erase this bad data.

How can I do please ?

Many thanks.

Phil BERGER

[f5...@free.fr]

I am using weewx v3.2.1

[Thomas Keffer]

See the Wiki, Cleaning up old 'bad' data and see if that answers your question.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[f5...@free.fr]

Hello Thomas

Many thanks for quick answer.

I have installed sqlite3 as explain in the wiki : https://github.com/weewx/weewx/wiki/Cleaning%20up%20old%20'bad'%20data

I tried to enter the following line, but I have an error.

pi@raspiwx ~ $ echo "SELECT * FROM archive WHERE (windSpeed > 100);" | sqlite3 mydb.sdb
Error: near line 1: no such table: archive

Phil

Web page on http://f5phw.free.fr
F5PHW on FACEBOOK : https://www.facebook.com/phil.berger.75

----- Mail original -----
De: "Thomas Keffer" <tke...@gmail.com>
À: "weewx-user" <weewx...@googlegroups.com>
Envoyé: Jeudi 22 Octobre 2015 06:50:54
Objet: Re: [weewx-user] Re: Pse, how to erase bad data on weewx.sdb

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/f8vbRkAyNXY/unsubscribe .
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com .

[gjr80]

Hi,
Is your database really mydb.sdb or is it in fact weewx.sdb as per the title of your post?

Gary

[f5...@free.fr]

Hello Gary

That's a good question !
Sorry, I am a newbie in linux.

Can you tell me please where I can find this database ? Wich folders please ?

Many thanks.

Phil

Web page on http://f5phw.free.fr
F5PHW on FACEBOOK : https://www.facebook.com/phil.berger.75

----- Mail original -----

De: "gjr80" <gjrod...@gmail.com>
À: "weewx-user" <weewx...@googlegroups.com>
Envoyé: Jeudi 22 Octobre 2015 09:53:15

Objet: Re: [weewx-user] Re: Pse, how to erase bad data on weewx.sdb

Hi,
Is your database really mydb.sdb or is it in fact weewx.sdb as per the title of your post?

Gary

[Andrew Milner]

Have you copied the database to mydb.sdb from archive_sqlite.sdb?

[gjr80]

Depends on how you installed weewx. Try /home/weewx/archive or /var/lib/weewx. Unless you changed something you are looking for weewx.sdb.

Gary

[f5...@free.fr]

Gary, Andrew.

Thanks for answers.

1/I have found database : /var/lib/weewx/weewx.sdb

2/I saved it on my PC.

3/I tried like in the wiki with the right name of database : weewx.sdb

pi@raspiwx ~ $ echo "SELECT * FROM archive WHERE (windSpeed > 100);" | sqlite3 weewx.sdb

Error: near line 1: no such table: archive

The same pbm.

4/Andrew.
If I understand, I have to copy my database to a sqlite3 database to correct it ?
And after ? I have to copy the database in sqlite3 to weewx ?
Sorry, I am really a newbie and I don't know how to do this.

Many thanks for help.

Phil

Web page on http://f5phw.free.fr
F5PHW on FACEBOOK : https://www.facebook.com/phil.berger.75

----- Mail original -----

De: "gjr80" <gjrod...@gmail.com>
À: "weewx-user" <weewx...@googlegroups.com>
Envoyé: Jeudi 22 Octobre 2015 10:24:10

Objet: Re: [weewx-user] Re: Pse, how to erase bad data on weewx.sdb

Depends on how you installed weewx. Try /home/weewx/archive or /var/lib/weewx. Unless you changed something you are looking for weewx.sdb.

Gary

[gjr80]

There are 2 approaches you can take: if you are game you can alter the erroneous records on your in use database (ie /var/lib/weewx/weewx.sdb) or the safer way is to make a copy of weewx.sdb and make the changes to the copy. Once you are happy with your changes you can then stand the old database aside and copy the amended database in its place. You have a vp2 so, assuming you have a Davis logger attached, it will have saved any records that have been generated while you were doing all this and weewx will catch up the missing records.

Steps to fix your situation:
- copy weewx.sdb to a know location eg /var/tmp
- run the command you mentioned using /var/tmp/weewx.sdb (or wherever you copied weewx.sdb) in place of weewx.sdb
- make your changes as per the wiki entry that Tom linked
- once you are satisfied with your changes, stop weewx.
- move /var/lib/weewx/weewx.sdb aside by renaming it, say weewx_orig.sdb
- copy your amended database to /var/lib/weewx
- start weewx, if you watch the log it should catch in any missed archive records

Gary

[Andrew Milner]

The steps should be something like:

a) find the SQLite database (.sdb) file and copy it to somewhere safe.  You found it in /var/lib/weewx/weewx.sdb, and you say you copied it to your PC - I think the wiki copies it to somewhere else but that is not a problem.

b) issue the echo commands as in the wiki - you may need to specify the full path to the database

this is where you currently are getting your error

c) When you are happy you have finished.  If you think you have made a mistake just copy the original SQLite (.sdb) database file back again and start over.

d) Alternatively - specify the full path to the COPY of the database, and make the edits to the copy instead of the original.

 

[f5...@free.fr]

Hello

Thanks for your help. It was not easy.

I copied weewx.sdb to /var/tmp/ (with winSCP)
I modified it with commands in https://github.com/weewx/weewx/wiki/Cleaning%20up%20old%20'bad'%20data
But when I wanted to copy it back to /var/lib/weewx/, it was impossible due to bad chmod.
I did something wrong (tried to work as root) and it was impossible to use weewx.

So I decided to use an old backup of weewx made 10 days agos.
Now the -40°C (measured yesterday) disapeared. Great. But not the good way to do it.

I found tonight I have on http://f5phw.free.fr/weewx/week.html radiations, UV and ET measures.
I don't have any sensor for ET, UV and radiations.

So, I decided to delete them

I made a new backup, install sqlite3.
I used (and learned how to) the raspi as root and I used the following sentences :

echo "SELECT * FROM archive WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

I found lines as predicted, then :

echo "UPDATE archive SET radiation=NULL WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

Then I made again to verify

echo "SELECT * FROM archive WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

And I found nothing. So it works.

I did the same for ET and UV.

I made a reboot of my rpi and wait.

On http://f5phw.free.fr/weewx/week.html
The measures for ET, UV and radiations are still here.
May be I have forgotten something.

The last question : can you give the right sentence to copy weewx.sdb from /var/tmp/ to /var/lib/weewx/ ?
And the same question to move the file ?

Many thanks again for your help and sorry for bad english if you find errors.

Phil






Web page on http://f5phw.free.fr
F5PHW on FACEBOOK : https://www.facebook.com/phil.berger.75

----- Mail original -----

De: "Andrew Milner" <andrew.s...@gmail.com>
À: weewx...@googlegroups.com
Envoyé: Jeudi 22 Octobre 2015 11:46:53

[gjr80]

Phil,

Comments/answers below:

I copied weewx.sdb to /var/tmp/   (with winSCP)
I modified it with commands in https://github.com/weewx/weewx/wiki/Cleaning%20up%20old%20'bad'%20data
But when I wanted to copy it back to /var/lib/weewx/, it was impossible due to bad chmod.
I did something wrong (tried to work as root) and it was impossible to use weewx.

Assuming you were using winSCP the user you were logged in as under winSCP probably did not have write permission for /var/lib/weewx. A few possible solutions (1) under winSCP log in as as a user who does have permission, this may or may not mean using root which is probably not a good idea, if in fact your system will allow it (2) change the permissions for /var/lib/weewx so that the user account you were using does have permission or (3) open a shell into your RPi using something like putty and then run the command to copy the file using sudo so that you have temporary previleged access. I would be doing (3).

Not sure why it was 'impossible to use weewx', but I would be wary of doing too much as root as you can easily screw up your system.

I found tonight I have on http://f5phw.free.fr/weewx/week.html radiations, UV and ET measures.
I don't have any sensor for ET, UV and radiations.

So, I decided to delete them

I made a new backup, install sqlite3.
I used (and learned how to) the raspi as root and I used the following sentences :

echo "SELECT * FROM archive WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

I found lines as predicted, then :

echo "UPDATE archive SET radiation=NULL WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

Then I made again to verify

echo "SELECT * FROM archive WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

And I found nothing. So it works.

I did the same for ET and UV.

I made a reboot of my rpi and wait.

On http://f5phw.free.fr/weewx/week.html
The measures for ET, UV and radiations are still here.
May be I have forgotten something.

A couple of things here. Did you rebuild the daily summaries after you modified the radiation/UV/ET fields? Your weekly page shows high and low values for these so that indicates maybe you did not (those values come from the daily summaries). You need to stop weewx, use wee_database (see Customizing the database) to drop your daily summaries then start weewx and the daily summaries will be automatically rebuilt during weewx startup. The plots are generated at different frequeuncies; daily plots are generated every archive period, weekly every hour, monthly every 3 hours and yearly plots every 24 hours. So even once you have removed any bad data the plots will remain as is until they are generated. The way to force them to be updated is to delete them and any that are missing will be generated on the next archive period.

The last question : can you give the right sentence to copy weewx.sdb from /var/tmp/ to /var/lib/weewx/   ?
And the same question to move the file ?

 The command to copy the file is cp /var/tmp/weewx.sdb /var/lib/weewx/weewx.sdb though if you have permissions issues you are best using sudo cp /var/tmp/weewx.sdb /var/lib/weewx/weewx.sdb. you may or may not need to install sudo depending on your system. To move the file mv /var/tmp/weewx.sdb /var/lib/weewx/weewx.sdb (using sudo if necessary).

I would very strongly suggest you do some sort of linux intro course to learn your way around the basics as almost certainly you will have issues come up when running weewx, whether they be caused by weewx or the system. There are many courses around, Vince has mentioned this one a few times in the weewx forums (though I have never looked at it so can vouch for its usefulness).

Gary

[f5...@free.fr]

Hello Gary

Yessss !

It works.

I have forgotten to rebuild the database.

This is what I did as root command :

wee_database --reconfigure

This created a new file weewx.sdb_new

I renamed weewx.sdb as weewx_old.sdb
I renamed weewx.sdb_new as weewx.sdb

Restarted weewx and it is ok.

Gary, Andrew and Thomes, many many many many thanks for your help.

Gary, I will try to learn more on basics linux commands.

All the best from France and thanks again.

Phil

Web page on http://f5phw.free.fr
F5PHW on FACEBOOK : https://www.facebook.com/phil.berger.75

----- Mail original -----
De: "gjr80" <gjrod...@gmail.com>
À: "weewx-user" <weewx...@googlegroups.com>

Envoyé: Vendredi 23 Octobre 2015 03:42:23

Objet: Re: [weewx-user] Re: Pse, how to erase bad data on weewx.sdb

Phil,

Comments/answers below:



I copied weewx.sdb to /var/tmp/ (with winSCP)
I modified it with commands in https://github.com/weewx/weewx/wiki/Cleaning%20up%20old%20'bad'%20data
But when I wanted to copy it back to /var/lib/weewx/, it was impossible due to bad chmod.
I did something wrong (tried to work as root) and it was impossible to use weewx.

Assuming you were using winSCP the user you were logged in as under winSCP probably did not have write permission for /var/lib/weewx . A few possible solutions (1) under winSCP log in as as a user who does have permission, this may or may not mean using root which is probably not a good idea, if in fact your system will allow it (2) change the permissions for /var/lib/weewx so that the user account you were using does have permission or (3) open a shell into your RPi using something like putty and then run the command to copy the file using sudo so that you have temporary previleged access. I would be doing (3).

Not sure why it was 'impossible to use weewx', but I would be wary of doing too much as root as you can easily screw up your system.



I found tonight I have on http://f5phw.free.fr/weewx/week.html radiations, UV and ET measures.
I don't have any sensor for ET, UV and radiations.

So, I decided to delete them

I made a new backup, install sqlite3.
I used (and learned how to) the raspi as root and I used the following sentences :

echo "SELECT * FROM archive WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

I found lines as predicted, then :

echo "UPDATE archive SET radiation=NULL WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

Then I made again to verify

echo "SELECT * FROM archive WHERE (radiation > 0);" | sqlite3 /var/lib/weewx/weewx.sdb

And I found nothing. So it works.

I did the same for ET and UV.

I made a reboot of my rpi and wait.

On http://f5phw.free.fr/weewx/week.html
The measures for ET, UV and radiations are still here.
May be I have forgotten something.

A couple of things here. Did you rebuild the daily summaries after you modified the radiation/UV/ET fields? Your weekly page shows high and low values for these so that indicates maybe you did not (those values come from the daily summaries). You need to stop weewx, use wee_database (see Customizing the database ) to drop your daily summaries then start weewx and the daily summaries will be automatically rebuilt during weewx startup. The plots are generated at different frequeuncies; daily plots are generated every archive period, weekly every hour, monthly every 3 hours and yearly plots every 24 hours. So even once you have removed any bad data the plots will remain as is until they are generated. The way to force them to be updated is to delete them and any that are missing will be generated on the next archive period.

The last question : can you give the right sentence to copy weewx.sdb from /var/tmp/ to /var/lib/weewx/ ?
And the same question to move the file ?


The command to copy the file is cp /var/tmp/weewx.sdb /var/lib/weewx/weewx.sdb though if you have permissions issues you are best using sudo cp /var/tmp/weewx.sdb /var/lib/weewx/weewx.sdb. you may or may not need to install sudo depending on your system. To move the file mv /var/tmp/weewx.sdb /var/lib/weewx/weewx.sdb (using sudo if necessary).

I would very strongly suggest you do some sort of linux intro course to learn your way around the basics as almost certainly you will have issues come up when running weewx, whether they be caused by weewx or the system. There are many courses around, Vince has mentioned this one a few times in the weewx forums (though I have never looked at it so can vouch for its usefulness).

Gary

[gjr80]

Good to hear it is working Phil, using --reconfigure was the long way round though, stopping weewx, using wee_database --drop-daily followed by starting weewx would have achieved the same result without the need to rename databases etc.

Gary