How to import archives from vantage data logger

[Praveen Chandrasekaran]

Hi,

For some reason my pi went down yesterday morning for 5 hours. I tried rebuilding database using wee_database but to no avail. How do I import data from data logger to weewx so that I can use wunderfixer then to publish again to wu?

Regards,
Praveen

[Thomas Keffer]

This should happen automatically, but it's hard to say what the problem is without a log. See Help! Posting to weeWX. 

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Praveen Chandrasekaran]

Thanks. Will post with all details from next time. For now I did a wee_device --dump and that populated the sql database after which I could update WU with wunderfixer (the summary still has not got updated in WU as per updated records, I can see updated records in graphs and table - I believe that is an issue at WU end and I have filed a bug report with them for the same). Not sure why weewx did not populate the sql on its own though without forcing through wee_device --dump.

On Sunday, 28 January 2018 23:22:40 UTC+5:30, Tom Keffer wrote:

This should happen automatically, but it's hard to say what the problem is without a log. See Help! Posting to weeWX. 

-tk

On Sun, Jan 28, 2018 at 10:15 AM, Praveen Chandrasekaran <pravee...@gmail.com> wrote:

Hi,

For some reason my pi went down yesterday morning for 5 hours. I tried rebuilding database using wee_database but to no avail. How do I import data from data logger to weewx so that I can use wunderfixer then to publish again to wu?

Regards,
Praveen

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Andrew Milner]

The log would have shed light on any problems. 

You should not have needed to use any utility at all - what made you decide to use wee_database??. 

If you had done nothing weewx would have retrieved records from the logger from after the last archive record in its database automatically - this sometimes takes some time.  On a RPi you should either add a RTC module or completely disable the fake hw clock to ensure weewx does not start until it has got a correct time from the NTP servers to ensure correct automatic operations which are reliant on accurate timestamps (such as recovery from a power outage).

The log is always the key - and without a log everything becomes guesswork and speculation whilst with the log we can be talking actual facts!!!!

[Praveen Chandrasekaran]

One additional detail - my pi is read only mode. On abrupt power down the latest SQL database may not have been stored. Not sure if that messed up anything though I believe the pi should have still rebuilt the records. 

Right now I am using ntp with fake hw clock removed, I am planning to put in a RTC soon. 

Will dig more in the log sometime and also post the log here. 

[Praveen Chandrasekaran]

From weewx user guide:

However, if the internal memory gets garbled, the console will immediately return archives in the past, and so it looks like the timestamps have decreased in value and so weeWX figures that is it: there is no more data. 

I believe this is what happened. Previously my weewx would start up on powerup with times in 2016 and corrupt the data logger memory. Now I have modified it to start only if time is in 2018. But I believe weewx stopped filling missing archives as it found decreasing timestamps. Clearing my logger memory (anyway it has only 2 weeks worth contents) should fix this issue for good I guess. 

The catchup messages come in log only if wee_debug is set to 1 I believe? Anyway will look at the log again tonight.

What is still a mystery is why did the pi go down and why did it need a power on cycle to start up again! 

[Andrew Milner]

The log of the start-up is what is needed.

You say the database was empty when you started weewx??  Is that correct??  If it was totally empty then I don't think weewx tries to fill in any gaps - as there are no gaps!! Tom can say if weewx reads history records when weewx starts and the database is empty.

[Praveen Chandrasekaran]

No idea if it was empty or not. I am not backing it up to an external USB ( I know that is very bad!) so I am not sure what would be it contents in case of abrupt power down and power up. My setup is only since 2 weeks and I am still learning. Need to do some backup plan. 

[Andrew Milner]

I would suggest you have a proper plan on how to cope with a read only rpi - or maybe even just run the rpi and weewx in a standard configuration until you know how everything works - especially if you do not have a RTC installed.  Also make sure that you are not running the latest NTP software which I believe has some form of buffering similar to fake hwclock - but not quite as bad!!  I saw someone mention this in a recent thread in this forum.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/XYmglRBbWrg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.

[Praveen Chandrasekaran]

I think backing up to a USB stick say once or twice in a day should do. Glen has instructions on how to set it up and I do have a lot of USB sticks lying around. That would ensure that on next powerup I dont have to restore more than a day's worth of data.

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Andrew Milner]

Your decision!!  Personally I prefer my rpi to have a hdd when it is being used as a 'real' computer, and a decent power supply helps with usb issues.  The rpi can be finicky and temperamental where power is concerned.  Everyone has their own 'solution' though, so i do not argue!!!

[Praveen Chandrasekaran]

I am infact on lookout for a good USB powerbank to use as UPS. I am using Xiaomi which supports pass through charging but glitches during transition. I read that EasyAcc is good but it is turning out to be as costly as the pi itself! :)

[Glenn McKechnie]

On 29 January 2018 at 22:24, Praveen Chandrasekaran <pravee...@gmail.com> wrote:

I think backing up to a USB stick say once or twice in a day should do. Glen has instructions on how to set it up and I do have a lot of USB sticks lying around. That would ensure that on next powerup I dont have to restore more than a day's worth of data.

Or even better yet just forget about the memory database and write the database directly to a mounted drive. It's far safer. Maybe it's still not foolproof if you're having power issues, but it is much more robust.

Reconfigure /etc/fstab to mount the stick or HDD somewhere permanent.
Edit the DatabaseTypes section in /etc/weewx.conf to point to that new drive.

[DatabaseTypes]
····
    # Defaults for SQLite databases
    [[SQLite]]
        driver = weedb.sqlite
        # Directory in which the database files are located
        SQLITE_ROOT = /mnt/my_usb_mounted_drive/weewx

And then continue backing up that database! (of course!)

I'd hasten the RTC addition, and when you do set that up make sure that it actually works when the network is disconnected. Correct time is an essential for weewx, it's the primary key in the database for starters.

If you have any questions on implementing any of the above, then I'll see you back over at rorpi discusion :-)

--

Cheers
 Glenn

rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie

[Praveen Chandrasekaran]

Glenn,

Even the usb stick will wear out if I write every 5 minutes i assume. Is there way to wrote out from memory to usb say every 1 hour and restore from usb on power up. That will extend the usb stick lifetime significantly. Or is this a rorpi discussion :) I found how to backup to usb in your page but couldn't find restoring on power up.

Regards,

Praveen 

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/XYmglRBbWrg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.

[Glenn McKechnie]

Probably belongs on rorpi, but this is worth saying here.

Tom K (weew2x's author) has reported that he's had a raspberry pi running for 5 years?? at 5 minute intervals, with an sqlite database and standard skin. I have to say that is mightly impressive. Seriously impressive; but it wasn't my experience.

Toms and my setups were/are different. I prefer a minute archive, had several skins going (cmon for one), and very unreliable power. On top of that I tried running mysql locally. But the biggest error of all was I had an ad-hoc backup plan, and I got caught!
I paid dearly, more than once. I possibly also struck a bad batch of Sandisk usb sticks? YMWV

5 minutes with an sqlite database shouldn't be a problem - depending on make manufacturer / quality and direction the wind is blowing.  But then you are doing regular backups so you're a long way along the path to recovery if the proverbial does hit the fan. If you don't have to reinstall the OS on top of rebuilding a corrupt database, then you'll be a lot happier. Trust me on that last one.

P.S. Did I mention it's a good idea to have backups, regular backups?

:-)

Cheers
 Glenn

rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie

[Thomas Keffer]

Not quite that impressive. It's 3 years and 2+ months: http://www.threefools.org/weewx/status/index.html

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Glenn McKechnie]

Andrew,

On 29 January 2018 at 22:19, Andrew Milner <andrew.s...@gmail.com> wrote:

I would suggest you have a proper plan on how to cope with a read only rpi - or maybe even just run the rpi and weewx in a standard configuration until you know how everything works - especially if you do not have a RTC installed. 

 

That is certainly good advice. ie:- get weewx working first.

If you stop the installation before the readonly step you'll have a working raspian OS with weewx installed. It's all ready to go.

i will add that note too,  even though they are suffering 'signage blindness' as it is  :-)

Also make sure that you are not running the latest NTP software which I believe has some form of buffering similar to fake hwclock - but not quite as bad!!  I saw someone mention this in a recent thread in this forum.

 

I'm intrigued by this but my googlefu might not be up to it, although I did find a reference by Gary to systemd-timesyncd possibly causing an issue. Would that be it?


-- 

[Glenn McKechnie]

Ah, I've just relocated the goal posts then.

It's still a winner though.

--

[Praveen Chandrasekaran]

Thanks everyone, i guess I will start writing my database to a usb stick then. My power supply is not as reliable (buying an easyacc power bank is one of todo items) . Hence i would prefer to keep sd card as RO so that OS doesn't get corrupted atleast. 

I will do a manual backup to a secondary usb stick or to a web server maybe once a week. Sounds good ? If to a web server i may make it once a day. 

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/XYmglRBbWrg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.