adding records from one weewx database to another?

[Dana Roode]

Is there a way to extract archive records from one weewx.sdb file and import then into another?  I see there is a utility that allows importing of CSV data files.

My linux system died on September 9th and I did not have time to build a new one right away (the system and weather station is located 80 miles away from me).  I did have the old one from about 10 months ago, and I put that in line.  It received a couple of weeks of records from the Davis Vantage Pro2 weather station.  I setup the new system with the database I had through September 9th and now that database is the active one.  I'd like to move a few weeks of data from the interim system to the active one.

I'm running the latest weewx, now using Ubuntu rather than Centos.  It went very well, other than a bug I previously reported (fixed with a vantage.py change).  I will post about that later.

  Dana

[Andrew Milner]

I am not sure I understand what databases you have got!!

one from 10 months ago

one from september 9 which was built on the database backup from 10 months ago - is this the 'interim' one??

an active one - how does this differ from the interim one??

it is not clear to mne what you are trying to do/achieve as you seem to have got the data up to 10 months ago and data from a data prior to sept 9th to current date

am confused

the simplest thing to do is probably to

a) stop weewx

b) make a copy the current database (weewx.sdb - > weewx.copy.sdb

c) delete the current database
d) copy the old database from 10 months ago to weewx.sdb

e) restart weewx and if you are lucky the records from 9th september will still be in the vantage and will be retrieved and added to the database

Your database will now consist of the 10 month old records plus records from within vantage pro logger

Does this achieve what you need??

[Greg Troxel]

Dana Roode <dana....@gmail.com> writes:

> Is there a way to extract archive records from one weewx.sdb file and
> import then into another? I see there is a utility that allows importing
> of CSV data files.
>
> My linux system died on September 9th and I did not have time to build a
> new one right away (the system and weather station is located 80 miles away
> from me). I did have the old one from about 10 months ago, and I put that
> in line. It received a couple of weeks of records from the Davis Vantage
> Pro2 weather station. I setup the new system with the database I had
> through September 9th and now that database is the active one. I'd like to
> move a few weeks of data from the interim system to the active one.

So you have:

a system that ran until sep 9, and has all data from the beginning
until then, plus from a few days ago until now, and is getting new
data ("active")

a system from 10 months ago, that you also ran recently. This has
data from the beginning to december 2018, and also sep 9 to a few days
ago ("interim")

I am unclear on programs, etc. but you can do this with sqlite3 command
line tools and an editor, carefully.


first, BACK UP the active system weewx.sdb, to multiple other places.
Do the same with the interim one.


run 'sqlite3 weewx.sdb .dump' onto a file on the interim system.
Extract just the insert statements for the archive table. Sort them
numerically (with sort(1) perhaps).

Do the same with the active system, just so you can look.

Figure out which timestamps are missing from the active system.
Select the lines from the interim dump that correspond and put them in "missing.sql"

then on the active system

sqlite3 weewx.sdb < missing.sql

then you'll need to rebuild archive records. It really helps to know a
little sql but you only need to sort and select.


It is of course possible to write a program to open both databases and
look for what's missing, tell you about that, and copy records.

[vince]

On Wednesday, September 25, 2019 at 4:41:54 AM UTC-7, Greg Troxel wrote:

Dana Roode <dana...@gmail.com> writes:

> Is there a way to extract archive records from one weewx.sdb file and
> import then into another?  I see there is a utility that allows importing
> of CSV data files.
>
>

Yes - there are many ways.

Simplest is probably to do a sqlite3 dump of all the database archive tables, then restore from 'each' of them into your aggregate database.   It'll whine when it inevitably finds duplicate records, but nothing bad will happen.  The result should be the union of all the databases.   Weewx should handle the summary tables, but you could always rebuild them manually too if you wanted.

[Dana Roode]

Thanks everyone, I appreciate the info.  I may dump the databases with sqlite3 and verify what I have and what I'm missing.  

I took another look this morning and I may only be missing the days when the system was down after the initial failure.  The only way to get that data back would be if the Vantage weather station had it.  I see that weewx has been able to go back a ways and get most of the missing data, but I don't know what triggers that and whether or not there is any more data in the weather station that has not already been restored.

I see a "dump" option in wee_device to move all the data in the weather station to the archive, perhaps that would do the trick.

Its not overly relevant at this point, but since I confused folks with my post, let me explain again "for the record" my recent hardware/system-building fun:

- Production system in place 12/2018->9/9/2019, died on 9/9

- Got to site on 9/12, the production system would not power up.  Installed the old system I had replaced in 12/2018; it had an old weewx.sdb on it (referred to as "interim" in my original post)

- Later in the month, I moved the production disk into "replacement system 1;" it had the production weewx.sdb file.  Took system to the site on 9/22 and it ran for a few hours.  But it had troubles mounting sysroot, making me suspicious of the disk drive.  I reverted to the interim system.

- Built "replacement system 2" (new computer and disk) with the former production weewx.sdb on it (with data from 12/2018 through 9/9/2019 and some that was evidently added by "replacement system 1" on 9/22)

- On 9/24 took "replacement system 2" to site and brought it up; message in the log indicated weewx added records to weewx.sdb that came from 9/22 until present 

I believe most of the other data was placed in the production database during the short period using "replacement system 1" on 9/22.  So, this is why I am not missing as much data as I thought.   I am only missing data from 9/9 until 9/12 when no system was up.  

Weewx is pretty smart!  Sorry for confusing everyone.  I hope my luck in hardware improves and the new system stays up for awhile.

  Dana

[vince]

On Wednesday, September 25, 2019 at 12:52:15 PM UTC-7, Dana Roode wrote:

 I see that weewx has been able to go back a ways and get most of the missing data, but I don't know what triggers that and whether or not there is any more data in the weather station that has not already been restored.

Weewx will grab whatever data iit hasn't previously put into your database from your Davis datalogger.    If you're running the typical 5-minute archive period, that's about 8 or so days of storage.  If your archive period is quicker/longer than that, you'd need to do the math for how long you can be down before the datalogger starts deleting oldest-first to save the most current N days (actually N readings) of data.

[Dana Roode K6NR]

Ok, thanks.  Yes, I'm using 5 minute intervals.  So, all the data the logger has is already in the archive.  This is good, as the wee_device --dump didn't work.

  Dana

/var/lib/weewx# wee_device --dump

Using configuration file /etc/weewx/weewx.conf

Using Vantage driver version 3.1.1 (weewx.drivers.vantage)

Proceeding will dump all data in the logger.

Are you sure you want to proceed (y/n)? y

Starting dump ...

Traceback (most recent call last):

  File "/usr/bin/wee_device", line 66, in <module>

    main()

  File "/usr/bin/wee_device", line 62, in main

    device.configure(config_dict)

  File "/usr/share/weewx/weewx/drivers/__init__.py", line 69, in configure

    self.do_options(options, parser, config_dict, prompt)

  File "/usr/share/weewx/weewx/drivers/vantage.py", line 2018, in do_options

    self.dump_logger(station, config_dict, options.noprompt)

  File "/usr/share/weewx/weewx/drivers/vantage.py", line 2618, in dump_logger

    for record in converted_generator:

  File "/usr/share/weewx/weewx/units.py", line 1247, in next

    _record = self.input_generator.next()

  File "/usr/share/weewx/weewx/drivers/vantage.py", line 660, in genArchiveDump

    _page = self.port.get_data_with_crc16(267, prompt=_ack, max_tries=self.max_tries)

  File "/usr/share/weewx/weewx/drivers/vantage.py", line 207, in get_data_with_crc16

    raise weewx.CRCError("Unable to pass CRC16 check while getting data")

weewx.CRCError: Unable to pass CRC16 check while getting data

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/t52g84-JiOo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/98c822c4-1b93-435f-89c6-cec68182a433%40googlegroups.com.