Merging databases

[Simon]

is it possible to use the wee_config_database command to merge two weewx.sdb together? if so what is the commandline?

TIA Simon

[vds]

On Thursday, December 26, 2013 11:45:47 AM UTC-8, Simon wrote:

is it possible to use the wee_config_database command to merge two weewx.sdb together? if so what is the commandline?

Part of the fun of weewx is researching and learning stuff. Try searching the weewx forums newsgroup a little for "merging two sqlite databases" and read the thread that is found. 

• Tom answered the question from somebody else earlier this year - https://groups.google.com/d/msg/weewx-user/T2VuiIkcAEM/wZzemNzWknQJ

• If you do the same query in Google, you get 113,000 hits :-) including this one, which is probably how I'd do it - http://stackoverflow.com/questions/14834613/merge-two-sqlite-files-on-osx

If there are any matching records between the two databases you're merging, you'll see a bunch of complaining to the screen when you import the files, but all it'll be saying is that duplicate records were ignored, which is almost certainly exactly what you want anyway.  Definitely never work off live copies, stop weewx and grab the database that's current and and work off offline copies just in case.

[Thomas Keffer]

Sorry, but wee_config_database was not designed for that. But, it's a fairly simple Python program to accomplish this.

I assume that both databases use the same schema? If so, call them old.sdb and weewx.sdb. The goal is to merge the data from old.sdb into weewx.sdb. Be sure to do a backup first. You may have to adjust paths if your databases are not in /home/weewx/archive. NOT TESTED.

from weewx.archive import Archive

old_archive_dict = {'driver' : 'weedb.sqlite',

                    'database' : '/home/weewx/archive/old.sdb'}

merged_archive_dict = {'driver' : 'weedb.sqlite',

                       'database' : '/home/weewx/archive/weewx.sdb'}

with Archive.open(old_archive_dict) as old_archive:

    with Archive.open(merged_archive_dict) as merged_archive:

        # This is very fast because it is done in a single transaction context:

        merged_archive.addRecord(old_archive.genBatchRecords())

See how that works for you.

-tk

On Thu, Dec 26, 2013 at 11:45 AM, Simon <simon.s...@gmail.com> wrote:

is it possible to use the wee_config_database command to merge two weewx.sdb together? if so what is the commandline?

TIA Simon

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Thomas Keffer]

Oops. Saw Vince's response after I posted my own. Good advice.

These days, I'd say 90% of the answers to support questions can be found in the manual or in the weewx Google group.

-tk

[zkab]

I followed the instructions https://groups.google.com/d/msg/weewx-user/T2VuiIkcAEM/wZzemNzWknQJ

but got a bunch of errors ...

loke@loke:~/Downloads$ ls -l *sdb

-rw-r--r-- 1 loke loke 16837632 sep  3 15:10 weewx.sdb

-rw-r--r-- 1 loke loke 16439296 aug 28 05:55 wview.sdb

loke@loke:~/Downloads$ sqlite3 wview.sdb

SQLite version 3.16.2 2017-01-06 16:32:41

Enter ".help" for usage hints.

sqlite> .output wview.dump

sqlite> .dump

sqlite> .quit

loke@loke:~/Downloads$ ls -l *sdb *dump

-rw-r--r-- 1 loke loke 16837632 sep  3 15:10 weewx.sdb

-rw-rw-r-- 1 loke loke 31878436 sep  3 15:27 wview.dump

-rw-r--r-- 1 loke loke 16439296 aug 28 05:55 wview.sdb

loke@loke:~/Downloads$ sqlite3 weewx.sdb

SQLite version 3.16.2 2017-01-06 16:32:41

Enter ".help" for usage hints.

sqlite> .read wview.dump

Error: near line 3: table archive already exists

Error: near line 4: UNIQUE constraint failed: archive.dateTime

Error: near line 5: UNIQUE constraint failed: archive.dateTime

Error: near line 6: UNIQUE constraint failed: archive.dateTime

Error: near line 7: UNIQUE constraint failed: archive.dateTime

Error: near line 8: UNIQUE constraint failed: archive.dateTime

...

[Andrew Milner]

well you will get errors if the same datetime is in both databases - was that the situation??