Database merge?

[PJO]

I have been running a FineOffset WH1080 for a few years with a Raspberry Pi 2 running Raspbian. Recently I added an Ecowitt GW1000 and have it running satisfactorily using this driver https://github.com/gjr80/weewx-gw1000 on Ubuntu Core on a Pi 3. 

Last night, having finished a new skin, I tried to transfer the weewx.sdb from the old to the new system, thinking (stupidly, hah) that with the same versions of WeeWX (4.4.0) it would be compatible. The result:

Mar 17 01:11:27 ubuntu weewx[1844]:  * Starting weewx weather system weewx

Mar 17 01:11:28 ubuntu weewx[1907] INFO __main__: Initializing weewx version 4.4.0

Mar 17 01:11:28 ubuntu weewx[1907] INFO __main__: Using Python 3.8.5 (default, Jan 27 2021, 15:41:15) #012[GCC 9.3.0]

Mar 17 01:11:28 ubuntu weewx[1907] INFO __main__: Platform Linux-5.4.0-1030-raspi-aarch64-with-glibc2.29

Mar 17 01:11:28 ubuntu weewx[1907] INFO __main__: Locale is 'LC_CTYPE=en_GB.UTF-8;LC_NUMERIC=en_GB.UTF-8;LC_TIME=en_GB.utf-8;LC_COLLATE=en_GB.UTF-8;LC_MONETARY=en_GB.UTF-8;LC_MESSAGES=C;LC_PAPER=en_GB.UTF-8;LC_NAME=en_GB.UTF-8;LC_ADDRESS=en_GB.UTF-8;LC_TELEPHONE=en_GB.UTF-8;LC_MEASUREMENT=en_GB.UTF-8;LC_IDENTIFICATION=en_GB.UTF-8'

Mar 17 01:11:28 ubuntu weewx[1907] INFO __main__: PID file is /var/run/weewx.pid

Mar 17 01:11:29 ubuntu weewx[1934] ERROR __main__: Error parsing config file: Parsing failed with several errors.#012First error at line 153. File /etc/weewx/weewx.conf

Mar 17 01:11:29 ubuntu weewx[1844]:    ...done.

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****  Traceback (most recent call last):

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****    File "/usr/share/weewx/weewxd", line 126, in main

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****      config_path, config_dict = weecfg.read_config(options.config_path, list(args))

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****    File "/usr/share/weewx/weecfg/__init__.py", line 176, in read_config

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****      config_dict = configobj.ConfigObj(config_path,

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****    File "/usr/lib/python3/dist-packages/configobj.py", line 1229, in __init__

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****      self._load(infile, configspec)

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****    File "/usr/lib/python3/dist-packages/configobj.py", line 1318, in _load

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****      raise error

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****    File "<string>", line None

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****  configobj.ConfigObjError: Parsing failed with several errors.

Mar 17 01:11:29 ubuntu weewx[1934] CRITICAL __main__:     ****  First error at line 153. File /etc/weewx/weewx.conf

I inspected weewx.conf and did't find anything obviously wrong, nor was there anything very interesting at line 153 (part of [[wunderground]], all commented out).

There must be something I missed. The GW1000 is reporting some extra variables so I presume this may be an issue?

I should have run wee_config? I thought just manually editing weex.conf was all I needed to do (having read this: https://github.com/weewx/weewx/wiki/Change-driver)

Then: What is the best way to merge two overlapping weewx.sdb databases? I think I'm not the only one who would like it to be easier (suggestion not a complaint)

I found references here to a file called transfer_db.py, indicated as NOT TESTED, and what followed required some tacit knowledge / familiarity with Python and, imo, would benefit from some elaboration. 

Alternatively, shouldn't wee_import be able to import from another weewx.sdb file (ideally with start and end dates)? A straight transfer isn't what I want, in any case, as the databases overlap. I know I can go figure out what the dateTimes are and could look at using SQL to drop records, but I think a convenient default MIGHT be to provide an option like

merge_db weewx1.sdb weewx2.sdb --output=weewx3.sdb --overlap-prefer=weewx2.sdb

I had a quick look at the databases with SQLbrowser but didn't do anything with it yet as I don't know what I should be expecting to see (I'm happy to manually merge dumped SQL records if I can interpret what they are and if I know that the database structure is the same/compatible; as it was my first time using this I realized I'd prefer, ideally, to have some instructions and perhaps one or more plugins, if there's one for displaying dateTimes eg. I haven't looked into this further yet--it'll involve a time commitment I'd happily avoid for a one-off operation; though SQLbrowser does look potentially useful for editing anomalous records however).

Would a simple merge tool be handy for anyone running two stations and suffering occasional outages? A wiki page on "Moving to a New Station"? 

[vince]

That's many items in one post.....

You likely have a typo in weewx.conf but without seeing yours it's impossible to know where.   Typically people mess up the [[ ]] number of brackets and have a mismatched number of brackets someplace.  It's also possible you have a non-printable character in there, or alternately you have a windows EOL rather than unix EOL if you did anything on a windows system.  There are many possible ways to mess the file up.

You can easily merge databases by doing a dump/restore of the archive table, then rebuild your daily summaries and 'probably' the NOAA pages.  There are numerous posts for how to do this.

Moving to a new station is in the wiki as "Change from one driver to another" which mentions how to do that.   Follow it.  Some drivers ask questions so simply editing weewx.conf is not a guaranteed win.

[Tom Keffer]

You likely have a typo in weewx.conf but without seeing yours it's impossible to know where.   

In this case, we do know where: line 153 in weewx.conf.

[PJO]

This (attached) is the file with passwords, ids and coordinates removed.  

Here's what I did:

I stopped weewx on both RPis; copied the database to the new one, having backed up the database there, and then swapped in a copy of my weewx.conf with the uploads to wunderground etc. enabled. When that didn't work I went through and disabled all those. I did check both the weewx.conf files carefully using a split screen file comparison utility and didn't see anything wrong. I then went through and disabled the uploads, as shown here. I reverted to the original database and previous weewx.conf--and weewx ran again, so I assumed the problem was something in the database/configuration. 

If you can see anything wrong at line 153 I'll be interested to know what it is! And many thanks, of course, for looking.

[PJO]

I don't use Windows; Linux only. I didn't really change driver. I changed database. What changed in the working weewx.conf file (on the new system) was that I uncommented things (copied over originally from the old weex.conf) to enable uploads to wunderground and other services. I may have been precipitate in assuming that there must be something wrong with the database but I couldn't see anything wrong in the weewx.conf file. 

My hesitation about a dump/restore was that SQLbrowser showed at lot of tables and I didn't know if the database structure would be fully identical across two versions--given additional fields and data and starting versions. If it is, then of course I'm happy to follow the posts to address the merge question -- I've done similar things with MySQL dumps (I still think a merge tool would be nice to have--like a "wunderfixer" of one's own :-).

I've uploaded the bad weewx.conf with credentials etc removed. I'll take another look -- for non-printable characters and mismatched brackets tomorrow. Thanks.

[Graham Eddy]

no lines removed or added i.e. line 153 still in situ?

i seem to recall configobj has difficulties with empty stanza and more so with trailing comment lines. probably worth a try ensuring neither of those in the file e.g. leave enable=false uncommented

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/451deb4d-665a-40fc-ad2b-f1a801bdc7e7n%40googlegroups.com.
<badweewx.conf>

[Tom Keffer]

The version you posted parsed just fine, at least on my Python 3.9 implementation. Your log says weewxd was reading /etc/weewx/weewx.conf. Is that the version you posted? If so, the offending line either got commented out, or was removed.

It's in there somewhere!

-tk

--

[Tom Keffer]

ConfigObj has a problem with comments and empty stanzas. I'm not aware of any parsing problems.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/1AE2B36D-D53D-4C7D-9088-87627D11B3D1%40gmail.com.

[gjr80]

When I see issues that involve weewx.conf being copied around I always prefer to to look at a wee_debug output; it's output is the weewx.conf being used by WeeWX (provided you are not telling WeeWX to use another config file). Removes all doubt about whether you have the right or wrong weewx.conf.

Gary

[PJO]

I didn't remove any lines, only credentials etc. and yes, that's the version I had a problem with (am on Python v3.8.5). I'm happy to know that the problem is not the database. To be clear, this was the sequence

1. Fresh install of WeeWX 4.x on new Pi (upgraded later)

2. Old skin and conf files copied to new Pi; simulator configured initially in place of WH1080 driver; uploads to Internet services commented out

3. GW1000 driver installed on new Pi, skin changes made

4. weewx.conf copied in preparation for cutover; uploads reinstated by removing commenting

5. Old WeeWX (also 4.4) stopped and database copied and swapped in

6. New weewx.conf with uncommented uploads swapped in and weewx restarted

When it failed I compared the old and new weewx.conf files using Beyond Compare, which highlights changes. I started commenting things out again to see if I could figure out what was wrong before eventually reverting to prior db and conf file. File edits were made using Atom via an exported NFS share, which I haven't had any issues with doing before. What I was aiming to do was keep both systems running but swap the uploads task from one to another. My concern to minimize the outage time -- and perhaps doing it at midnight -- may have led to some mistake. I wouldn't have posted at all if I knew for sure it wasn't potentially db related. Many thanks for checking the file. I'll repeat the process and try again after I've merged the records I want. 

[PJO]

I've now managed to create two files with the data overlap removed (using SQL; and found this a valuable aid https://www.epochconverter.com/). In doing so I discovered that the weewx.sdb file I was indeed the problem as the step I missed was converting the database to a new schema! I was unaware of using the new extended schema.

Am I right in thinking that this is what I need to do:

using old weewx.sdb and new weewx.conf,

wee_database weewx.conf --reconfigure

mv weews.sdb weewx.sdb_bak

then merge weewx.sdb_bak and weewx.sdb_new, creating a new weewx.sdb and

wee_database weewx.conf --rebuild-daily