UnicodeDecodeError in database

[David Marshall]

Hi

Installed a fresh version of weewx 4.1.1 without problem using setup.py on a raspberry.

Transferred my existing sqlite database from weewx 3.9.1. It is big, about 3 years of data. This worked fine.

But when I tried to use reconfigure I got the following error

pi@raspberrypi:~ $ /home/weewx/bin/wee_database --reconfigure /home/weewx/archive/weewx.sdb

Traceback (most recent call last):

  File "/home/weewx/bin/wee_database", line 974, in <module>

    main()

  File "/home/weewx/bin/wee_database", line 128, in main

    config_path, config_dict = weecfg.read_config(options.config_path, args)

  File "/home/weewx/bin/weecfg/__init__.py", line 179, in read_config

    default_encoding='utf-8')

  File "/usr/lib/python3/dist-packages/configobj.py", line 1229, in __init__

    self._load(infile, configspec)

  File "/usr/lib/python3/dist-packages/configobj.py", line 1287, in _load

    content = self._handle_bom(content)

  File "/usr/lib/python3/dist-packages/configobj.py", line 1437, in _handle_bom

    return self._decode(infile, self.encoding)

  File "/usr/lib/python3/dist-packages/configobj.py", line 1517, in _decode

    infile[i] = line.decode(encoding)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 26: invalid continuation byte

Anyone know how to fix this?

[Tom Keffer]

The decode error is actually in your configuration file, weewx.conf, and not in the database.

Somehow, your copy of weewx.conf got corrupted. One of the lines has a character sequence that starts out as a utf-8 encoding, but the following byte is not what was expected. 

Unfortunately, the error does not tell you which line the bad character sequence is in, only that it is in byte position 26 of that line. Perhaps this will help (NOT TESTED):

od -x -A d weewx.conf | grep e0

This should print out the line and offending sequence.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/97c28bf4-30c7-455c-851f-469187c31c79o%40googlegroups.com.

[David Marshall]

Thanks Tom

Tried your code to show the offending line but nothing appears. Then tried a clean weewx.conf from the install folder but still the same error message. Then tried a full reinstall, but still the same error.

Then tried it with the the default database using simulator and still the same error - so you are right it is not my database. What next?

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Tom Keffer]

1. All these attempts were with running wee_database? Or, something else?

2. I assume all these different attempts used the same path, namely /home/weewx/weewx.conf. And, we're sure they used different copies (albeit, at the same path location)? 

3. It's hard to explain how fresh, untouched versions of weewx.conf are all getting the same error. Are you editing these copies before running?

Can you send me the offending weewx.conf to my personal email? tkeffer at gmail.com

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/93094213-6958-49a1-ba13-7f3ca3a90274o%40googlegroups.com.

[David Marshall]

Hi Tom

 

1. Yes wee_database
2. Yes same path different  copies
3. No edits – in fact I tried not to change anything in the final install apart from setting it to simulator.

 

I attach the last version of weewx.conf, which is still producing the error.

 

David

 

Sent from Mail for Windows 10

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEDyDz5xh8gtJRyeCKxmCbJGuAYqChLJeq4d%2BGQrD3vUFA%40mail.gmail.com.

 

[Tom Keffer]

I don't have any problem with either copy of weewx.conf that you sent to me. Things to try:

1. This is /home/weewx/weewx.conf, right?  

2. Try reading it directly:

python -c "import configobj; c=configobj.ConfigObj('/home/weewx/weewx.conf')"

3. Set debug=1 in weewx.conf, then try running weewxd, then send us the log. 

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/5f3d34eb.1c69fb81.4c509.d9cd%40mx.google.com.

[David Marshall]

1. Yes
2. Seemed to work ok
3. Syslog attached – but seems to me to be ok.

 

The problem only occurs when using wee_database -interestingly the offending byte has moved from position 26 to 31!

 

pi@raspberrypi:~ $ /home/weewx/bin/wee_database --reconfigure /home/weewx/archive/weewx.sdb

Traceback (most recent call last):

  File "/home/weewx/bin/wee_database", line 974, in <module>

    main()

  File "/home/weewx/bin/wee_database", line 128, in main

    config_path, config_dict = weecfg.read_config(options.config_path, args)

  File "/home/weewx/bin/weecfg/__init__.py", line 179, in read_config

    default_encoding='utf-8')

  File "/usr/lib/python3/dist-packages/configobj.py", line 1229, in __init__

    self._load(infile, configspec)

  File "/usr/lib/python3/dist-packages/configobj.py", line 1287, in _load

    content = self._handle_bom(content)

  File "/usr/lib/python3/dist-packages/configobj.py", line 1437, in _handle_bom

    return self._decode(infile, self.encoding)

  File "/usr/lib/python3/dist-packages/configobj.py", line 1517, in _decode

    infile[i] = line.decode(encoding)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf2 in position 31: invalid continuation byte

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEB99R%3D5MrZB6EYfi1MOPcW96E06bPs3ayV5FAPF_aK7NA%40mail.gmail.com.

 

[Tom Keffer]

Stupid me. The answer is in how you are invoking wee_database. Any positional argument is interpreted as the path to weewx.conf, so wee_database is thinking that /home/weewx/archive/weewx.sdb is your configuration file, weewx.conf. Naturally, it being a sqlite database, it has all sorts of unusual byte sequences.

wee_database finds your database from the information in the configuration file, not from the command line. This way, it can work on MySQL databases, which do not use a single file. So, you want

wee_database --reconfigure /home/weewx/weewx.conf

[David Marshall]

Solved thanks. More stupid me I think – a clear case of RTFM!

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEAKFYrufMVdf%3DL3YKuy2uXp%2B5sXdQeD77DV7LEWXB6qrg%40mail.gmail.com.