CRITICAL: 'utf8' codec can't decode byte 0x96 in position 19: invalid start byte

[Ovidiu-Florin BOGDAN]

I've made some changes and added a few new files in my SVN checkout. Now I'm trying to update a review request using rbtools.

$ rbt post -r 27
One or more files in your changeset has history scheduled with commit. Please try again with '--svn-show-copies-a
s-adds=y/n'

$ rbt post -r 27 --svn-show-copies-as-adds=y
CRITICAL: 'utf8' codec cant decode byte 0x96 in position 19: invalid start byte

I have no idea what this refers to.

[Ovidiu-Florin BOGDAN]

I'm pasting the debug output here:

$ rbt post -r 27 --svn-show-copies-as-adds=y -d                            
>>> RBTools 0.7.1
>>> Python 2.7.9 (default, Dec 11 2014, 04:42:00)  
[GCC 4.9.2]
>>> Running on Linux-3.18.6-1-ARCH-x86_64-with-glibc2.2.5
>>> Home = /home/bogdano
>>> Current directory = /home/bogdano/WORK/path/to/project
>>> Checking for a Subversion repository...
>>> Running: svn info --non-interactive
>>> Running: diff --version
>>> repository info: Path: http://server/svn, Base path: /TRUNK/path/to/project, Supports changesets: False
>>> Making HTTP GET request to http://reviewboard/reviews/api/
>>> Making HTTP GET request to http://reviewboard/reviews/api/repositories/
>>> Running: svn info --non-interactive
>>> Running: diff --version
>>> repository info: Path: http://server/svn, Base path: /TRUNK/path/to/project, Supports changesets: False
>>> Running: svn status --ignore-externals
>>> Running: svn diff --diff-cmd=diff --notice-ancestry -r BASE --show-copies-as-adds
Traceback (most recent call last):
  File "/usr/bin/rbt", line 9, in <module>
    load_entry_point('RBTools==0.7.1', 'console_scripts', 'rbt')()
  File "/usr/lib/python2.7/site-packages/rbtools/commands/main.py", line 133, in main
    command.run_from_argv([RB_MAIN, command_name] + args)
  File "/usr/lib/python2.7/site-packages/rbtools/commands/__init__.py", line 538, in run_from_argv
    exit_code = self.main(*args) or 0
  File "/usr/lib/python2.7/site-packages/rbtools/commands/post.py", line 639, in main
    extra_args=extra_args)
  File "/usr/lib/python2.7/site-packages/rbtools/clients/svn.py", line 297, in diff
    diff = self._run_svn(diff_cmd, split_lines=True)
  File "/usr/lib/python2.7/site-packages/rbtools/clients/svn.py", line 658, in _run_svn
    return execute(cmdline, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rbtools/utils/process.py", line 104, in execute
    data = [line.decode('utf-8') for line in data]
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec cant decode byte 0x96 in position 19: invalid start byte

I ran 

svn diff --diff-cmd=diff --notice-ancestry -r BASE --show-copies-as-adds

and it executed with no errors. I cannot show the diff, but I can look for something in it.

[Christian Hammond]

Can you upgrade to 0.7.2 and give that a try? We've fixed some Unicode issues since 0.7.1.

Christian

--
Christian Hammond - chri...@beanbaginc.com
Review Board - https://www.reviewboard.org
Beanbag, Inc. - https://www.beanbaginc.com

> --
> Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups "reviewboard"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Ovidiu-Florin BOGDAN]

I've Updated it. Now it's broken:

$ rbt diff --svn-show-copies-as-adds=y -d                                              
>>> RBTools 0.7.2

>>> Python 2.7.9 (default, Dec 11 2014, 04:42:00)  
[GCC 4.9.2]
>>> Running on Linux-3.18.6-1-ARCH-x86_64-with-glibc2.2.5
>>> Home = /home/bogdano
>>> Current directory = /home/bogdano/WORK/path/to/project
>>> Checking for a Subversion repository...
>>> Running: svn info --non-interactive
>>> Running: diff --version

>>> Running: svn --version -q

Traceback (most recent call last):

  File "/usr/bin/rbt", line 9, in <module>
    load_entry_point('RBTools==0.7.2', 'console_scripts', 'rbt')()
  File "/usr/lib/python2.7/site-packages/rbtools/commands/main.py", line 133, in main
    command.run_from_argv([RB_MAIN, command_name] + args)
  File "/usr/lib/python2.7/site-packages/rbtools/commands/__init__.py", line 555, in run_from_argv
    exit_code = self.main(*args) or 0
  File "/usr/lib/python2.7/site-packages/rbtools/commands/diff.py", line 41, in main
    client_name=self.options.repository_type)
  File "/usr/lib/python2.7/site-packages/rbtools/commands/__init__.py", line 584, in initialize_scm_tool
    client_name=client_name)
  File "/usr/lib/python2.7/site-packages/rbtools/clients/__init__.py", line 370, in scan_usable_client
    repository_info = tool.get_repository_info()
  File "/usr/lib/python2.7/site-packages/rbtools/clients/svn.py", line 88, in get_repository_info
    self.subversion_client_version = tuple(map(int, ver_string.split('.')))
ValueError: invalid literal for int() with base 10: 'svn: warning: cannot set LC_CTYPE locale svn: warning: envir
onment variable LC_ALL is en_US'

$ locale
LANG=ro_RO.UTF-8
LC_CTYPE="ro_RO.UTF-8"
LC_NUMERIC="ro_RO.UTF-8"
LC_TIME="ro_RO.UTF-8"
LC_COLLATE="ro_RO.UTF-8"
LC_MONETARY="ro_RO.UTF-8"
LC_MESSAGES="ro_RO.UTF-8"
LC_PAPER="ro_RO.UTF-8"
LC_NAME="ro_RO.UTF-8"
LC_ADDRESS="ro_RO.UTF-8"
LC_TELEPHONE="ro_RO.UTF-8"
LC_MEASUREMENT="ro_RO.UTF-8"
LC_IDENTIFICATION="ro_RO.UTF-8"
LC_ALL=

It seems to me it has a problem with my locale. Why?

[Christian Hammond]

It's possible it's missing the en_US locale, which is needed to keep Subversion from localizing some strings they shouldn't be localizing, which we need to parse. There's going to be some package available for your distribution that provides it, but it'll be distribution-dependent. You'll need to figure out what that is and install it.

It actually looks to be getting further than it did before, though.

> load_entry_point('RBTools==0.7.2', 'console_scripts', 'rbt')()

> File "/usr/lib/python2.7/site-packages/rbtools/commands/main.py", line 133
> , in main
> command.run_from_argv([RB_MAIN, command_name] + args)
> File "/usr/lib/python2.7/site-packages/rbtools/commands/__init__.py",

[Ovidiu-Florin BOGDAN]

No, it didn't get further.

I've fixed the locale. I had :

$ locale -a
C
en_US
en_US.iso88591
POSIX
ro_RO.utf8

And I've changed it to:

$ locale -a                      
C
en_US.utf8
POSIX
ro_RO.utf8

Now I get the same error as before:

$ rbt diff --svn-show-copies-as-adds=y -d
>>> RBTools 0.7.2
>>> Python 2.7.9 (default, Dec 11 2014, 04:42:00)  
[GCC 4.9.2]
>>> Running on Linux-3.18.6-1-ARCH-x86_64-with-glibc2.2.5
>>> Home = /home/bogdano

>>> Current directory = /home/bogdano/path/to/project

>>> Checking for a Subversion repository...
>>> Running: svn info --non-interactive
>>> Running: diff --version
>>> Running: svn --version -q

>>> repository info: Path: http://server/svn, Base path: /TRUNK/path/to/project, Supports changesets: False
>>> Making HTTP GET request to http://reviewboard/reviews/api/

>>> Running: svn info --non-interactive
>>> Running: diff --version
>>> Running: svn --version -q

>>> repository info: Path: http://server/svn, Base path: /TRUNK/path/to/project, Supports changesets: False

>>> Running: svn diff --diff-cmd=diff --notice-ancestry -r BASE --show-copies-as-adds

Traceback (most recent call last):

  File "/usr/bin/rbt", line 9, in <module>
    load_entry_point('RBTools==0.7.2', 'console_scripts', 'rbt')()
  File "/usr/lib/python2.7/site-packages/rbtools/commands/main.py", line 133, in main
    command.run_from_argv([RB_MAIN, command_name] + args)
  File "/usr/lib/python2.7/site-packages/rbtools/commands/__init__.py", line 555, in run_from_argv
    exit_code = self.main(*args) or 0
  File "/usr/lib/python2.7/site-packages/rbtools/commands/diff.py", line 68, in main
    extra_args=extra_args)
  File "/usr/lib/python2.7/site-packages/rbtools/clients/svn.py", line 319, in diff
    diff = self._run_svn(diff_cmd, split_lines=True)
  File "/usr/lib/python2.7/site-packages/rbtools/clients/svn.py", line 707, in _run_svn
    return execute(cmdline, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rbtools/utils/process.py", line 104, in execute
    data = [line.decode('utf-8') for line in data]
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)

[Christian Hammond]

Would you be able to provide the output of "svn diff --diff-cmd=diff --notice-ancestry -r BASE --show-copies-as-adds" ?

Christian

--
Christian Hammond - chri...@beanbaginc.com
Review Board - https://www.reviewboard.org
Beanbag, Inc. - https://www.beanbaginc.com

-----Original Message-----
From: Ovidiu-Florin BOGDAN <ovidi...@gmail.com>
Reply: Ovidiu-Florin BOGDAN <ovidi...@gmail.com>>
Date: March 11, 2015 at 1:35:23 AM
To: revie...@googlegroups.com <revie...@googlegroups.com>>
Cc: ovidi...@gmail.com <ovidi...@gmail.com>>, chri...@beanbaginc.com <chri...@beanbaginc.com>>

[Ovidiu-Florin BOGDAN]

I cannot provide the diff. Sorry. It seems to me like a regular diff file.

I've opened the diff in Kate and it chose the ISO-8859-15 encoding automatically.

I've manually switched the encoding to Unicode UTF-8 and it complains about invalid characters.

I think this character might be the culprit: "–". You might not see it, but it's there, depends on your reader encoding.

[Ovidiu-Florin BOGDAN]

Fixed. Once the character "–" ( using ISO-8859-15 encoding ) that looks like <==> was removed form the diff, RBT worked fine.

This seems to me like a bug in RBT.

[David Trowbridge]

Thanks for the debug info. I'm currently working on this, and we'll get it fixed in the next version of RBTools.

-David

[Ovidiu-Florin BOGDAN]

When will this be fixed?

When will the next release of RBT be?

[David Trowbridge]

Hi,

The bug is fixed in the git release-0.7.x branch. We're not sure exactly when the next release will be, but it will probably be within the next few weeks.

-David