Cannot push commits encoded on ISO-8859-14 charset to gerrit

[Ahmed Hosni]

Hi there,

When mirroring a community git repo to our gerrit instance (V 2.5.2), i faced an issue with the identity of some comitters, their names (König, Bjørn, etc..) was in an unsupported charset and gerrit failed to push those refs. As you can see in the attached error_log, it throws the following exception " java.nio.charset.UnsupportedCharsetException: ISO-8859-14 ".

I added this java option to the [container] section of gerrit config file like below:

javaOptions = -Dfile.encoding="ISO-8859-14"

After restarting gerrit, the JVM seems to use the 'ISO-8859-14' encoding to use to manage filenames/pathnames at the local disk FS:

$ ./bin/gerrit.sh check
Checking arguments to Gerrit Code Review:
  GERRIT_SITE     =  /home/gerritadm/gerrit_site
  GERRIT_CONFIG   =  /home/gerritadm/gerrit_site/etc/gerrit.config
  GERRIT_PID      =  /home/gerritadm/gerrit_site/logs/gerrit.pid
  GERRIT_TMP      =  /home/gerritadm/gerrit_site/tmp
  GERRIT_WAR      =  /home/gerritadm/gerrit_site/bin/gerrit.war
  GERRIT_FDS      =  1024
  GERRIT_USER     =  gerritadm
  JAVA            =  /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java
  JAVA_OPTIONS    =   -Dfile.encoding=ISO-8859-14
  RUN_EXEC        =  /usr/bin/perl -e '$x=$ENV{JAVA};exec $x @ARGV;die $!' -- GerritCodeReview
  RUN_ARGS        =   -Dfile.encoding=ISO-8859-14 -jar /home/gerritadm/gerrit_site/bin/gerrit.war daemon -d /home/gerritadm/gerrit_site

Unfortunately, this is not the right solution.

In order to test, i changed the i18n within my git config and tried to push a commit with the new encoding to gerrit:

$git clone ssh://hosniah@mygerritserver:29418/codex-cc/gitweb
$cd gitweb/
$git config --local i18n.commitencoding ISO-8859-14
$touch test_encoding
$git add test_encoding
$git commit -m 'Test ISO-8859-14 encoding'
[master fa9c802] Test ISO-8859-14 encoding
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 test_encoding
$git commit -m 'Test ISO-8859-14 encoding'
[master fa9c802] Test ISO-8859-14 encoding
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 test_encoding

 Checking that the commit is actually specifying the right encoding:

$git log --format=raw -n1 HEAD
commit fa9c8026470528c1569a8cc4023c91a8574e3f4c
tree 68b9e4a8aa531ee5bad7092a15bf4e055a2e8b3d
parent f332c58de6afa9f0f536a526fac66de73ad24b3f
author Ahmed HOSNI <xxxxx@xxxx> 1402064085 +0100
committer Ahmed HOSNI <xxxxx@xxxx> 1402064085 +0100
encoding ISO-8859-14
    Test ISO-8859-14 encoding

When trying to push , it fails:

$git push origin HEAD:refs/for/master
Counting objects: 4, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 332 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Processing changes: done
remote: error: internal error while processing changes
To ssh://hosniah@mygerritserver:29418/codex-cc/gitweb
 ! [remote rejected] HEAD -> refs/for/master (internal error)
error: failed to push some refs to 'ssh://hosniah@mygerritserver:29418/codex-cc/gitweb'

The following exception is raised at Gerrit error_log:

[2014-06-06 15:54:20,825] WARN  com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits
java.util.concurrent.ExecutionException: java.nio.charset.UnsupportedCharsetException: ISO-8859-14
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262)
        at java.util.concurrent.FutureTask.get(FutureTask.java:119)
       ....

Is there any way to help gerrit receiving my commits encoded in the iso-8859-14 charset?

Is there any other option to patch gerrit or jgit?

Thanks in advance.

Best regards,

Ahmed HOSNI

[Shawn Pearce]

On Fri, Jun 6, 2014 at 7:35 AM, Ahmed Hosni <ahmedhosn...@gmail.com> wrote:

Hi there,

When mirroring a community git repo to our gerrit instance (V 2.5.2), i faced an issue with the identity of some comitters, their names (König, Bjørn, etc..) was in an unsupported charset and gerrit failed to push those refs. As you can see in the attached error_log, it throws the following exception " java.nio.charset.UnsupportedCharsetException: ISO-8859-14 ".

Your installation of Java does not support ISO-8859-14. You need to modify the JRE/JVM to support this character set. Gerrit and JGit have nothing to do with the JVM not understanding this particular character encoding.

[Robin Rosenberg]

----- Ursprungligt meddelande -----
> Från: "Ahmed Hosni" <ahmedhosn...@gmail.com>
> Till: repo-d...@googlegroups.com
> Skickat: fredag, 6 jun 2014 16:35:29
> Ämne: Cannot push commits encoded on ISO-8859-14 charset to gerrit

>
> Hi there,
>
> When mirroring a community git repo to our gerrit instance (V 2.5.2), i
> faced an issue with the identity of some comitters, their names (König,
> Bjørn, etc..) was in an unsupported charset and gerrit failed to push those

Where did you get the ISO-8859-14 from? It seems you added that ISO-8859-14 commit
message yourself...

-- robin

[Bassem Rabil]

I think if you use UTF-8 encoding, such latin special characters will be supported. Make sure you set this in your container if you use tomcat web server, i.e. URIEncoding="UTF-8"  

[Ahmed Hosni]

Hi all,

As Shawn noticed, my JRE doesn't support ISO-8859-14 (i'm using Jetty as container running on openjdk-1.6.0 / RHEL 6.3).

The workaround was to provide a class that handles the encoding and decoding of single-byte ISO-8859-14 where the byte-to-char conversion is performed using a simple lookup table.

I exported the whole stuff to a jar file and upload it to jre/lib/ext under my openjdk-1.6.0, without a clue. Same result when upgrading to openjdk-1.7.0.

After changing the vendor (i've installed JDK 1.7.0 from Oracle) , i updated container.javaHome within my gerrit conf then added my jar file in the lib/ext directory of the new JRE,

 it works like a charm.