Issue 491 in msysgit: Multiple errors when installing msysgit with a Windows user name that has non-ASCII characters

[msy...@googlecode.com]

Status: New
Owner: ----

New issue 491 by ajriesgo: Multiple errors when installing msysgit with a
Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

Have you searched in the issues (both open and closed) for keywords of your
current issue?

Yes, and it is similar to issue 108, which appears as fixed. Feel free to
merge it with that one if that is appropriate.
This issue is also mentioned by user jiho in a discussion I found on Google
Groups:
http://groups.google.com/group/msysgit/browse_thread/thread/aae8f28906d1c1aa?fwc=2 .
I wasn't able to reply to that discussion (I only have a "reply to author"
link, even after subscribing as a user). So, I've finally decided to post
this as a new issue. My apologies if this is not the best way of reopening
something that has been mentioned in the past.

What steps will reproduce the problem?

1. In Windows XP create a user account with a name that has non-ASCII
characters (in my case "Ángel". I've also tried names like "José" and "李安
和").

2. Open a Windows session with the non-ASCII user name.

3. Install Git-1.7.0.2-preview20100309.exe

4. After installing, errors will occur. In particular, running Git Bash
will display the following error:
C:\Documents and Settings\'ngel' doesn't exist!
This is an error which might be related to msysGit issue 108.
You may want to set the environment variable HOME explicitely (sic).
Falling back to [...]
[..]

Another one (which has driven me crazy for two days) is that after creating
a new repository any operations with files that require a comment
like "commit" bring up an error "commit-tree failed", as well as a
warning "message does not conform to utf-8". This was mentioned by user
jiho in the link above. Note that I have done this using Git Gui, not
through a console, but I don't think that should be relevant.

I also found similar errors when trying to change the option "Default File
Contents Encoding" in the Edit->Options dialog in Git Gui.

What steps did you perform to try to fix the issue yourself?

After carrying out some web searches and experimenting, I figured out that
the problem was that my Windows user name has an accented letter. The
inability to commit disappeared when I did the following: I opened the
Edit->Options dialog using Git Gui and filled in the User Name and Email
Address (both the repository and the global ones). It turns out that Git
was relying on my Windows user name to generate default values for these
fields when they are blank, but there seems to be an error in how the user
name is retrieved (see the additional information below)

What version of the product are you using? On what Windows version (release
and CPU)?

1.7.0.2.msysgit.0
Windows XP SP3 (Spanish)

Please provide any additional information below.

I have downloaded the code (I haven't been able to debug though) and I
think I may have found what the problem is. The file connect.c has a
function getusername (line 627) that has a WIN32-specific implementation
that uses the function GetUserName. The problem is that it is using the
ANSI-specific function GetUserNameA and not the wide-character
GetUserNameW, needed to support any Unicode character in the UTF-16
representation used by Windows. Basically, this means that a user name with
Spanish accented letters will be codified as ISO-8859-1 (as in any Western
European Windows system) and a user name with characters outside the scope
of my local encoding (like the 李安和 of my tests) will end up corrupted.
The rest of the system seems to expect UTF-8-encoded names, but this is not
what getusername is returning.

My guess is that in order to support any Unicode user name, the getusername
function should be modified so that it uses the wide-character Unicode
function GetUserNameW rather than the ANSI version (either explicitly or by
#defining the UNICODE macro). Then the retrieved wide-character string
should be converted to a char-based UTF-8-encoded string before the
function returns. This way, the client code that calls getusername would be
guaranteed a UTF-8-encoded string.

I hope my description is clear enough. As I said, feel free to merge this
issue with an existing one or change its format or whatever.

Ángel José Riesgo

[msy...@googlecode.com]

Comment #1 on issue 491 by johannes.schindelin: Multiple errors when

installing msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

Have you tried with the net installer? It might be that the issue is
already fixed in the developer version.

[msy...@googlecode.com]

Comment #2 on issue 491 by sschuberth: Multiple errors when installing

msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

To start with, I've fixed the "explicitely" typo (not sure whether this
will make it into the upcoming release) and created a setup from the
current code base. Would you mind testing this unofficial installer:

http://threekings.tk/tmp/Git-Issue491.exe

[msy...@googlecode.com]

Comment #3 on issue 491 by ajriesgo: Multiple errors when installing

msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

Sschuberth and Johannes, thanks a lot for your quick replies. I have just
tried the test installer provided by sschuberth, and I still get the same
errors. This is what happens:

Running Git Bash:

ERROR: HOME directory 'C:\Documents and Settings\μngel' doesn't exist!

This is an error which might be related to msysGit issue 108.

You might want to set the environment variable HOME explicitly.

Falling back to / (C:/Archivos de programa/Git).

Welcome to Git (version Issue491)

Run 'git help git' to display the help index.
Run 'git help <command>' to display help for specific commands.

ngel
μngel@ARM /c/Documents and Settings/Ángel
$

Then I opened Git Gui and accessed one of the test repositories I created.
I opened the Edit->Options dialog and emptied the user name and email
address fields, saved the changes and restarted Git Gui in order to
reproduce the same original scenario, where neither the user name nor the
email address had been set yet. I created a new dummy file text4.txt in the
root directory of the repository. Using Git Gui I managed to add this new
file to the staging area, but when I tried to commit it (by clicking on
the 'Commit' button), the error I described in my first post appeared
again. I have attached an image (error1.jpg). Then I opened the
Edit->Options dialog and I tried to modify the encoding settings and save
the new settings. An error occurred because the program tries to access the
directory C:\Documents and Settings\ængel (note that this time it is ængel,
not μngel). The workaround to avoid these two errors consists in setting a
user name and a valid email address in the options. It is only when those
fields are blank that msysGit seems to try to generate default values based
on my Windows user name. Note that I can manage to save most of the
settings in the Edit->Options dialog. It is only with some of the settings
(at least the default encoding in the text files) that I get the error
displayed in error2.jpg.


Attachments:
error1.jpg 28.2 KB
error2.jpg 49.9 KB

[msy...@googlecode.com]

Comment #4 on issue 491 by johannes.schindelin: Multiple errors when

installing msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

What happens if you set the HOME variable explicitely, as suggested by the
error in Git Bash?

[msy...@googlecode.com]

Comment #5 on issue 491 by ajriesgo: Multiple errors when installing

msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

I've just done that by editing the git/etc/profile file and replacing the
line 'HOME="$USERPROFILE"' with 'HOME="C:\Archivos de profgrama\Git\home"'
(and creating that 'home' subdirectory). What happens in that case is that
the initial error in the Git Bash console about the HOME directory not
existing doesn't happen, which makes sense, but then I still get the 'ngel'
and 'μngel' strings in the command line (see error3.jpg attached). On the
other hand, the error of image error1.jpg continues happening. The
workaround for that error, which I've mentioned before, consists in setting
a valid user name AND a non-empty e-mail address. Otherwise, committing
fails. I haven't been able to reproduce error2.jpg now, but that error only
happens sometimes and I'm not sure about the actual steps that lead to it,
so I cannot say whether that one is related to there being an explicit HOME
directory or if I simply need to continue working and doing something else
for it to happen again.

I can continue testing, but anyway a name like "Ángel" should never
become "ngel", "μngel" or "ængel". There is another issue about Unicode
file names not being supported (issue 80), which is closely related to this
one, and I have also found that I can't create a file called test五.txt and
stage it and commit it successfully, so these character encoding issues
seem to be quite general. Basically, it seems that char-based strings are
sometimes codified as utf-8 and sometimes with the locale that Windows uses
for the ANSI-style functions (and C run-time library functions like fopen).
Unless the general issue of mixing up string representations is addressed,
such problems will continue happening. As an example, the Win32 function
GetUserNameA should never be used internally if we want to support Unicode
user names.

Attachments:
error3.jpg 16.5 KB

[msy...@googlecode.com]

Comment #6 on issue 491 by johannes.schindelin: Multiple errors when

installing msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

FWIW I did not mean that you should edit /etc/profile. I meant that you
should set the HOME environment variable (in case you do not know how to do
that on Windows: right-click on the "My Computer" symbol,
choose "Properties" from the context menu, and then click on
the "Environment" tab).

[msy...@googlecode.com]

Comment #7 on issue 491 by ajriesgo: Multiple errors when installing

msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

I understood what you meant, but I thought what I did it was equivalent to
setting the HOME environment variable. I didn't do it that way because I
thought I'd installed msysgit in a mode that doesn't make use of global
environment variables (but I may have misunderstood the install options).
Anyway, I can't see what relation this has with the issue.

[msy...@googlecode.com]

Comment #8 on issue 491 by johannes.schindelin: Multiple errors when

installing msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

ajriesgo: it is quite possible that the HOME variable is interpreted before
/etc/profile is even read, and that subsequent changes to the HOME variable
are ignored. It is also quite possible that setting the HOME variable fixes
the encoding issues you see (at least way back when I tested it, it
helped). This would help understanding the issue, so that you could work on
a patch that fixes the issue by fixing the encoding even if the user did
not set HOME explicitly.

[msy...@googlecode.com]

Comment #9 on issue 491 by ajriesgo: Multiple errors when installing

msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

Ok, I've set the HOME environment variable. If I set it to the
value "C:\Archivos de programa\Git\Home" (and create that subdirectory) Git
Bash works in the same way as when I edited the profile file (comment 5
above; error3.jpg). Using Git GUI to stage and commit a file gives me the
non-utf-8 message error (error1.jpg in comment 3 above). The workaround for
that is to set the user name and email address explicitly, as I've
mentioned before.

If I rename both the environment variable and the directory to "C:\Archivos
de programa\Git\Hóme" (note the accented ó), then Git Bash does not find
the directory and tries to fall back on my Windows user directory, which it
doesn't find either and displays the familiar "ERROR: HOME
directory 'C:\Documents and Settings\μngel' doesn't exist!".

To sum up my tests, if the Windows user name has non-ASCII characters:

1. If the HOME environment variable is set to an ASCII path, and that path
exists, Git Bash finds it.

2. If there is no HOME environment variable and the HOME constant
in "git/etc/profile" is set to an ASCII path, and that path exists, Git
Bash finds it. The behaviour seems to be exactly the same as in 1.

3. If the HOME environment variable is set to a non-ASCII path
(like "C:\Höme"), Git Bash doesn't find it. It attempts to fall back on the
C:\Documents and settings\[User Name] directory, but it fails to find it
because it mangles the non-ASCII characters in the [User Name].

4. After the HOME system environment variable (or the constant in
git/etc/profile) is set to a fully ASCII path, Git Bash finds the home
directory, but it still mangles the user name in the displayed messages (it
gets one correct out of three; see error3.jpg in comment 5 above).

5. The first time I run Git GUI, when I try to stage and commit a file,
there is an error that says that there are invalid utf-8 characters in the
commit message. The workaround for this consists in opening the
Edit->Options dialog and entering a valid user name and an email address in
the corresponding edit boxes. It seems that Git is composing an extended
message based on my message plus the user name and email. Because the
fields are empty, it seems to try to generate default values out of the
Windows user name, but it probably retrieves that as an ISO-8859-1 string
on my system, and then another part of the code expects that to be valid
utf-8, but it isn't.

6. If the user name and email fields are left empty, I've also come across
the error shown in the image "error2.jpg" above when trying to modify some
options in the Edit->Options dialog. However, this error doesn't always
happen, and I'm not sure about the exact steps that lead to it.

7. After I've set the HOME variable and the user name and email address, I
can work without any problems, except for the next one.

8. Non-ASCII paths and file names can be staged and committed if they are
made up of characters in the local Windows encoding (I think that's
ISO-8859-1 for Spanish, English and most Western European Windows systems).
So, I've been able to stage and commit a file with the name "cañón.txt".
However, if the path contains Unicode characters impossible to represent in
the local Windows encoding, like "C:\My tests\test五.txt", such files
appear mangled in the Git GUI window, and give a "no such file or
directory" error when I attempt to stage them.

I insist that there seems to be an issue with 1/the Windows layer
retrieving system information in char-based strings in the local encoding,
and 2/ passing those strings over to the non-Windows parts of the code,
which probably expect well-formed utf-8 strings. The only solution I can
think for that is to ensure that all Win32 text functions be used in their
wide-character form (by defining the UNICODE macro globally or explicitly
using the ugly W-terminated function names). Those wide strings can then be
converted into utf-8 char-based strings before passing them around to the
non-Windows bits of code. There is a nasty problem, though, which I came
across in the past when trying to fix this same problem at a company I used
to work with, which is that C run-time functions like fopen will fail
miserably in Windows with utf-8 strings because there doesn't seem to be a
way to specify a utf-8 locale for the Microsoft C run-time library. If
that's still the case, the only solution I can think of would consist in
adding a level of indirection to functions such as fopen, which would be
replaced with a "my_nifty_fopen" wrapper that would have a trivial
non-Windows implementation that simply forwards the call to fopen and a
Windows implementation that would convert the utf-8 string to a Unicode
wchar_t string and then call wfopen (converting to the local encoding
before calling fopen would work with "cañón.txt", but not with "test五.txt"
so it wouldn't be a complete solution). I don't know how the git/msysgit
code is organised, and I don't know if these things are already being done
or if an alternative approach has been thought out. Anyway, I thought I
would share my experience with these issues in case these ideas help.

Feel free to ask me about anything that's still not clear in this summary
or if there are any further tests you guys would like me to try.

Ángel José Riesgo

[msy...@googlecode.com]

Comment #10 on issue 491 by ajrie...@gmail.com: Multiple errors when

installing msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

It's been nearly two years since I reported this issue, and I'm glad to
find that it has largely been fixed thanks to the excellent Unicode support
provided in the latest version 1.7.10.msysgit.1. The Git bash console now
recognises my user name with an accented letter and Git Gui can now cope
with extended characters in file names, paths and comments. I attach an
image of my attempt to commit a file called "C:\test\tést-中文-عربية\cañón
- 信 - رسالة.txt".

The only issue I've found so far, which can also be seen in the attached
image, is that the Git Bash console doesn't show these paths with non-ASCII
characters correctly and displays a bunch of question marks instead of the
extended characters. Anyway, it seems we're getting closer to full Unicode
support.

Great work!

Attachments:
screenshot.jpg 100 KB

[msy...@googlecode.com]

Updates:
Status: Fixed

Comment #11 on issue 491 by sschu...@gmail.com: Multiple errors when

installing msysgit with a Windows user name that has non-ASCII characters
http://code.google.com/p/msysgit/issues/detail?id=491

First of all, please note that the issue tracker is not in use anymore, any
issues should be reported / discussed on the mailing list.

Secondly, as a last remark before closing this issue, make sure you're
using a TrueType font for the console, and maybe try changing the console
font, and report back on the mailing list whether that solves your issue.
Thanks!

[karste...@dcon.de]

msy...@googlegroups.com wrote on 16.04.2012 11:07:57:
[...]

> The only issue I've found so far, which can also be seen in the attached  
> image, is that the Git Bash console doesn't show these paths with non-ASCII  
> characters correctly and displays a bunch of question marks instead of the  
> extended characters. Anyway, it seems we're getting closer to full Unicode  
> support.
>
> Great work!
>
> Attachments:
>    screenshot.jpg  100 KB
>

In the case of your screenshot, the culprit actually seems to be ls rather than bash. Try 'ls --show-control-chars' or 'ls | cat' for Unicode output.

The reason is that msys.dll is simply too old and doesn't have all the fancy wide-char support functions, e.g. to calculate the width of multi-byte character on screen.

[Ángel José Riesgo]

Thanks for the suggestion. I've found that by using 'ls --show-control-chars' and 'ls | cat' I can see the accented letters that are part of my local code page, but not other Unicode characters, like the Arabic and Chinese characters, which are still displayed as '?????'.

I've found that this problem of supporting extended characters in the local code page but not Unicode characters in general also affects the commit message edit field when I reopen the Git Gui application. I can type Chinese or Arabic there, but the message I had when I previously closed the application ("我很高兴 انا مسرور خدا ¡Ahora sí...!") hasn't been restored correctly. Whe re-opening the application it is retrieved as "???????? ????? ???¡Ahora sí...!".

It appears as if the Unicode text is being transformed into the local code page encoding (ISO-8859-1, I suppose) at some point, which would explain why the Arabic and Chinese parts get mangled.

I attach a screenshot of this latest test. 

[Ángel José Riesgo]

Actually, I've just realised that when using Git Bash, the Unicode characters don't appear as question marks, but as square blocks, which probably indicates that it is just that the font doesn't support those characters. I will try to change the font used by Git Bash and see how it goes. Sorry for the misleading information in my previous message.

[Pat Thoyts]

2012/4/19 Ángel José Riesgo <ajri...@gmail.com>:

>
> I've found that this problem of supporting extended characters in the local
> code page but not Unicode characters in general also affects the commit
> message edit field when I reopen the Git Gui application. I can type Chinese
> or Arabic there, but the message I had when I previously closed the
> application ("我很高兴 انا مسرور خدا ¡Ahora sí...!") hasn't been restored
> correctly. Whe re-opening the application it is retrieved as "???????? ?????
> ???¡Ahora sí...!".
>
> It appears as if the Unicode text is being transformed into the local code
> page encoding (ISO-8859-1, I suppose) at some point, which would explain why
> the Arabic and Chinese parts get mangled.

If you type in a commit message and exit git gui, it saves that
messsage to .git/GITGUI_MSG so you don't have to re-enter the message
next time. However, the code for this does not specify the encoding of
this file so it will default to the current system encoding which can
fail to correctly handle unicode strings. The following patch should
sort this out for you.

From 207b7dbf6c94adcaba46b6af1bb515a23dce3927 Mon Sep 17 00:00:00 2001
From: Pat Thoyts <patt...@users.sourceforge.net>
Date: Thu, 19 Apr 2012 11:19:58 +0100
Subject: [PATCH] git-gui: preserve commit messages in utf-8
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The commit message buffer is automatically preserved to a local file
but this uses the system encoding which may fail to properly encode
unicode text. Forcing this file to use utf-8 preserves the message
correctly.

Reported-by: Ángel José Riesgo <ajri...@gmail.com>
Signed-off-by: Pat Thoyts <patt...@users.sourceforge.net>
---
git-gui.sh | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/git-gui.sh b/git-gui.sh
index ba4e5c1..2360cb1 100755
--- a/git-gui.sh
+++ b/git-gui.sh
@@ -1463,7 +1463,7 @@ proc rescan {after {honor_trustmtime 1}} {
(![$ui_comm edit modified]
|| [string trim [$ui_comm get 0.0 end]] eq {})} {
if {[string match amend* $commit_type]} {
- } elseif {[load_message GITGUI_MSG]} {
+ } elseif {[load_message GITGUI_MSG utf-8]} {
} elseif {[run_prepare_commit_msg_hook]} {
} elseif {[load_message MERGE_MSG]} {
} elseif {[load_message SQUASH_MSG]} {
@@ -1549,7 +1549,7 @@ proc rescan_stage2 {fd after} {
fileevent $fd_lo readable [list read_ls_others $fd_lo $after]
}

-proc load_message {file} {
+proc load_message {file {encoding {}}} {
global ui_comm

set f [gitdir $file]
@@ -1558,6 +1558,9 @@ proc load_message {file} {
return 0
}
fconfigure $fd -eofchar {}
+ if {$encoding ne {}} {
+ fconfigure $fd -encoding $encoding
+ }
set content [string trim [read $fd]]
close $fd
regsub -all -line {[ \r\t]+$} $content {} content
@@ -2266,6 +2269,7 @@ proc do_quit {{rc {1}}} {
&& $msg ne {}} {
catch {
set fd [open $save w]
+ fconfigure $fd -encoding utf-8
puts -nonewline $fd $msg
close $fd
}
@@ -3835,7 +3839,7 @@ if {[is_enabled transport]} {
}

if {[winfo exists $ui_comm]} {
- set GITGUI_BCK_exists [load_message GITGUI_BCK]
+ set GITGUI_BCK_exists [load_message GITGUI_BCK utf-8]

# -- If both our backup and message files exist use the
# newer of the two files to initialize the buffer.
@@ -3872,6 +3876,7 @@ if {[winfo exists $ui_comm]} {
} elseif {$m} {
catch {
set fd [open [gitdir GITGUI_BCK] w]
+ fconfigure $fd -encoding utf-8
puts -nonewline $fd $msg
close $fd
set GITGUI_BCK_exists 1
--
1.7.10.msysgit.1

[Ángel José Riesgo]

Well, that was quick! Thanks for the patch. It's not a very serious problem, and since I type all my commit messages in English (or occasionally Spanish) I don't really need this functionality right now. I simply wanted to let you guys know about these issues, just to help the development. The lack of support for Unicode file names in Git for Windows gave me problems in the past and I think Git should handle Unicode smoothly, so I've performed these nasty tests mixing Chinese, Arabic and Spanish accented letters just to check if msysgit is able to handle Unicode correctly now or if there were still any problems left.

That problem with the saved commit message has been the only problem regarding Unicode support that I've found with the latest version of msysgit. With your patch that fixes this, everything should be working fine with Unicode paths and commit messages now. Thanks for the great work.

[Ángel José Riesgo]

I can confirm that the Git Bash problem when using 'ls | cat' instead of bare 'ls' is simply a font display problem. Basically, the fixed-width font used by the console supports a limited number of characters. I tried changing it to a Unicode font, but it's not easy because Git Bash, like the Windows command prompt, will only accept fixed-width fonts. In the end, the furthest I got was to enable the DejaVu Sans Mono font, which supports Arabic letters in isolated form and arranged form left to right (come to think of it, I guess there's no way you can render proper Arabic ligatures with a fixed-width font). By enabling that font in the Windows console (I added a value ['000', 'DejaVu Sans Mono'] to the registry key "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Console\TrueTypeFont"), I was able to select the DejaVu font through the properties dialog in both the Windows command prompt and the Git Bash console).

The attached screenshot shows the Arabic letters in the directory name in the Git Bash window and the Windows command prompt after setting the DejaVu Sans Mono font. The Chinese characters appear as square blocks simply because that font doesn't have the glyphs for those characters. The only pronlem I've found with the Git Bash window is that, unlike the Windows prompt, it doesn't allow me to type accented letters with my Spanish keyboard.