supplementary character rendering problem, gvim with MacVim 7.4(87)

[Kenneth Reid Beesley]

Problem: supplementary character rendering problem, MacVim 7.4(87)

MacVim is running for me (gvim flavor), but supplementary characters look like they are displayed double-width.
These same supplementary characters, from the same (mono) font, used to work. Something has changed,
and I haven’t quite tracked it down. Any help would be much appreciated.

Background: Running OS X Yosemite 10.10.5
MacVim 7.4(87) installed in /Applications/MacVim.app

Running gvim
‘which gvim’ returns ~/bin/gvim
which is just a pointer to ~/bin/mvim, which was supplied along with MacVim.app

Invoking ‘gvim’ from the command line does indeed launch MacVim Custom Version 7.4(87)

My ~/.gvimrc file does get run, and includes the command
set anti guifont=BrighamVu\ Sans\ Mono:h12

This font does get selected (I checked with ‘:set guifont’ inside MacVim)
Normal editing using 16-bit characters from the Basic Multilingual Plane works as expected.

The BrighamVuSansMono.ttf font resides in my ~/Library/Fonts/. It is the DejaVu_Sans_Mono.ttf
font that I augmented with Deseret Alphabet glyphs from the range U+10400 — U+1044F.
I.e. I added new glyphs in the supplementary area.

This font has worked for me for a long time. But now something has changed. Perhaps my fault, but I’m stumped.

Behavior:
Now when I enter a character from the Deseret Alphabet range, it gets rendered on the gvim page as if it were double width.
I.e. the Deseret glyphs from the modified font get rendered on the screen, but it _looks_ like there is a space between the characters.
But there is no space (unless a space is actually typed). Advancing the pointer, using the usual l (l.c. L) command in command
mode progresses from one Deseret character to the next, and the correct supplementary code point value is displayed on the bottom
line for each character.

It’s just the _display_ of the supplementary characters that is the problem. Each Deseret (supplementary) character glyph
looks like it has a space after it.

A similar problem occurred when I first upgraded from 7.3 to 7.4, and it was solved by opening
MacVim > Preferences > Advanced
and UNticking “Use Core Text renderer” (Thanks Bjorn)

This option is still unticked. (Ticking it doesn’t help.)

I’m stuck. Any help would be appreciated.

Thanks,

Ken


********************************
Kenneth R. Beesley, D.Phil.
PO Box 540475
North Salt Lake UT 84054
USA

[René Köcher]

Hi Ken,

I can confirm your findings and extend them to the CoreText-renderer.
My first guess was that we might have introduced a regression when the ligatures support was integrated into MacVim but since you're reporting the same without CT enabled it looks like something else is moot..
(Ligatures are only suppportrd with CT enabled)

For the record I'm observing the same behavior with:

MacVim Preferences -> Advanced -> Use CoreText-Renderer [X]

set guifont=Fira\ Code:h10
set mac ligatures

The macligatures doesn't influence the outcome, I just included it for completeness.

Regards,
René

[Shirk]

OK, I did some bisecting and narrowed the problem down to this commit:

81a64f5848676778cd76d1291a8af1fe562b556f is the first bad commit
commit 81a64f5848676778cd76d1291a8af1fe562b556f
Author: Kazuki Sakamoto <sakamoto@xxxxxxxxx>
Date: Thu Nov 5 09:17:46 2015 -0800

Support Emoji<U+1F431> #65

:040000 040000 a75a1fd7dcb1d7e70acf245502310d7cba61df55 bc5375ae4ac8484ea1b50f522765df8091e05c0c Makefile src

Reverting 81a64f restores the supplementary characters to their old width but also breaks emojis.
I'm going to open a new issue on github so we can see if there is a fix for both worlds.

Cheers,
René

[Tony Mechelynck]

If this means version 7.4.087 of the underlying Vim code, then it is just over a year old (11/11/2013) and in that time many water has gone under the bridge: the current patchlevel is 7.4.979. You can check it by running ":version" (without the quotes of course), mone says:

VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Dec 19 2015 17:08:28)
Included patches: 1-979

as its first two lines. ("Included patches: 1-979" means it's 979 patchlevels "higher" than the first "stable" 7.4 release.) So the first action I would recommend would be to install the most recent MacVim you can find.

This said, the ranf U+10400-U+1044F is the official Deseret range (see http://www.unicode.org/Public/8.0.0/ucd/Blocks.txt and search for what to Vim would be the pattern /^10400\./), but before it was adopted there used to be a "temporary Deseret range" in one of the UserCode Areas.
See also http://www.unicode.org/charts/PDF/U10400.pdf for a look at what these characters look lite.

I still use the Bitstream Vera Sand Mono font and I have no problems with it; however I don't use it for Deseret, only Latin, Cyrillic, and more rarely Greek and Arabic. I tried switching to FreeMono but strangely enough Vim is the only application which misplaces that font's composing characters so I've come back.

The Unicode ranges are periodically updated, but only by addition and IIUC the Vim code is periodically updated so it can use the proper values for singlewidth/doublewidth, for what is the canonical case conjugate of what, what is a printing character and what is as yet unassigned, etc.

Best regards,
Tony.

[Kenneth Reid Beesley]

> On 24Dec2015, at 11:23, Tony Mechelynck <antoine.m...@gmail.com> wrote:
>
> On Monday, December 21, 2015 at 2:50:01 AM UTC+1, Kenneth R. Beesley wrote:
>> Problem: supplementary character rendering problem, MacVim 7.4(87)
>>
>> MacVim is running for me (gvim flavor), but supplementary characters look like they are displayed double-width.
>> These same supplementary characters, from the same (mono) font, used to work. Something has changed,
>> and I haven’t quite tracked it down. Any help would be much appreciated.
>>

>> <snip>

>
> If this means version 7.4.087 of the underlying Vim code, then it is just over a year old (11/11/2013) and in that time many water has gone under the bridge: the current patchlevel is 7.4.979. You can check it by running ":version" (without the quotes of course), mone says:
>
> VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Dec 19 2015 17:08:28)
> Included patches: 1-979

Hi Tony,

Thanks for the response.

It’s snapshot 87. I just downloaded a new version that contains patches 1-972, and I see the same
problem. I can fix the problem by simply reverting back to 7.3.

The problem that I reported with 7.4 shapshot 87 was confirmed by René Kōcher, who wrote

> This said, the ranf U+10400-U+1044F is the official Deseret range (see http://www.unicode.org/Public/8.0.0/ucd/Blocks.txt and search for what to Vim would be the pattern /^10400\./), but before it was adopted there used to be a "temporary Deseret range" in one of the UserCode Areas.
> See also http://www.unicode.org/charts/PDF/U10400.pdf for a look at what these characters look lite.

I don’t think that the “temporay Deseret Range” is relevant here. My glyphs are in the official Deseret Range,
and always have been, in a
monospace font that used to work, but now the glyphs get rendered as if they had a space after them. (There
is no space in the buffer, unless you actually enter a space. It just appears to be a rendering problem for any
glyph with a supplementary code point value.) And it’s not just Deseret characters; I see the same broken
behavior for rendering Shavian characters, which are also in the supplementary area.

>
> I still use the Bitstream Vera Sand Mono font and I have no problems with it; however I don't use it for Deseret, only Latin, Cyrillic, and more rarely Greek and Arabic. I tried switching to FreeMono but strangely enough Vim is the only application which misplaces that font's composing characters so I've come back.

The issue seems to be tied to the rendering of characters from the supplementary area. So you won’t see it with Latin,
Cyrillic, Greek or Arabic characters, which are all in the BMP. Using something like Fontforge, try copying a glyph to the supplementary
area of your font of choice (at some code point value HHHHHHHH) and then entering it in a gvim buffer using Ctrl-V UHHHHHHHH
See how it’s rendered.

>
> The Unicode ranges are periodically updated, but only by addition and IIUC the Vim code is periodically updated so it can use the proper values for singlewidth/doublewidth, for what is the canonical case conjugate of what, what is a printing character and what is as yet unassigned, etc.

Again, I don’t think that it has anything to do with updated Unicode ranges. See the note above by René
about how a commit to support emojis apparently broke the rendering of supplementary characters, producing
the rendering problem that I reported.

Best wishes, and thanks for all your efforts to support a great editor,

Ken


>
> Best regards,
> Tony.
>
> --
> --
> You received this message from the "vim_mac" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups "vim_mac" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_mac+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[René Köcher]

Merry Christmas everyone,

FYR the github issue is here: https://github.com/macvim-dev/macvim/issues/184

Cheers,
René

[Tony Mechelynck]

On Friday, December 25, 2015 at 8:34:20 AM UTC+1, Kenneth R. Beesley wrote:
[...]

> I don’t think that the “temporay Deseret Range” is relevant here. My glyphs are in the official Deseret Range,
> and always have been, in a
> monospace font that used to work, but now the glyphs get rendered as if they had a space after them. (There
> is no space in the buffer, unless you actually enter a space. It just appears to be a rendering problem for any
> glyph with a supplementary code point value.) And it’s not just Deseret characters; I see the same broken
> behavior for rendering Shavian characters, which are also in the supplementary area.
>
>
>
> >
> > I still use the Bitstream Vera Sand Mono font and I have no problems with it; however I don't use it for Deseret, only Latin, Cyrillic, and more rarely Greek and Arabic. I tried switching to FreeMono but strangely enough Vim is the only application which misplaces that font's composing characters so I've come back.
>
> The issue seems to be tied to the rendering of characters from the supplementary area. So you won’t see it with Latin,
> Cyrillic, Greek or Arabic characters, which are all in the BMP. Using something like Fontforge, try copying a glyph to the supplementary
> area of your font of choice (at some code point value HHHHHHHH) and then entering it in a gvim buffer using Ctrl-V UHHHHHHHH
> See how it’s rendered.

Rather than construct a character of my own, I entered existing characters from the "Deseret" range (10400-1044F) and from the "Coptic Epact Numbers" range (102E0-102FF). Most of these codepoints have no glyph in my Bistream Vera Sans Mono font, so I get a "fallback glyph" whic is just the codepoint number in octal, and seems to cover to screen cells, but here the cursor moves onkly by one cell per character. In particular, the glyph for U+10400 DESERET CAPITAL LETTER LONG I is defined, and when I put it between two dashes it has no empty call on either side.

However I am on Linux, so I suppose that your problem lies with the rendering engin used on the Mac rather than with Vim's mostly cross-platform logic.

(Even though I'm on Linux, I still occasionally watch the vim_mac group, in order (among others) to catch (or even detect) problems which might be cross-platform and redirect them to the vim_use and vim_mac groups. In this case you seem to have found a genuinely "Mac-only" problem so I don't think I'll say much more on this thread.


Best regards,
Tony.

[Kenneth Reid Beesley]

Hello Tony,

Many thanks for keeping an eye on things, even in the Mac world.

Updating today to the very latest version of Macvim, using

$ brew install --HEAD macvim-dev/macvim/macvim

gave me Macvim 7.4 with patches 1-979, and that solves
the problem for me. My glyphs in the supplementary area
are once again rendered as they should be.

Thanks to René and everyone else who helped.

Ken