How to solve Bidi support in Vim/GVim

[Mostafa Shahverdy]

Dear Vimmers,

Is there any way to solve lack of Bidi sopport for vim/gvim in linux?

Any kind of help would be appreciated.

-- 

Regards,

Mostafa Shahverdy

[C. Meissa]

Hi Mostafa,

Mostafa Shahverdy schrieb am 11.11.2012 08:33:33:
> Is there any way to solve lack of Bidi sopport for vim/gvim in linux?

This is more a matter of the terminal you are using.
Vim on mlterm¹ works ok with RTL languages for me.

Cheers
CM

¹ http://sourceforge.net/projects/mlterm/screenshots/70031

[Mostafa Shahverdy]

But how about gvim?

> --
> You received this message from the "vim_use" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>


--
Regards,
Mostafa Shahverdy

[Ben Fritz]

On Sunday, November 11, 2012 5:10:53 AM UTC-6, Mostafa Shahverdy wrote:
> But how about gvim?
>
>

If a feature works in ANY terminal it should work in gvim.

What is your *specific* problem with bidi language support?

[Tony Mechelynck]

On 11/11/12 17:21, Ben Fritz wrote:
> On Sunday, November 11, 2012 5:10:53 AM UTC-6, Mostafa Shahverdy wrote:
>> But how about gvim?
>>
>>
>
> If a feature works in ANY terminal it should work in gvim.

No, Ben: Bidi works in Vim in mlterm because the terminal handles bidi
(and Vim sees at startup that 'term' is mlterm, and sets 'termbidi').
Gvim is not that clever: in gvim you can set a whole window to display
RTL (:setlocal rightleft), but you cannot display LTR and RTL text in a
single window depending on which script is used where. The mlterm
terminal can do that, and it will happily display Hebrew and Arabic (but
not Arabic-Hindic digits) RTL and Latin, Cyrillic, Chinese, etc.
(including Arabic-Hindic digits) LTR, even on the same line.

>
> What is your *specific* problem with bidi language support?
>

AFAIK, in gvim there is no way to display Unicode bidirectional text,
whose characters are stored in logical sequence, from start to end of
each word or sentence, but must be displayed either LTR or RTL, possibly
mixing both on a single line, as e.g. when quoting Arabic or Hebrew text
within an English (or French or Russian or�) sentence (or English text
within Hebrew or Arabic, or even in Arabic-only text when using
numerals, even if written with Arabic-Hindic digits).

If you want to see LTR and LTR text properly displayed at the same time,
I recommend (on Linux) using Vim in console mode in an mlterm terminal.
It is possible to edit bidi text in gvim but it will be displayed either
RTL or LTR (depending on the window-local 'rightleft' setting) but not
both in a single window.


Best regards,
Tony.
--
It is illegal to drive more than two thousand sheep down Hollywood
Boulevard at one time.

[Dotan Cohen]

> The mlterm terminal can do that,
> and it will happily display Hebrew and Arabic (but not Arabic-Hindic digits)
> RTL and Latin, Cyrillic, Chinese, etc. (including Arabic-Hindic digits) LTR,
> even on the same line.
>

The Hindu numerals are LTR, just like the Arabic numerals that
westerners are familiar with.

--
Dotan Cohen

http://gibberish.co.il
http://what-is-what.com

[Mostafa Shahverdy]

Ben: I want to write some web pages with PHP, in Persian.

Tony: How is it so that there can not be any kind of extension to handle BiDi algorithms in Vim itself(and not terminal)?

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--

Regards,

Mostafa Shahverdy

[Ben Fritz]

On Sunday, November 11, 2012 11:40:08 PM UTC-6, Tony Mechelynck wrote:
> On 11/11/12 17:21, Ben Fritz wrote:
>
> > On Sunday, November 11, 2012 5:10:53 AM UTC-6, Mostafa Shahverdy wrote:
>
> >> But how about gvim?
>
> >>
>
> >>
>
> >
>
> > If a feature works in ANY terminal it should work in gvim.
>
>
>
> No, Ben: Bidi works in Vim in mlterm because the terminal handles bidi
>
> (and Vim sees at startup that 'term' is mlterm, and sets 'termbidi').
>
> Gvim is not that clever: in gvim you can set a whole window to display
>
> RTL (:setlocal rightleft), but you cannot display LTR and RTL text in a
>
> single window depending on which script is used where. The mlterm
>
> terminal can do that

My mistake. I've never heard of a feature that works in a terminal but not in gvim. Are there other features that rely on the terminal to work and therefore don't work in gvim? This is the first I've seen.

[Tony Mechelynck]

On 12/11/12 07:37, Mostafa Shahverdy wrote:
> Ben: I want to write some web pages with PHP, in Persian.
>
> Tony: How is it so that there can not be any kind of extension to handle
> BiDi algorithms in Vim itself(and not terminal)?

I suppose that the need has not been felt, or the programming ability
not found, or both. RTL display (pagewise) required somme additions to
Vim (Vim 6.0, I think, I'm not sure); Arabic presentation forms
(displaying correct initial / medial / final / isolate forms, with the
additional complication that laam-alif are always written together in a
single character cell, and that alif, daal, dhaal, raa, zaay and waa are
never bound to the next letter even if there is one in the same word,
forcing that next letter to take an "initial" or "isolate" shape) was
something even more difficult. Taking care of proper full-bidi
(including what to do if 'wrap' is on and line-wrapping happens in the
middle of a word with reversed directionality, and then how to handle
Unicode directional controls U+202A to U+202E, how to display HTML text
when it includes <bdo> elements, or elements with "dir=…" attributes; or
worse, how to handle HTML text with external CSS style sheets specifying
dir=… attributes; and how to move the cursor across changes of
directionality…) I don't think it would be easy.

If you think you know:
- the C language
- the Vim codebase
- the full-bidi conventions
- how they apply differently in different 'fileencoding's
all of that well enough to implement it, I suppose you can try writing
(at first) an unofficial patch (by "a patch" I mean "a set of changes,
possibly in several files, and possibly including the creation of new
source files"). If it works, then maybe Bram will decide to incorporate
it into Vim 8 or Vim 9. I have some textfiles with bidi text, but no
doubt you will be able to devise your own.

But I think doing it well would be a whole can of worms. I'm not sure it
is possible to do it correctly in all cases, including those where part
or all of the directionality is defined in a file external to the one
being edited (e.g.in a style sheet for HTML or similar). It might
require a complete overhaul of not only "ordinary" display but also
syntax highlighting, since how to handle <span dir="rtl">…</span> in
HTML might be better defined in an HTML syntax script than in the "core"
display routines… Not to mention more complicated (recursive) embeddings
such as

<p dir="ltr">english1 <span dir="rtl">farsi2 <span dir="ltr">english3
<span dir="rtl">farsi4</span> english5</span> farsi6</span> english7</p>

(I ought to have shown it with mixed Arabic and Latin script but I'm
lazy) which will have to be displayed (if on a single screen line!) as

english1 6israf english3 4israf english5 2israf english7

(IYSWIM) because the underlying structure (English text quoting a Farsi
phrase which itself quotes an English phrase which etc.) is

<p> → → → → → → → → → → → → → → → → → → → → → → → → </p>
</span> ← ← ← ← ← ← ← ← ← ← ← ← <span>
<span> → → → → → </span>
←<span>

Then if soft-wrapping (due to narrow window width and 'wrap' on) happens
in the middle of the "farsi4" span, we get:

[english1 english3 {raf 2israf
6israf 4is] english5 english7}

where [ or ] is the start of the screen line and { or } its end. Not
obvious, is it?


Best regards,
Tony.
--
"But this has taken us far afield from interface, which is not a bad
place to be, since I particularly want to move ahead to the kludge.
Why do people have so much trouble understanding the kludge? What is a
kludge, after all, but not enough Ks, not enough ROMs, not enough RAMs,
poor quality interface and too few bytes to go around? Have I
explained yet about the bytes?"

[Tony Mechelynck]

On 12/11/12 16:53, Ben Fritz wrote:
[...]

> My mistake. I've never heard of a feature that works in a terminal
> but not in gvim. Are there other features that rely on the terminal
> to work and therefore don't work in gvim? This is the first I've seen.
>

The only other one that I can think of is invoking (by :! or system() or
:shell etc.) an external program which uses curses or ANSI escapes to
move the cursor all about the screen. The "terminal" which gvim passes
to external programs is a "dumb" one, and by that I mean, something that
works as a teletypewriter used to: you can write on successive positions
on a line from the left margin to wherever it is that you end the line,
and then the paper moves to the next line, but you can never move
leftwards on the current line, nor to a line higher up than the current one.

In Console mode, the underlying terminal is still there, it usually
supports curses or at least ANSI, and it is used for :! system() and
:shell, so you can invoke a "full-screen" external console program.


Best regards,
Tony.
--
Murphy's Law is recursive. Washing your car to make it rain doesn't
work.

[Mostafa Shahverdy]

I have experienced some other differences myself and I am sure there are much more. For example consider running external commands using :! in Gvim, then you will see a simulator for terminal running your command, a really terrible simulator. I think lack of BiDi support in Gvim is related to that simulator being really elementary.

--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--

Regards,

Mostafa Shahverdy

[Moshe Kamensky]

* Tony Mechelynck <antoine.m...@gmail.com> [13/11/12 03:30]:

> On 12/11/12 07:37, Mostafa Shahverdy wrote:
> >Ben: I want to write some web pages with PHP, in Persian.
> >
> >Tony: How is it so that there can not be any kind of extension to handle
> >BiDi algorithms in Vim itself(and not terminal)?
>

> If you think you know:
> - the C language
> - the Vim codebase
> - the full-bidi conventions
> - how they apply differently in different 'fileencoding's
> all of that well enough to implement it, I suppose you can try writing (at
> first) an unofficial patch (by "a patch" I mean "a set of changes, possibly
> in several files, and possibly including the creation of new source files").
> If it works, then maybe Bram will decide to incorporate it into Vim 8 or Vim
> 9. I have some textfiles with bidi text, but no doubt you will be able to
> devise your own.
>

I once tried it, and it does not seems very easy. But I think there is
an alternative approach. The way I edit bidi text is the following: I
use rxvt-unicode, which has perl plugins. In particular, it allows
callbacks before (and after) displaying the text, and the plugin may
modify the displayed text. I use it together with a perl interface to
the Fribidi library to display the text correctly. This results in
something similar to the mlterm solution, but with urxvt, which I
prefer.

My point, though, is that if vim allows similar callbacks, i.e., the
possibility to call a vim script before and after the visual text is
displayed, then it will be possible to apply the bidi algorithm on the
level of a vim script, rather than on the level of the code itself. Of
course, it will be much slower, but in my experience with urxvt, it is a
price I am willing to pay for the possibility to edit mixed text
properly. And I imagine it would be useful for other purposes as well.

Moshe

[Dotan Cohen]

Though I cannot code it, I would be very happy to help with the
implementation details. To be honest, Mozilla applications have this
down pat, and that could be used as a reference for cursor behaviour.
In any case, see this page (disclaimer: my own website) explaining how
RTL and LTR spans work together:
http://dotancohen.com/howto/rtl_right_to_left.html

[Mohammad Heydari]

On Tuesday, 13 November 2012 05:00:42 UTC+3:30, Tony Mechelynck wrote:
> If you think you know:
> - the C language
> - the Vim codebase
> - the full-bidi conventions
> - how they apply differently in different 'fileencoding's
> all of that well enough to implement it, I suppose you can try writing
> (at first) an unofficial patch (by "a patch" I mean "a set of changes,
> possibly in several files, and possibly including the creation of new
> source files"). If it works, then maybe Bram will decide to incorporate
> it into Vim 8 or Vim 9. I have some textfiles with bidi text, but no
> doubt you will be able to devise your own.

Tony, Is it possible to integrate a well implemented bidi library like fribidi?

[Yuri Chernyavsky]

I see that vim touch open source app on android does bidi