Re: [vim/vim] patch 9.0.0696: it is unclear if the +rightleft and +arabic features are used (fbdce18)
Yee Cheng Chin
Hi @brammool, I have definitely received bug reports on Arabic support for MacVim before. See macvim-dev/macvim#696 and macvim-dev/macvim#800. Admittedly I kind of punted on it because I don't speak the language and it requires a fair amount of context to understand how to implement / fix properly but I was planning to do a deeper dive into it to make sure MacVim (which is basically just a downstream fork of Vim with Mac GUI) could deal with Arabic properly.
With this feature turned off, would Arabic be completely broken? That seems not ideal. That said, I am still trying to figure out what the current state of things are. For example, if you type the texts (from the bug report above) فارسی English in Vim, in the Apple terminal or Ubuntu terminal it will look correct (even though Vim doesn't have arabic set), but in iTerm it won't. I think that's because the terminals are automatically doing right-to-left conversions in the rendering, but that actually leads to weird issues because in those terminals, set rightleft will actually do the wrong thing, and also if you make a split window in the middle of the text it will get screwed up as well. This is also how web browsers and text editors work it seems. But it does seem like the kind of thing that has inconsistent support and I'm having trouble figuring out what the proper behavior is, especially for mixed English/Arabic texts. There is termbidi but I can't quite figure out what it does.
I'm just wondering if this deserves a little more thought? Just some background research on people asking Arabic sentiment:
- VSCode: microsoft/vscode#145122
- iTerm: https://gitlab.com/gnachman/iterm2/-/issues/1611#note_107406363
- MacVim: (see above)
- Sublime Text: https://stackoverflow.com/questions/25603500/arabic-language-with-sublime-text-editor
- Emacs: https://www.gnu.org/software/emacs/manual/html_node/emacs/Bidirectional-Editing.html
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Bram Moolenaar
> Hi @brammool, I have definitely received bug reports on Arabic support
> and https://github.com/macvim-dev/macvim/issues/800. Admittedly I kind
> of punted on it because I don't speak the language and it requires a
> fair amount of context to understand how to implement / fix properly but
> I was planning to do a deeper dive into it to make sure MacVim (which is
> basically just a downstream fork of Vim with Mac GUI) could deal with
> Arabic properly.
if people switched to another editor or just gave up.
It does require at least some knowledge of right-to-left text to work on
this. And Arabic shaping requires knowing Arabic, at least the basics.
> With this feature turned off, would Arabic be completely broken? That
> seems not ideal. That said, I am still trying to figure out what the
> current state of things are. For example, if you type the texts (from
> the bug report above) `فارسی English` in Vim, in the Apple terminal or
> Ubuntu terminal it will look correct (even though Vim doesn't have
> `arabic` set), but in iTerm it won't. I think that's because the
> terminals are automatically doing right-to-left conversions in the
> rendering, but that actually leads to weird issues because in those
> terminals, `set rightleft` will actually do the wrong thing, and also if
> you make a split window in the middle of the text it will get screwed up
> as well. This is also how web browsers and text editors work it seems.
> But it does seem like the kind of thing that has inconsistent support
> and I'm having trouble figuring out what the proper behavior is,
> especially for mixed English/Arabic texts. There is `termbidi` but I
> can't quite figure out what it does.
idea if that is in any way workable. The text may show up correctly,
but is it possible to do editing, such as deleting and inserting text?
Does Visual mode show properly?
In the GUI Vim would have to do this. The current right-left
implementation is primitive, it's not bidi.
> I'm just wondering if this deserves a little more thought? Just some background research on people asking Arabic sentiment:
> - iTerm: https://gitlab.com/gnachman/iterm2/-/issues/1611#note_107406363
> - MacVim: (see above)
> - Sublime Text: https://stackoverflow.com/questions/25603500/arabic-language-with-sublime-text-editor
> - Emacs: https://www.gnu.org/software/emacs/manual/html_node/emacs/Bidirectional-Editing.html
I looked at a few. One thing is clear: When editing code and the
comments are in a right-to-left language bidi support is needed. I
guess this is the most common usage, and we currently do not support it
(although you could switch 'rightleft' on and off constantly).
I was planning to leave the current state for a while, hoping some
people speak up. Perhaps someone even volunteers to work on it. But if
nothing happens I think we can drop the features, since I expect them to
be half-broken already.
--
A vacation is a period of travel during which you find that you
took twice as many clothes and half as much money as you needed.
/// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Yee Cheng Chin
Right, after around 2019 comments like these stopped coming. I wonder if people switched to another editor or just gave up.
I guess since Vim doesn't have telemetry it could ultimately be a little hard to gauge this kind of stuff (recall the recent Windows XP support discussion), but I think for this, people probably just stop complaining and found some half-broken solution that they kept working with.
In a terminal we can let the terminal do the bidi stuff
Playing around, it does kind of work, but not perfect. And yes for GUI, it wouldn't just work unless the GUI handles it specifically. The other issue is that it really only works if you have set wrap on, since let's say you have a long unwrapped line, there is no way for the GUI (which mostly works as a smart terminal in Vim) or the terminal to know how to handle it properly.
I was planning to leave the current state for a while, hoping some people speak up. Perhaps someone even volunteers to work on it. But if nothing happens I think we can drop the features, since I expect them to be half-broken already.
I think the issue with this is that people already rely on set arabic / set rightleft today. While it's half-broken, it seems to at least get it workable for some people as a primitive way to use Vim with right-to-left languages (it's also not that broken unless you are mixing languages a lot). I feel like disabling it just to acquire feedback could be a surprising breakage for people who just wanted to update Vim and have it work. For example I just found this thread from 1 year ago (https://stackoverflow.com/questions/66381230/how-can-i-get-vim-to-render-arabic-text-correctly) which asked about Arabic support. There is an older but in-depth post (http://andreasmhallberg.github.io/typing-arabic-in-vim/ and reddit thread ) about how to enter Arabic in Vim (it's written in 2017 but last update was 2022-06-22! Interestingly the author found the lack of bidi a bonus). This is clearly something people are interested in.
I agree that the current support for bidi and RTL langauges (Arabic, Persian, Hebrew, and more) are not great in Vim but my fear with just dropping a feature like this is that there are likely people using them despite the jankiness associated with it because they want to use Vim, and editing RTL languages is just a part of the workflow. I think a lot of people are shy about opening issues, especially when they are not really "in" the open-source community so to speak, and so they may have just Googled some StackOverflow question or blog post and just deal with it that way. Also, given that vim_dev and GitHub are primarily English based it may deter some as well.
The other issues with disabling the feature is that it will make it harder to maintain the code, leading to progressive bitrot. Even if a volunteer picks it up later it will be harder to reuse existing rightleft support because it's been unmaintained. I think it would be easier to pick up if the volunteer didn't have to do everything from scratch again.
I think the trend in general for computing is to make sure our software can be better localized and serve more communities. I played around with other text editors and most do have some bidi support, mostly because they reuse existing OS text views so they get it "for free", which Vim doesn't, but then Emacs also has native support for RTL/bidi as well, and can do per-paragraph right-to-left alignment (see this recent EmacsConf video). I'm wondering if there is a way to broadcast this more widely before turning this off? My guess is there would definitely be people interested in making Vim more RTL-compliant, rather than less. Given the large swath of languages that are RTL (Hebrew, Persian, Arabic, and like a dozen others), I think it's a good aspiration at least to try to accommodate the ability to enter and edit them.
I'm also wondering if there's a reason to turn this off now? Incidentally I'm working on a Vim PR to support overlay GUI scrollbars, and did run into the issue of needing to support RTL layouts so I can see the pain, but I think as I said, if the reason to turn it off now is to allow not having to deal with RTL I think it will make it harder to pick this up in the future as more and more features don't work well with it.
I would definitely be interested in taking part in this if I could find native speakers to collaborate with.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Yee Cheng Chin
I think another aspect of this is maybe we could try to provide some bidi support if full-on RTL support is too annoying to support (since it affects everything like virtual text, number column, signs, wrapping, etc). Not sure if it's harder or easier to be honest. My vote would be full RTL support as it would be useful to keep supporting if it already works, but I do understand there are maintenance cost to doing so.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
