Proposal/Discussion on decoupling clipboard functionality from X11

[luca.sa...@saccarosium.com]

Hi all,
I would like to make a proposal for decoupling the clipboard
functionality from X11.

I've study the code base for some time but I still need some help on
where and how exactly do this.

The problem
===========

Currently in vim, on Linux, rely on X11 to interact with the system
clipboard. This come with the limitation that vim needs to be compiled
agaist X11. This means that the base vim package, in most distribution,
as no way to interact with the system clipboard which is a really nice,
sometimes essential, feature to have.

If a normal user wants to have this feature they need to compile vim,
which is not something that everyone is comfortable doing, or install
gvim, which a lot of users don't want to have it installed.

Also if you currently on wayland and use the `xterm_clipboard` feature
the startup time is significantly higher due to the start of an xwaland
instance on vim startup.

Possible solutions
==================

1. Rely on external tools

This is what neovim does with his 'providers' [1] mechanism.
Basically they call at runtime external tools to interact with the
system
clipboard (e.g. 'xclip' on Linux X11 or'pbcopy' on OSX).

Advantages:
- Low cost in maintenance and in adding support for new operating
systems,
since every OS either has a CLI clipboard utility already built-in or
available to download.
- It makes trivial to support more niche or esoteric setups, since you
can provide a simple variable to tell vim what commands to call
(example from neovim's implementation):

let g:clipboard = {
\ 'name': 'myClipboard',
\ 'copy': {
\ '+': ['tmux', 'load-buffer', '-'],
\ '*': ['tmux', 'load-buffer', '-'],
\ },
\ 'paste': {
\ '+': ['tmux', 'save-buffer', '-'],
\ '*': ['tmux', 'save-buffer', '-'],
\ },
\ 'cache_enabled': 1,
\ }

Disadvantages:
- The user needs, in the worst case scenario, to install an additional
package (but it is better than the current experience)
- There may be some performance penalties [2].

2. Create our own implementation for dealing with the system clipboard
without rely on X11

Advantages:
- The user is ready to go without needing to install anything
- There shouldn't be a big performance loss

Disadvantages:
- Maintenance in theory will be more difficult since I assume is not an
easy task to support all vim's targets while still support some
more esoteric/less common setups.

Recap
=====

I'm personally in favor of the neovim approach (the first one) but I
would like to have some feedback on this.

[1]: https://neovim.io/doc/user/provider.html#provider
[2]: https://github.com/neovim/neovim/issues/11804

[Tony Mechelynck]

AFAIK on Unix/Linux (as opposed to Windows) the cliboard is a
functionality of X11/Wayland/etc. If you boot up without X11 (or
equivalent), for example at init runlevel 3 or equivalent, there is no
clipboard at all. Similarly, if you log in to a Linux text console
(i.e. after hitting one of Ctrl-Alt-F1 to Ctrl-Alt-F6) you have no
graphical display and no clipboard, even if X11 is otherwise running
(usually on tty7). So on Unix-like systems, you cannot access the
system clipboard without ultimately relying on X11 (or on an X11
substitute such as Wayland) because the clipboard is a function of the
[graphical] display manager. If X11 (or Wayland or…) is not running,
no one has access to a clipboard. If you are logged in to a Linux text
console with no access to the display manager, you have no access to
the clipboard (even if some other user on the same computer has one
because (s)he's logged in to X11).

Morality: T'aint broke, so don' fix it.

Best regards,
Tony.

[Luca Saccarola]

Ok, so option two for a possible solution is not really an option.
Thanks for letting me know.

But in your reply you aren't even acknowledge the first possible
solution witch I think solve a legitimate problem in vim's usability,
not to mention that this approach could unlock clipboard functionality
even on a tty via tmux.

-- Luca Saccarola

[Christian Robinson]

Vim compiled with GPM support has mouse access if the GPM daemon is running

--
Christian J. Robinson

> On Feb 9, 2024, at 5:15 PM, Luca Saccarola <luca.sa...@saccarosium.com> wrote:
>
> 

> --
> --
> You received this message from the "vim_dev" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> --- You received this message because you are subscribed to the Google Groups "vim_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/vim_dev/U16M8S.C5Z8VCSNGT8E1%40saccarosium.com.

[Tony Mechelynck]

Well, I never found it really hard to compile Vim. It just takes a
little getting used to. I used to have a HowTo page about it — two,
actually, one for Unix/Linux and one for Windows — but my ISP took
down my user site, saying "Our engineers will gladly design a user
site for you". I thought I had saved the contents somewhere, but alas,
at the moment I cant't find it back, or I would have attached the
Linux page to this message.

My "smallest" Vim — named vi — is compiled with -x11 and -clipboard,
but that doesn't mean that I can't copy what it displays to the
clipboard: when running in a konsole terminal, I can use the
terminal's clipboard facilities transparently to vi (for instance,
select an area with the mouse ( which copies the selected area to the
XTERM_SELECTION i.e. the "* of my clipboard-enabled gvim) then
Ctrl-Shift-C will make konsole copy the selected area to the clipboard
(i.e. gvim's "+ register)). I'm not sure if konsole can paste into vi
and I think it cannot cut from it though.

If gpm is installed *and* your Vim instance is compiled with
+mouse_gpm, you can use the mouse to move the cursor in Vim in the
Linux console (which has no access to X11, remember) and even to
select text within Vim (and yank, put or delete it within that single
Vim instance if you hit Ctrl-O if necessary to switch from -SELECT- to
-VISUAL-) but I think you cannot paste into or from other programs.

Best regards,
Tony.

[Gary Johnson]

On 2024-02-09, luca.sa...@saccarosium.com wrote:
> Hi all,
> I would like to make a proposal for decoupling the clipboard
> functionality from X11.
>
> I've study the code base for some time but I still need some help on
> where and how exactly do this.
>
> The problem
> ===========
>
> Currently in vim, on Linux, rely on X11 to interact with the system
> clipboard. This come with the limitation that vim needs to be compiled
> agaist X11. This means that the base vim package, in most distribution,
> as no way to interact with the system clipboard which is a really nice,
> sometimes essential, feature to have.
>
> If a normal user wants to have this feature they need to compile vim,
> which is not something that everyone is comfortable doing, or install
> gvim, which a lot of users don't want to have it installed.

There are a couple of misunderstandings here.

Many distributions offer a vim compiled with the X11 feature. The
name of the package may not make that clear. For example, in
Ubuntu, you would install the vim-gtk package. That would install
_one_ vim binary that is linked to several names, including vim and
gvim. The user can use vim or gvim as they choose--it's one binary
that runs with a GUI or a terminal interface depending on the name
by which it is invoked.

Alternatively, if you have a gvim on your system, you can create
a link to it named vim like this:

ln -s /usr/bin/gvim ~/bin/vim

as long as the directory containing the new vim link comes before
the directory containing your non-X11 vim in your PATH.

You could instead create an alias such as this:

alias vim='gvim -v'

Any of those will give you a vim with X11 enabled.

Regards,
Gary

[Luca Saccarola]

On ven, feb 9 2024 at 23:01:50 -08:00:00, Gary Johnson

<gary...@spocom.com> wrote:
>>
> There are a couple of misunderstandings here.
>
> Many distributions offer a vim compiled with the X11 feature. The
> name of the package may not make that clear. For example, in
> Ubuntu, you would install the vim-gtk package. That would install
> _one_ vim binary that is linked to several names, including vim and
> gvim. The user can use vim or gvim as they choose--it's one binary
> that runs with a GUI or a terminal interface depending on the name
> by which it is invoked.
>
> Alternatively, if you have a gvim on your system, you can create
> a link to it named vim like this:
>
> ln -s /usr/bin/gvim ~/bin/vim
>
> as long as the directory containing the new vim link comes before
> the directory containing your non-X11 vim in your PATH.
>
> You could instead create an alias such as this:
>
> alias vim='gvim -v'
>
> Any of those will give you a vim with X11 enabled.
>
> Regards,
> Gary

Yes, I know and I've already acknowledge it:

> If a normal user wants to have this feature they need to [...]

> install gvim, which a lot of users don't want to have it installed.

My point was that a user should be able to use the system clipboard,
via 'set clipboard=unnamedplus', _without_ installing gvim or
recompile vim.

-- Luca Saccarola

[Christian Brabandt]

> Recap
> =====
>
> I'm personally in favor of the neovim approach (the first one) but I
> would like to have some feedback on this.
>
> [1]: https://neovim.io/doc/user/provider.html#provider
> [2]: https://github.com/neovim/neovim/issues/11804

I personally don't like the Neovim approach for several reasons:

- it puts the burden to configure the clipboard to the users. Yes, I
know we can ship pre-configured clipboard provider, but initially this
has to be configured by each user separately (and then later
eventually merged into Vim)
- I think, the clipboard functionality should work out of the box. There
is a reason, why no other application uses a clipboard provider
feature but implements this directly in the application
- I think shelling out to an external clipboard provider causes a
performance penalty, especially when pasting large amount of texts
(a few years ago, we had an issue when using `:g/../d` which caused
a huge slowdown, just because Vim was copying each delete into the X11
clipboard. We added some logic to prevent this, but using a clipboard
provider would make this experience worse)
- There are too many corner cases that need to be considered:
- different encodings of the text being pasted
- different selection types (character- block- or linewise)
- It doesn't help with Vim tiny, since it doesn't support Vim script, so
one should still need to re-compile Vim
- If you are using xclip or xsel you would still require the X11
libraries to work with the clipboard, so I don't think this gains us
anything


There are probably a few others, e.g. in my professional work, I have to
jump to customer environments and work with what is installed there. I
cannot start configuring clipboard provider (or install additional
tools) just to do my work there. In addition, I am not sure, how this
would work on containers and similar environments.

I suppose if one really wants to, users can already make use of
clipboard providers using some Vim plugin (with the restrictions
mentions above), there shouldn't be anything hindering you to use
external tools to interact with the clipboard already. That doesn't
mean, we should get rid of the existing clipboard code however. I am not
sure if such a Vim plugin exists.

> 2. Create our own implementation for dealing with the system clipboard
> without rely on X11
>
> Advantages:
> - The user is ready to go without needing to install anything
> - There shouldn't be a big performance loss
>
> Disadvantages:
> - Maintenance in theory will be more difficult since I assume is not an
> easy task to support all vim's targets while still support some
> more esoteric/less common setups.

Clipboard functionality is a feature of X11. I don't see how we could
support this without X11 libraries available. I suppose Wayland supports
something similar, but the fact remains, we need to connect to some
existing frame work that enables the inter-process communication. We
cannot just develop our own here.

Thanks,
Christian
--
I hear what you're saying but I just don't care.

[Michael Henry]

On 2/10/24 00:35, Tony Mechelynck wrote:
> I used to have a HowTo page about it — two, actually, one for
> Unix/Linux and one for Windows — but my ISP took down my user
> site, saying "Our engineers will gladly design a user site for
> you". I thought I had saved the contents somewhere, but alas,
> at the moment I cant't find it back, or I would have attached
> the Linux page to this message.

Tony,

Your site has a number of snapshots saved by the Internet
Archive's Wayback Machine:
https://web.archive.org/web/20210501000000*/http://users.skynet.be/antoine.mechelynck/

The snapshot from 2021-07-22 appears to be the most recent, but
that seems to be after the site went down:
https://web.archive.org/web/20210722042200/http://users.skynet.be/antoine.mechelynck/

The most recent snapshot that appears to work is from 2021-03-30:
https://web.archive.org/web/20210330142852/http://users.skynet.be/antoine.mechelynck/

Your link to "The Vim Editor" page works in that snapshot, with
the Linux instructions for compiling Vim still present:
https://web.archive.org/web/20210411035912/http://users.skynet.be/antoine.mechelynck/vim/compunix.htm

Hopefully you can resurrect whatever portions of your original
web site that you'd like to keep.

Michael Henry

[Luca Saccarola]

On sab, feb 10 2024 at 11:34:19 +01:00:00, Christian Brabandt

<cbl...@256bit.org> wrote:
> - it puts the burden to configure the clipboard to the users. Yes, I
> know we can ship pre-configured clipboard provider, but initially
> this
> has to be configured by each user separately (and then later
> eventually merged into Vim)

But its the same thing ask the the user to install gvim, recompile vim
or knowing
in the first place that they need to do that. And at least, in my
experience
(witch is obviously pretty limited), neovim's just works.

> - I think, the clipboard functionality should work out of the box.
> There
> is a reason, why no other application uses a clipboard provider
> feature but implements this directly in the application

I found that a lot of recent text editors do the providers thing.
Obviously my point isn't to do exactly this but maybe think on something
that makes the clipboard out of the box even on non gui vim packages.

> - It doesn't help with Vim tiny, since it doesn't support Vim script,
> so
> one should still need to re-compile Vim

Vim tiny to my knowledge doesn't have support for the system clipboard
so
I don't think it isn't in the scope of the discussion.

> - If you are using xclip or xsel you would still require the X11
> libraries to work with the clipboard, so I don't think this gains us
> anything

Ok, but if I am on wayland I'll use something like wl-clipboard[1] and
that
doesn't require X11 libraries. If I'm on a tty in tmux it will use the
tmux
clipboard.

> Clipboard functionality is a feature of X11. I don't see how we could
> support this without X11 libraries available. I suppose Wayland
supports
> something similar, but the fact remains, we need to connect to some
> existing frame work that enables the inter-process communication. We
> cannot just develop our own here.

When I first wrote that I was thinking of what tmux does but it is not
a good example. Also I thought that some text editor I've used did this,
but I found that they where just shelling commands.

[1]: https://github.com/bugaevc/wl-clipboard/tree/master
-- Luca Saccarola

[Enan Ajmain]

I like Luca's first suggestion. Unix philosophy: use the tool meant for
the job. Why write and maintain logic for all platforms when dedicated
tools exist to do that one job.

As for the problem with copying large amount of text, my argument is: we
shouldn't use 'clipboard+=unnamedplus' all the time. Use vim's internal
clipboard for editing and interact with the system clipboard only when
necessary, i.e., with * or + registers.

However, your comment about existing plugins to manipulate system
clipboard in other ways is my comment, too. I use a plugin that relies
on OSC52 of my terminal emulator. So I don't have to fight with Windows
and X11 clipboards. But this isn't a solution for everyone since not
everyone uses a terminal emulator with OSC52 support. My argument is:
find a plugin that serves your purpose.

Enan

[youcai]

I made some effort one year ago trying to combine pyperclip[1] -- a python clipboard library with vim and make a new plugin[2]; However, as the things went on I came to understand that a clipboard is not as simple as it seems. The clipboard itself is deeply coupled with the graphic and windowing subsystem and some impl(like Qt and gtk) requires additional setup like mainloops to function properly.

Now I consider a tty program interacting with clipboard a very bad idea. I use unamedplus in gvim, but when I have to use a terminal emulator I just use paste mode and access the clipboard with the emulator.

Consider this: you started a tmux server from your desktop, then you detached and went out with your laptop and ssh back to attach the session. Now when you copy sth from the laptop browser and attempt to paste it inside a vim instance running in that session, guess what will happen if you used approaches other than the terminal emulator?

[1] https://github.com/asweigart/pyperclip

[2] https://github.com/omegacoleman/pyperclip.vim