Trailing ^M on Windows

[Matteo Landi]

Hello everyone,

I have been battling with trailing ^M on Windows, and I was wondering if
any of you could help me figure it out.

Let's create a file with `\r\n`s (that's how the clipboard appears to be
populated when I copy things from Chrome):

> echo -en "OS cliboards are hart...\r\naren't they?\r\n" > clipboard.txt

> cat clipboard.txt
OS cliboards are hart...
aren't they?

> cat clipboard.txt -A
OS cliboards are hart...^M$
aren't they?^M$

Let's open up vim now (`vim -u NONE`) and do some experiments:

1) read 'clipboard.txt', under the cursor

:read clipboard.txt

The content is loaded, not trailing ^M, at the bottom I see:
"clpboard.txt" [dos format] 2 lines, 40 characters

2) load 'clipboard.txt' using `read!`

:read! cat clipboard.txt

The content is loaded, no trailing ^M

3) load the content into a variable, then paste it under the cursor

:let @@ = system('cat clipboard.txt') | exe 'normal p'

The content is loaded, trailing ^M are added

Let's now push the file into the OS clipboard:

> cat clipboard.txt | clip.exe

Open up vim again, and:

4) paste from the `*` register (works with `Shift+Ins` too):

:exe 'normal "*p'

OS clipboard is read, no trailing ^M

5) load the content of the clipboard using `read!`

:read! powershell.exe Get-Clipboard

The content is loaded, no trailing ^M

6) Read the content of the clipboard into a variable, then paste it under the cursor

:let @@ = system('powershell.exe Get-Clipboard') | exe 'normal p'

OS clipboard is read, trailing ^M are added

These last 2 experiments, really, left me speechless: in both cases we
are reading the output of a command, but somehow `:read` seems to strip
^M, while `system()` not. Can someone help me figure this out?

Thanks

--
Matteo Landi
https://matteolandi.net

[Tony Mechelynck]

On Windows, when you read a file in Vim and every (or almost every)
line ends with ^M, it usually means that the last line lacks an
end-of-line, which made Vim read the whole file as if it were a Unix
file. This is how I would fix such a file:

:new ++ff=dos filename.ext
:wq

(this has the side-effect of opening then closing a new window for
that file) — see ":help ++opt".
Reading the file with 'fileformat' explicitly set to dos forces both
CR+LF or LF alone (or nothing at end of file) to be recognised as
ends-of-lines. Writing the file (which still has 'fileformat' set to
dos) writes a proper CR+LF Dos/Windows end-of-line at the end of every
line including the last one, so the next time you read it it will be
recognised as a Windows file and you won't see those pesky ^M (i.e.
carriage-return) characters (they are still there but they are part of
the normal Dos/Window end-of-line).

Or if you want to transmit the file to be read on a Unix system, you
can replace :wq by :wq ++ff=unix — in that case all lines will get a
proper LF-only Unix end-of-line, which both Vim (on any platform) and
any Unix program will be able to recognise properly. In this case the
^M characters are not even there so no one will see them.

Best regards,
Tony.

[Matteo Landi]

Thanks Tony for your reply, but I am afraid I did not properly explain
myself earlier, apologies.

The actual problem I dealing with is trailing ^M when reading from the
OS clipboard; I started with dumping trailing \r\n data on a file only
to make it easier for people to reproduce my problem, but ultimately I
need to figure out how to read from the clipboard without having ^M
added at the end of each line.

So, going back to my experiments, can anyone suggest why, the following
strips trailing ^M:

:read! powershell.exe Get-Clipboard

While this one instead, doesn't?

:let @@ = system('powershell.exe Get-Clipboard') | exe 'normal p'

Also, for those who might wonder: reading from the */+ registers
unfortunately is not an option, as I am trying build a mini-plugin
around "cb" (https://github.com/iamFIREcracker/cb) which is a script
that can work via ssh too where you don't have access to the OS
clipboard; anyway, this is the reason why I am trying to read the
content of the OS clipboard by using a command, `powershell.exe
Get-Clipboard`, instead of using the */+ register.

Let me know if this makes things a little more clear.

Thanks.

[Tony Mechelynck]

My previous answer still applies to this usecase.

As the documentation for :read says, this command accepts a ++opt
modifier, even when used with an !external command.

I expect that

:read ++ff=dos !powershell.exe Get-Clipboard

(with the ++ff= modifier before the exclamation mark) will give you
the result you want, without the ^M characters. Try it, then tell us
if it works.

Best regards,
Tony.

[Matteo Landi]

Hi Tony,

The thing is, :read! appears to be stripping ^M already, irrespective of the use of ++opts:

  :read !powershell.exe Get-Clipboard

  :read ++ff=unix !powershell.exe Get-Clipboard

  :read ++ff=dos !powershell.exe Get-Clipboard

To be honest, I'd expect the second one, where we specify ++ff=unix, to leave trailing ^M, but somehow that's not happening, and for your reference (after I vim -u NONE):

  :verbose set ff

Returns 'fileformat=unix'

  :verbose set ffs

Returns 'fileformats=unix,dos'

So am I correct if I say that there is something "weird" going on with system()?  I also found the following at the end of system()'s help page, but somehow the experienced behavior is not the documented one:

  To make the result more system-independent, the shell output
  is filtered to replace <CR> with <NL> for Macintosh, and
  <CR><NL> with <NL> for DOS-like systems.

I even tried to give systemlist() a go, but each entry of the array still has that trailing ^M, so it really seems like Vim cannot properly guess the fileformat from the command output.

I am really in the dark here.

Thanks,

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vim_use/CAJkCKXvV7bzJfAGK%2BhnMmwqutb-Pee3gs1yv%3DKepK7929AL%3DDg%40mail.gmail.com.

[Tony Mechelynck]

On Mon, Nov 18, 2019 at 4:11 PM Matteo Landi <mat...@matteolandi.net> wrote:
>
> Hi Tony,
>
> The thing is, :read! appears to be stripping ^M already, irrespective of the use of ++opts:
>
> :read !powershell.exe Get-Clipboard
> :read ++ff=unix !powershell.exe Get-Clipboard
> :read ++ff=dos !powershell.exe Get-Clipboard
>
> To be honest, I'd expect the second one, where we specify ++ff=unix, to leave trailing ^M, but somehow that's not happening, and for your reference (after I vim -u NONE):
>
> :verbose set ff
>
> Returns 'fileformat=unix'
>
> :verbose set ffs
>
> Returns 'fileformats=unix,dos'
>
> So am I correct if I say that there is something "weird" going on with system()? I also found the following at the end of system()'s help page, but somehow the experienced behavior is not the documented one:
>
> To make the result more system-independent, the shell output
> is filtered to replace <CR> with <NL> for Macintosh, and
> <CR><NL> with <NL> for DOS-like systems.
>
> I even tried to give systemlist() a go, but each entry of the array still has that trailing ^M, so it really seems like Vim cannot properly guess the fileformat from the command output.
>
> I am really in the dark here.

So am I.

As a last resort, the following Normal-mode mapping will remove (after
the fact) all ^M characters found at he end of a line:

:map <F5> :%s/<Ctrl-M>$//<CR>

Of course, you can replace <F5> by something else (if for instance
your Normal-mode F5 key is already mapped).

Meaning of the {rhs}:
: start an ex-command
% or 1,$ : from top to bottom of the file
s :s[ubstitute] (search and replace)
/ replace what
<Ctrl-M> (typed as such: less-than, C, etc.) : ^M (carriage return)
$ at end-of-line
/ replace by what
(nothing) replace by nothing
/ end of replace-what text
(nothing) no flags: replace once (at most) per line (N.B.
there can be only one end-of-line per line)
<CR> end of ex-command

Best regards,
Tony.

[Matteo Landi]

Hi Tony,

I was actually writing a plugin for this, and since I could change its implementation I ended up with something like (note the call to substitute() right after system()):

function! s:paste(...) " {{{
  let l:after = get(a:, 1, 1)
  let reg_save = @@

  let @@ = system(g:cb_paste_prg)
  " XXX pastin on Windows somehow leaves trailing ^M
  " gotta figure out a better way to fix this, but
  " for now, this will do!
  let @@ = substitute(@@, "\r\n", "\n", "g")
  setlocal paste
  if l:after
    exe 'normal p'
  else
    exe 'normal P'
  endif
  setlocal nopaste

  let @@ = reg_save
endfunction " }}}

I will leave this here for a little while, and then I will probably cross-post it on vim_dev.

Thanks again for the help.

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vim_use/CAJkCKXsaTWz2Mw-8nKe5H05_2WFG_iPNnEtMa1bCQ3aN0Ja-7g%40mail.gmail.com.

[Andy Wokula]

Am 16.11.2019 um 12:25 schrieb Matteo Landi:
> Hello everyone,
>
> I have been battling with trailing ^M on Windows

Are you using the Cygwin version of Vim?
:echo has('win32unix')

--
Andy

[Matteo Landi]

Yes, it was the Cygwin version of Vim. Is it expected to behave differently, in terms of EOL automatic recognition? (sorry for asking something I could have checked myself, but I am on the go, and figured it'd be best if I replied to you and kept thr thread alive).

Thanks 

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vim_use/5DDFF358.6000502%40yahoo.de.