[patch] file encoding detection doesn't work with process substitution

[Ken Takata]

Hi,

When process substitution (e.g. "<(cmd)") is used on bash (or zsh), file
encoding detection doesn't work, and an empty buffer is opened.
E.g.:

$ vim -u NONE --cmd "set fencs=cp932,utf-8" <(echo テスト)
# Assuming the system encoding is utf-8.

Vim will be executed with the command line like:

$ vim -u NONE --cmd "set fencs=cp932,utf-8" /dev/fd/63

"/dev/fd/63" is a fifo and it is not seekable.
The echoed string "テスト" (utf-8) cannot be converted from cp932 to utf-8, so
Vim retries with utf-8, but it fails because the input is not seekable.

However reading from stdin can detect the encoding. E.g.:

$ echo テスト | vim -u NONE --cmd "set fencs=cp932,utf-8" -

I think file encoding detection should also work with process substitution.
Attached patch fixes the problem.

Regards,
Ken Takata

[Bram Moolenaar]

Thanks. It would be good to have a test for this. The encoding
detection is important, but I don't think there is a test for it.

Reading from stdin and fifo require starting Vim. Test49 does this, it
just uses ../vim for that. That doesn't work on Windows though. But
the fifo won't work on Windows either.


--
Mrs Abbott: I'm a paediatrician.
Basil: Feet?
Mrs Abbott: Children.
Sybil: Oh, Basil!
Basil: Well, children have feet, don't they? That's how they move
around, my dear. You must take a look next time, it's most
interesting. (Fawlty Towers)

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Ken Takata]

Hi Bram,

2016/8/6 Sat 22:30:05 UTC+9 Bram Moolenaar wrote:
> Ken Takata wrote:
>
> > When process substitution (e.g. "<(cmd)") is used on bash (or zsh), file
> > encoding detection doesn't work, and an empty buffer is opened.
> > E.g.:
> >
> > $ vim -u NONE --cmd "set fencs=cp932,utf-8" <(echo テスト)
> > # Assuming the system encoding is utf-8.
> >
> > Vim will be executed with the command line like:
> >
> > $ vim -u NONE --cmd "set fencs=cp932,utf-8" /dev/fd/63
> >
> > "/dev/fd/63" is a fifo and it is not seekable.
> > The echoed string "テスト" (utf-8) cannot be converted from cp932 to utf-8, so
> > Vim retries with utf-8, but it fails because the input is not seekable.
> >
> > However reading from stdin can detect the encoding. E.g.:
> >
> > $ echo テスト | vim -u NONE --cmd "set fencs=cp932,utf-8" -
> >
> > I think file encoding detection should also work with process substitution.
> > Attached patch fixes the problem.
>
> Thanks. It would be good to have a test for this. The encoding
> detection is important, but I don't think there is a test for it.
>
> Reading from stdin and fifo require starting Vim. Test49 does this, it
> just uses ../vim for that. That doesn't work on Windows though. But
> the fifo won't work on Windows either.

I added a test for this. I also wanted to add a test for stdin on Windows,
but Windows' echo command cannot write Unicode characters to a pipe. So the
test is disabled on Windows.

BTW, I found that RunVim() in shared.vim didn't work as expected. I also
fixed this.

Regards,
Ken Takata

[Bram Moolenaar]

Thanks. It would be better to use "cat" instead of "echo". Does this
also exist on MS-Windows? Or perhaps we can check if it exists.
Then we can also use multiple lines in the test.

> BTW, I found that RunVim() in shared.vim didn't work as expected. I also
> fixed this.

You mean the return value? Strange that everything worked anyway...
Oh, it just skipped the actual test!

--
If the Universe is constantly expanding, why can't I ever find a parking space?

[Ken Takata]

Hi,

OK, "type" can be used on Windows.
Please check the attached patch.

Regards,
Ken Takata

[Bram Moolenaar]

Thanks. Glad you could make it work. I verified that without your
patch the utf8 fifo test fails.

--
Normal people believe that if it ain't broke, don't fix it. Engineers believe
that if it ain't broke, it doesn't have enough features yet.
(Scott Adams - The Dilbert principle)