Encountered problem with changed Windows 'shcf' and 'sxq' defaults.

[Ingo Karkat]

Hello Vim developers,

Sorry for touching that can of worms again, but I've come upon a problem with
the changed 'shellxquote' option on Windows (patch 7.3.443 (MS-Windows: 'shcf'
and 'sxq' defaults are not very good) and the few patches following it. Here's
the discussion as a refresher:
http://groups.google.com/group/vim_dev/browse_thread/thread/bd53b29c5e5f2a50/be11dca5e9d501bb)

The problem:

#v+
vim -N -u NONE
:echo system('echo.hi(ho)hi')
E484: Can't open file D:\temp\VIo408F.tmp
#v-

It's the unquoted parentheses in the command. My actual command was
:echo system('icacls ' . shellescape('C:\foo') . ' /deny
%userdomain%\%username%:(WD,AD,DC)')

The problem lies in the way the command is passed to the shell, as the following
experiments show:

#v+
REM This is how Vim passes the command with shcf=/c and sxq=(
C:\> C:\Windows\system32\cmd.exe /c (echo.hi^(ho^)hi ^>D:\temp\VIo8945.tmp 2^>^&1)
hi was unexpected at this time.

REM Enclosing the entire parenthesized command in double quotes avoids the
error, but breaks the redirection.
C:\> C:\Windows\system32\cmd.exe /c "(echo.hi^(ho^)hi ^>D:\temp\VIo8945.tmp 2^>^&1)"
hi(ho)hi >D:\temp\VIo8945.tmp 2>&1

REM Enclosing the command without the redirection in double quotes and
parentheses seems to work.
C:\> C:\Windows\system32\cmd.exe /c "(echo.hi^(ho^)hi)" ^>D:\temp\VIo8945.tmp 2^>^&1

REM Best recommendation is probably to always quote all arguments, even when not
necessary in a plain shell.
C:\> C:\Windows\system32\cmd.exe /c (echo.^"hi^(ho^)hi^" ^>D:\temp\VIo8945.tmp
2^>^&1)
#v-

Therefore, every argument should be quoted; this works:
:echo system('icacls ' . shellescape('C:\foo') . ' /deny ' .
shellescape('%userdomain%\%username%:(WD,AD,DC)'))


I know that the arcane quoting rules of cmd.exe are to blame for this mess. It
was a real hassle to get from the E484 to the root cause of this (in Vim 7.3.000
(with the old quoting implementation), my command worked fine); something I'd
rather avert from others, so hopefully my findings will be helpful in some way.
It seems like an intractable problem that has no perfect solution.

#v+
Windows 7 SP1 Ultimate 64-bit English

VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Feb 20 2013 15:34:49)
MS-Windows 64-bit GUI version with OLE support
Included patches: 1-823
Compiled by in...@karkat.de
Huge version with GUI. Features included (+) or not (-):
#v-

-- regards, ingo
--
-- Ingo Karkat -- /^-- /^-- /^-- /^-- /^-- /^-- http://ingo-karkat.de/ --
-- http://vim.sourceforge.net/account/profile.php?user_id=9713 --

[Ben Fritz]

On Friday, February 22, 2013 3:27:41 AM UTC-6, Ingo Karkat wrote:
> Hello Vim developers,
>
> Sorry for touching that can of worms again, but I've come upon a problem with
> the changed 'shellxquote' option on Windows (patch 7.3.443 (MS-Windows: 'shcf'
> and 'sxq' defaults are not very good) and the few patches following it. Here's
> the discussion as a refresher:
> http://groups.google.com/group/vim_dev/browse_thread/thread/bd53b29c5e5f2a50/be11dca5e9d501bb)
>
> The problem:
>
> #v+
> vim -N -u NONE
> :echo system('echo.hi(ho)hi')
> E484: Can't open file D:\temp\VIo408F.tmp
> #v-
>
> It's the unquoted parentheses in the command. My actual command was
> :echo system('icacls ' . shellescape('C:\foo') . ' /deny
> %userdomain%\%username%:(WD,AD,DC)')
>

> [SNIP]

>
> REM Best recommendation is probably to always quote all arguments, even when not
> necessary in a plain shell.
> C:\> C:\Windows\system32\cmd.exe /c (echo.^"hi^(ho^)hi^" ^>D:\temp\VIo8945.tmp
> 2^>^&1)
> #v-
>
> Therefore, every argument should be quoted; this works:
> :echo system('icacls ' . shellescape('C:\foo') . ' /deny ' .
> shellescape('%userdomain%\%username%:(WD,AD,DC)'))
>

In other words, if you encounter errors passing arguments to the shell, try
using shellescape, or try escaping things manually with ^. Your original example
is:

:echo system('echo.hi(ho)hi')

Either of these variants work:

:echo system('echo.hi^(ho^)hi')
:echo system('echo.'.shellescape('hi(ho)hi'))

Alternatively, :help 'shellxquote' says that if it is "( then )" is appended.
This gave me a hint, which I tested; this also works:

:set shellxquote="(
:echo system('echo.hi(ho)hi')

What are you asking for exactly? A note in the help? Or some further update to
the quoting of shell arguments on Windows? I think a note in the help should
suffice. Where do you think it should go? Maybe we can add a note to E484 that
on Windows it is sometimes caused by unescaped special characters, and to try
using shellescape or a new value of shellxquote to fix it.

Can of worms, indeed! I didn't realize how bad cmd.exe is until that series of patches.

[Ingo Karkat]

Yes, this works in the toy example, but not in the general case.

> What are you asking for exactly? A note in the help? Or some further update to
> the quoting of shell arguments on Windows?

I was hoping for someone to have the magic 100% solution :-) Seriously, I now
doubt it can be perfected, so we have to live with what we have. I had the
command in my test suite for a very long time, and I was just very surprised how
(and in which strange ways) this was suddenly broken. Usually, there are very
few regressions in Vim (thanks, BTW!) I also see other plugin writers struggling
with escaping on Windows.

> I think a note in the help should suffice. Where do you think it
> should go? Maybe we can add a note to E484 that on Windows it is
> sometimes caused by unescaped special characters, and to try using
> shellescape or a new value of shellxquote to fix it.

Yeah, that's a good idea. How about this:

diff -r 8b86b69546a9 runtime/doc/message.txt
--- a/runtime/doc/message.txt Wed Feb 20 21:26:00 2013 +0100
+++ b/runtime/doc/message.txt Fri Feb 22 20:43:06 2013 +0100
@@ -730,7 +730,11 @@
Can't open file {filename}
Can't read file {filename}

-Vim cannot read a temporary file.
+Vim cannot read a temporary file. Especially on Windows, this can be caused
+by missing escaping of special characters for cmd.exe; the approach was
+changed with patch 7.3.443. Try using |shellescape()| for all shell arguments
+given to |system()|, or explicitly add escaping with ^. Also see
+'shellxquote' and 'shellxescape'.

*E464* >
Ambiguous use of user-defined command

-- regards, ingo