Read from command output broken on windows

[Jan.S...@t-online.de]

Hi,

 

if you read with

 

- Vim 7.4.22 (32bit)

- Windows 7 (x64)

 

from a command line output by

 

:[range]r[ead] [++opt] !{cmd}

 

and using parenthesis the command is completly broken.

 

If you write

 

:r! cmd /c echo Blah

 

you get "Blah"

 

:r! cmd /c echo B(la)h

 

is broken. It is neither executed nor a piped outline file is created, so VIm can't read the output and throws a read error on temp file. This is working under 7.3 without problems.

 

Anyone an idea where to start bug hunting?

 

Best regards,

 

 

Jan

 

 

[Ben Fritz]

:verbose set shellcmdflag? shellxquote? shellxescape? shell?

Halfway through Vim 7.3 patches, the 'shellxescape' option was added and defaults for the others modified so that Windows shell commands succeeded more often. But they still fail with some special characters, because cmd.exe quoting is "special".

For once I'm posting from a Linux system and can't test at the moment. Try with these options at their default value. If these options are still at their defaults, I think you can try shellxquote as "(

Yes, that's two characters, including a " character.

[Christian Brabandt]

I think, you need to escape the parenthesis with '^'.

regards,
Christian

[Jan Stocker]

> :r! cmd /c echo B(la)h
>
> is broken. It is neither executed nor a piped outline file is
> created, so VIm can't read the output and throws a read error on temp
> file. This is working under 7.3 without problems.
>
> Anyone an idea where to start bug hunting?
>

I think, you need to escape the parenthesis with '^'.

 

No this does not work either, if shellxquote is ( vim does this automatically, you can proof this by

 

:!echo b(l)ah

 

Sorry, only a German system available.

 

C:\Windows\system32\cmd.exe /c (echo b^(l^)ah)
"ah)" kann syntaktisch an dieser Stelle nicht verarbeitet werden.
shell returned 1
Hit any key to close this window...

 

 

Best regards,

 

Jan

[Jan Stocker]

> :r! cmd /c echo B(la)h
>
> is broken. It is neither executed nor a piped outline file is created, so VIm can't read the output and throws a read error on temp file. This is working under 7.3 without problems.
>
> Anyone an idea where to start bug hunting?
>

:verbose set shellcmdflag? shellxquote? shellxescape? shell?

Halfway through Vim 7.3 patches, the 'shellxescape' option was added and defaults for the others modified so that Windows shell commands succeeded more often. But they still fail with some special characters, because cmd.exe quoting is "special".

For once I'm posting from a Linux system and can't test at the moment. Try with these options at their default value. If these options are still at their defaults, I think you can try shellxquote as "(

  shellcmdflag=/c
  shellxquote=(
  shellxescape="&|<>()@^
  shell=C:\Windows\system32\cmd.exe

For Win32 the default for shellxquote is ( so the shellxescape characters will be escaped by ^. The help file and implementation is correct, but cmd.exe does not understand the escaped ^(

  

:!echo b(l)ah

 

C:\Windows\system32\cmd.exe /c (echo b^(l^)ah)
"ah)" kann syntaktisch an dieser Stelle nicht verarbeitet werden.
shell returned 1
Hit any key to close this window...

 

Using "( works, no escape characters will be used. But I do not understand

 

a) the goal of this implementation

b) the non working default setting for Win32

 

Of course I do not want to run an "echo" but "c:\program Files (x86)\...." and this behaviour will break many plugins...

 

 

Best regards,

 

Jan

[Christian Brabandt]

On Wed, September 11, 2013 15:35, Jan Stocker wrote:
>> :r! cmd /c echo B(la)h
> >
> > is broken. It is neither executed nor a piped outline file is
> > created, so VIm can't read the output and throws a read error on
> temp
> > file. This is working under 7.3 without problems.
> >
> > Anyone an idea where to start bug hunting?
> >
>
> I think, you need to escape the parenthesis with '^'.
>
> No this does not work either, if shellxquote is ( vim does this
> automatically, you can proof this by
>
> :!echo b(l)ah
>
> Sorry, only a German system available.
>

> C:Windowssystem32cmd.exe /c (echo b^(l^)ah)

> "ah)" kann syntaktisch an dieser Stelle nicht verarbeitet werden.
> shell returned 1
> Hit any key to close this window...

Hm, interesting. I confirm, I don't understand the quoting rules
completly and there were always cases, when it didn't work on Windows.
Interestingly, there are some differences between calling
:r !echo and :!echo

:!echo bl(ah)
cmd.exe /c (echo bl^(ah^))

")" kann syntaktisch an dieser Stelle nicht verarbeitet werden.
shell returned 1
Hit any key to close this window...

:!echo "bl(ah)"
cmd.exe /c (echo ^"bl^(ah^)^")
"bl(ah)"

Hit any key to close this window...

:r !echo bl(ah)
Rufe Shell auf, um "echo bl(ah) >
C:\Users\DE914854\AppData\Local\Temp\VIoB10D.tmp" auszuf�hren

E485: Kann Datei C:\Users\DE914854\AppData\Local\Temp\VIoB10D.tmp nicht lesen
no output

:r !echo bl^(ah^)
Rufe Shell auf, um "echo bl^(ah^) >
C:\Users\DE914854\AppData\Local\Temp\VIo1839.tmp" auszuf�hren
output: bl(ah)

:r !echo "bl(ah)"
Rufe Shell auf, um "echo "bl(ah)" >
C:\Users\DE914854\AppData\Local\Temp\VIoD0E0.tmp" auszuf�hren
output: "bl(ah)"

regards,
Christian

[Ben Fritz]

The intent of the implementation was to make this work by default:

:!"C:\Program Files\something\app.exe" "C:\Documents and Settings\user1\blah.txt"

The old defaults would make this fail by stripping off the first and last parenthesis. The current defaults were the best we could come up with in a series of patches: 7.3.443, 7.3.445-7.3.448, and 7.3.450.

Your problem looks very similar to this:

https://groups.google.com/d/topic/vim_dev/Uxx5KcB7ktA/discussion

But in that case, escaping with ^ seemed to work.

I do think your use case was why the "( support was added.

I think I remember that "echo" is special and doesn't work the same as calling an actual application, as well.

By the way, changing the command to:

:!cmd /c "echo b(l)ah"

also works, with default option values. This works around the special-ness of "echo" if that's actually the case.

[Ben Fritz]

On Wednesday, September 11, 2013 8:47:33 AM UTC-5, Jan Stocker wrote:
>  

> Using "( works, no escape characters will be used. But I do not understand
>
>  
>
> a) the goal of this implementation
>
> b) the non working default setting for Win32
>
>  
>
> Of course I do not want to run an "echo" but "c:\program Files (x86)\...." and this behaviour will break many plugins...
>

The "non-working default setting" actually DOES work in most cases, actually in more cases than the old default value of an empty shellxquote. There are special cases when using the characters in the default of shellxescape, where the defaults can still break, but normally they work.

For example, the following command-line DOES work with default settings:

:!"C:\Program Files (x86)\vim\vim74\diff.exe" --version

So does this:

:!cmd /c "echo b(l)ah ^& ble(e^|a)t"

I'm having trouble finding the discussions leading up to each of the patches, but I remember specifically that "( was added to take care of some really special cases not covered by the defaults. I don't recall why "( wasn't made the default, I think it broke some more common uses or something.

[Jan Stocker]

>> Using "( works, no escape characters will be used. But I do not understand
>>
>>
>>
>> a) the goal of this implementation
>>
>> b) the non working default setting for Win32
>>
>>
>>
>> Of course I do not want to run an "echo" but "c:\program Files (x86)\...." and this behaviour will break many plugins...
>>
>>
>
> The intent of the implementation was to make this work by default:
>
> :!"C:\Program Files\something\app.exe" "C:\Documents and Settings\user1\blah.txt"
>
> The old defaults would make this fail by stripping off the first and last parenthesis. The current defaults were the best we could come up with in a series of patches: 7.3.443, 7.3.445-7.3.448, and 7.3.450.
>
> Your problem looks very similar to this:
>
> https://groups.google.com/d/topic/vim_dev/Uxx5KcB7ktA/discussion
>
> But in that case, escaping with ^ seemed to work.
>
> I do think your use case was why the "( support was added.
>
> I think I remember that "echo" is special and doesn't work the same as calling an actual application, as well.
>
> By the way, changing the command to:
>
> :!cmd /c "echo b(l)ah"
>
> also works, with default option values. This works around the special-ness of "echo" if that's actually the case.

So how would you run a valid cmd.exe line

"c:\program files (x86)\xyz\xyz.exe" "arg1 with spaces" "arg 2 with spaces"

from a script rather than

exe 'r!cmd.exe /c ""c:\program files (x86)\xyz\xyz.exe" "arg1 with spaces" "arg 2 with spaces""'

which does not work with default settings under 7.4 (I patched it today using short file names, but that is no solution)?

And what to do to run this for 7.3 (some version before the patch) AND 7.4?

Best regards,

Jan

[Ben Fritz]

On Wednesday, September 11, 2013 10:13:17 AM UTC-5, Jan Stocker wrote:
>
>
>
> So how would you run a valid cmd.exe line
>
>
>
> "c:\program files (x86)\xyz\xyz.exe" "arg1 with spaces" "arg 2 with spaces"
>
>

This is exactly the situation that the changes were designed for! For me the defaults work. I just now launched vim with "gvim -N -u NONE -i NONE" and ran this command, getting the expected diff output in a command window:

:!"C:\Program Files (x86)\vim\vim74\diff.exe" "C:\Users\btfritz\AppData\Local\Temp\file 1.txt" "C:\Users\btfritz\AppData\Local\Temp\file 2.txt"

>
> from a script rather than
>
>
>
> exe 'r!cmd.exe /c ""c:\program files (x86)\xyz\xyz.exe" "arg1 with spaces" "arg 2 with spaces""'
>
>

This is an old-style workaround for the bad defaults. Adding an extra " at the beginning and end of the string and allowing cmd.exe to strip them off is how this works.

The problem is now you're escaping the special characters only once, but you're running cmd.exe twice. So you need to escape them manually to prevent the implicitly called cmd.exe from seeing unescaped special characters outside of matching quotes. Or, just call the program directly instead of wrapping it with cmd.exe.

I.e. you can use my command above, or this variant:

:!cmd /c ""C:\Program Files ^(x86^)\vim\vim74\diff.exe" "C:\Users\btfritz\AppData\Local\Temp\file 1.txt" "C:\Users\btfritz\AppData\Local\Temp\file 2.txt""

>
> which does not work with default settings under 7.4 (I patched it today using short file names, but that is no solution)?
>
>

Patch it instead to use the full command.

>
> And what to do to run this for 7.3 (some version before the patch) AND 7.4?
>

In this specific case, you can check the version and patchlevel using (v:version<7.3 || v:version==7.3 && !has('patch450')) and set 'shellxquote' to a single " character if true. This uses Vim to automatically surround the entire thing in an extra pair of quotes like the existing code does manually. This would have been the correct fix in the first place instead of this workaround.

Naturally you'll need to save and restore shellxquote around this. And only do this on Windows.

Something like (untested):

if has('win32') && (v:version<7.3 || v:version==7.3 && !has('patch450')) && &shell=~cmd
let s:shxq_sav = &shellxquote
set shellxquote="
endif
!"c:\program files (x86)\xyz\xyz.exe" "arg1 with spaces" "arg 2 with spaces"
if exists('s:shxq_sav')
let &shellxquote = s:shxq_sav
unlet s:shxq_sav
endif

[Tony Mechelynck]

Space, backslash and double-quote need backslash-escaping when used in a
:set statement, see :help option-backslash

Use
set shellxquote=\"
to set 'shellxquote' to just a double-quote character. Or else, use
let &shellxquote = '"'

> endif
> !"c:\program files (x86)\xyz\xyz.exe" "arg1 with spaces" "arg 2 with spaces"
> if exists('s:shxq_sav')
> let &shellxquote = s:shxq_sav
> unlet s:shxq_sav
> endif
>

Best regards,
Tony.
--
Only a mediocre person is always at his best.
-- Laurence Peter

[Jan Stocker]

>> Something like (untested):
>>
>> if has('win32') && (v:version<7.3 || v:version==7.3 && !has('patch450')) && &shell=~cmd
>> let s:shxq_sav = &shellxquote
>> set shellxquote="
>
> Space, backslash and double-quote need backslash-escaping when used in a :set statement, see :help option-backslash
>
> Use
> set shellxquote=\"
> to set 'shellxquote' to just a double-quote character. Or else, use
> let &shellxquote = '"'
>
>> endif
>> !"c:\program files (x86)\xyz\xyz.exe" "arg1 with spaces" "arg 2 with spaces"
>> if exists('s:shxq_sav')
>> let &shellxquote = s:shxq_sav
>> unlet s:shxq_sav
>> endif
>>

Thank you all for your help. Of course you have double escape stuff when using cmd.exe twice, I missed that. By the way, here is a code snippet tested on Win32 based on your notes:

if has('win32') && (v:version<703 || (v:version==703 && !has('patch450'))) && &shell=~'cmd'

let s:shxq_sav = &shellxquote

set shellxquote=\"
endif

silent! exe 'noautocmd r!'.a:cmdline

if exists('s:shxq_sav')
let &shellxquote = s:shxq_sav
unlet s:shxq_sav
endif

Best regards,

Jan