Shellquote in stdlib

[Christian Stewart]

Hi all,

There have been several discussions about shellquote in go-nuts in the past:

https://github.com/kballard/go-shellquote

https://github.com/gonuts/go-shellquote

I've found it necessary to use both Split and Join in some cases when working with exec.Command.

This seems like the type of thing that ought to be part of the stdlib, has there been any consideration towards including it or a similar implementation?

This is useful for processing os.Args especially in CLI or emulated shell environments.

Thanks!

Christian Stewart

[Kurtis Rader]

No, IMHO, because such quoting is OS specific. For example, the quoting for UNIX and Windows in this context are different. Furthermore, given the vagaries of the POSIX shell standard I am skeptical that either of the projects you linked to correctly manipulates such strings in every case.

It would help if you provided a concrete example of the problem you need help with.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Martin Atkins]

Indeed, the rules for quoting and parsing command lines are considerably different even in different context on the same OS (Windows) and so successfully building a command line to be run requires detailed knowledge of the context in which it will be evaluated.

Unix systems are at least relatively consistent but even then there are edge-cases such as differently-set IFS environment variables to be worried about.

I wrote a library that tries to deal with various different situations that I had to interact with in some software (not open source) that did automatic provisioning of virtual machines via SSH or local virtual console and physical machines accessed via network serial terminals:

    https://pkg.go.dev/github.com/apparentlymart/go-s...@v0.0.1/shquot

In the docs for that I tried to be very explicit about the behavior and limitations of each of the functions so that it's hopefully possible to understand what is the most appropriate combination of transforms for any particular context. Even then, there are plenty of exceptions where things might not work quite as expected. (I would not be surprised to learn that this code still has bugs and edge cases even though I've used it in a number of situations already!)

There's considerable subtlety and uncertainty in all of this. The situation on Windows is particularly fraught. I think this sort of thing is better handled in third-party libraries than in the standard library so that each library can be tailored to whatever requirements it has, rather than over-promising as being a general solution.

[Christian Stewart]

Kurtis,

This is what I'm trying to accomplish:

Start delve (dlv) with os/exec passing a set of []string os.Args as
--build-flags

Let's say I have this build arguments set: []string{"-gcflags", "-N -l"}

- How can I pass this to --build-flags in Delve? strings.Join(flags,
" ") is wrong.
- How can I pass this to "go" with os/exec correctly? The quotes are
not put properly.

There are some cross-platform cases where the shell quoting is used,
and the general "quote something" mechanics are similar on Windows and
Linux.

Thanks,
Christian

[Kurtis Rader]

On Wed, Nov 23, 2022 at 7:33 AM Christian Stewart <chri...@aperture.us> wrote:

This is what I'm trying to accomplish:

Start delve (dlv) with os/exec passing a set of []string os.Args as
--build-flags

Let's say I have this build arguments set: []string{"-gcflags", "-N -l"}

 - How can I pass this to --build-flags in Delve? strings.Join(flags,
" ") is wrong.

Note that there are two levels of quoting involved in your example and a package that does "shell quoting" isn't going to handle this scenario correctly except for uninteresting trivial cases.

 

 - How can I pass this to "go" with os/exec correctly? The quotes are
not put properly.

I'm not sure I understand that question. If you're asking how to run `go -gcflags="-N -l"` via a program written in Go the answer is to call os.exec with the literal string "-gcflags=-N -l" as an argument. Note that if you execute the `go` command directly, rather than indirectly via a shell, you don't need quotes around the "-N -l" portion of that string since a shell isn't parsing the command. This is why you should avoid running external commands indirectly through a shell. Doing so greatly, and usually needlessly, complicates constructing the command to be run.

 

There are some cross-platform cases where the shell quoting is used,
and the general "quote something" mechanics are similar on Windows and
Linux.

Similar is not the same as identical. The rules for how quoted strings, and other punctuation, are handled differs enough that only trivial, uninteresting, cases are likely to work correctly on both platforms. Heck, even on UNIX like systems if the user has set their login shell to a non-POSIX shell (like Fish or Elvish) and your code uses the login shell rather than something like /bin/sh you will be in for a surprise. :-)

On Sun, Nov 6, 2022 at 9:11 PM Kurtis Rader <kra...@skepticism.us> wrote:
>
> On Sun, Nov 6, 2022 at 8:48 PM 'Christian Stewart' via golang-nuts <golan...@googlegroups.com> wrote:
>>
>> There have been several discussions about shellquote in go-nuts in the past:
>>
>> https://github.com/kballard/go-shellquote
>> https://github.com/gonuts/go-shellquote
>>
>> I've found it necessary to use both Split and Join in some cases when working with exec.Command.
>>
>> This seems like the type of thing that ought to be part of the stdlib, has there been any consideration towards including it or a similar implementation?
>>
>> This is useful for processing os.Args especially in CLI or emulated shell environments.
>
>
> No, IMHO, because such quoting is OS specific. For example, the quoting for UNIX and Windows in this context are different. Furthermore, given the vagaries of the POSIX shell standard I am skeptical that either of the projects you linked to correctly manipulates such strings in every case.
>
> It would help if you provided a concrete example of the problem you need help with.
>
> --
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank

[Christian Stewart]

Hi Kurtis,

On Wed, Nov 23, 2022, 4:57 PM Kurtis Rader <kra...@skepticism.us> wrote:

On Wed, Nov 23, 2022 at 7:33 AM Christian Stewart <chri...@aperture.us> wrote:

This is what I'm trying to accomplish:

Start delve (dlv) with os/exec passing a set of []string os.Args as
--build-flags

Let's say I have this build arguments set: []string{"-gcflags", "-N -l"}

 - How can I pass this to --build-flags in Delve? strings.Join(flags,
" ") is wrong.

Note that there are two levels of quoting involved in your example and a package that does "shell quoting" isn't going to handle this scenario correctly except for uninteresting trivial cases.

Please explain?

 - How can I pass this to "go" with os/exec correctly? The quotes are
not put properly.

I'm not sure I understand that question. If you're asking how to run `go -gcflags="-N -l"` via a program written in Go the answer is to call os.exec with the literal string "-gcflags=-N -l" as an argument. Note that if you execute the `go` command directly, rather than indirectly via a shell, you don't need quotes around the "-N -l" portion of that string since a shell isn't parsing the command.

That doesn't work, from my tests

This is why you should avoid running external commands indirectly through a shell. Doing so greatly, and usually needlessly, complicates constructing the command to be run.

 

There are some cross-platform cases where the shell quoting is used,
and the general "quote something" mechanics are similar on Windows and
Linux.

Similar is not the same as identical. The rules for how quoted strings, and other punctuation, are handled differs enough that only trivial, uninteresting, cases are likely to work correctly on both platforms. Heck, even on UNIX like systems if the user has set their login shell to a non-POSIX shell (like Fish or Elvish) and your code uses the login shell rather than something like /bin/sh you will be in for a surprise. :-)

I'm not calling out to the shell, just passing args to another process that expects them to be shell quoted. 

Thanks,

Christian

[Martin Atkins]

I don't think the Go command is using a shell to parse the value of the -gcflags option.

I believe that the syntax for this option is handled by an internal function called quoted.Split, which defines its own syntax for quoting. The result is a slice of string ready to be included as part of the arguments to another command, without any intermediate shell evaluation.

If that is the syntax gcflags is using (which I'm not 100% sure about), then you can find a corresponding function Join in that same source file which performs the opposite operation: encoding a slice of string as a single string. It's an internal package and so you can't depend on it directly from your program, but its implementation is small so presumably you could copy it into your own program.