Re: [go-nuts] Unexpected behavior of exec.Command argument parsing

[Dan Kortschak]

The arguments passed by exec.Command are not split further than you have
by having them in a slice of strings. There is no shell between the Go
program and the exec'd command.

On Tue, 2013-01-29 at 20:15 -0800, Matt Tesauro wrote:
> //finalCmd := []string{"--insecure", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}
>
> // #2 actual version I was testing with
>
> //finalCmd := []string{"--insecure", "--proxy 127.0.0.1:8080", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}
>
> // #3 while this one works where I've broken up the --proxy argument into 2 values
>
> finalCmd := []string{"--insecure", "--proxy", "127.0.0.1:8080", "-include", "-H \"Accept: application/json\"", "http://www.google.com/"}

[Andrew Gerrand]

On 30 January 2013 15:15, Matt Tesauro <mtes...@gmail.com> wrote:

If you take those same options and manually type the command in a terminal, it works fine.

When you type:

  $ echo foo bar

at a shell, the shell breaks up your command into arguments, and passes them as an array of strings to the exec syscall. This array is known as argv to C programs, or os.Args to Go programs.

The important part here is that the *shell* splits the arguments at the spaces. When you quote arguments,

  $ echo "foo bar"

it's the shell that makes sure that "foo bar" is passed as a single argument. Note that it doesn't pass the quotes.

So when you type:

  $ echo -n "foo bar"

the argument array is []string{"echo", "-n", "foo bar"}.

Now when you use the exec.Command function, it does no interpretation of the arguments (except the first one, the program that should be executed). It just passes the args straight to the process as it execs it.

That's why "--proxy" and "127.0.0.1:8080" must be specified as separate args in your Go program, so that it works just as the shell does.

Andrew

[Nate Finch]

Yep, I hit this too. Took me a minute to figure it out my first time.

Just think of it this way - the args given to exec.Command directly populate the os.Args (argv) of the target application as-is.

Related - Anyone know of a go package that'll take a command line style string and split it up into arguments, like the shell does?  I'd like to be able to read in commandline-like strings and pass them to exec.Command... and the more I think about the syntax, the more I hope someone else has already done this :)

[Jesse McNelis]

On Wed, Jan 30, 2013 at 11:34 PM, Nate Finch <nate....@gmail.com> wrote:

Related - Anyone know of a go package that'll take a command line style string and split it up into arguments, like the shell does?  I'd like to be able to read in commandline-like strings and pass them to exec.Command... and the more I think about the syntax, the more I hope someone else has already done this :)

The format of a "command line like string" really depends on the shell you use, as each shell is different.

The simplest solution to this is to have exec start a shell and pass it that string.

eg.

exec.Command("/bin/sh", "echo 'pizza' | wc -c") 

--
=====================
http://jessta.id.au

[Nate Finch]

Ahh, of course. That's a good workaround, and one I've even used before, just didn't think of it. You have a good point that different shells parse command lines in different ways... this is probably the best bet, and just make different implementations per OS.

[Matt Tesauro]

@Nate

I did something like what you're talking about in my full program.  I take the command line arguments to jerry-curl and parse them to sort out those for jerry-curl and those to pass through to curl. You can see where I did this on GitHub in the parseArgs function on line 278 here: https://github.com/mtesauro/jerry-curl/blob/master/jerry-curl.go 

That's not quite what your asking but quite close.

[Matt Tesauro]

First, props to the golang group/golang community for sever very quick answers.  The answers informed my further experimentation today and now I see what's happening.

While I get why quoting is needed in certain circumstances in a shell, the thing confusing me was why I need to split the --proxy and its value but don't need to for -H.  I've been playing with it more this AM and it finally clicked.  My code is not only broken for --proxy, but -H isn't working as I thought.  For example, if finalCmd is this:

finalCmd := []string{"--proxy", "127.0.0.1:8080", "-H \"Accept: application/json\"", "http://www.google.com/"}

it works - or at least appears to work.  However, looking in my local proxy, I see that the Accept header is also wrong:

GET http://www.google.com/ HTTP/1.1

User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3

Host: www.google.com

Accept: */*

Proxy-Connection: Keep-Alive

"Accept: application/json"

Quoted headers - DOH!  If you do the same thing manually in a shell (bash on Linux), it works - headers and all:

$ curl --proxy 127.0.0.1:8080 -H "Accept: application/json" http://www.google.com

My mistake was to think of "--proxy 127.0.0.1:8080" and "-H \"Accept: application/json\"" as atomic units rather then items from a command line that need to be separated by space.  From how -H is handled, I see that any string in my slice which has a space is quoted while constructing the command.

Since the proxy error was very apparent with an error from curl (non-zero exit), the -H was quietly sent in HTTP which I missed until this AM.

It may help to have a more complex example for exec.Command in the docs since the tr example doesn't have command line options with following values.

Thanks all for your help - i'll add a bit of code to split any command line argument with an space into two strings in my slice.

Cheers!

[John Asmuth]

https://github.com/kballard/go-shellquote