exec.Command("cp", "-r", "./*.json", artifact.dir fails with exit status 1

[Deepak Jain]

util.ExecuteCommandWithOuput(exec.Command("cp", "-r", "./*.json", artifact.dir))

func ExecuteCommandWithOuput(cmd *exec.Cmd) {

output, err := cmd.Output()

if err != nil {

log.Print("Error executing ", cmd.Args, err)

}

fmt.Print(string(output))

}

Output

2017/01/16 13:26:35 Error executing [cp -r ./*.json myartifact] exit status 1

Questions

1. How do i get details of complete error message on failure of cp command ? I did have err != nill and Print err

2. Does exec.Command not support copy of files and recursive copy of directories ?

3. Any suggestions how i implement copy of files and recursive copy of directories ?

I have just started adopting Go and hence a new comer with Go.

Regards,

Deepak

[Dan Kortschak]

Before answering the questions below, you should know that exec.Command
will not do shell glob expansion since it does not invoke commands via
a shell. If you want to do that you can either invoke via a shell or do
the globbing yourself with filepath.Glob[1].

1. You can capture the combined output or stderr using the exec.Cmd
CombinedOutput method or Stderr field respectively.
2. exec.Command does not know anything about recursive copy of
directories, it just invokes commands.
3. This depends on exactly what it is that you are trying to achieve.

[1]https://golang.org/pkg/path/filepath/#Glob

[Deepak Jain]

Thanks for Stderr.

I would like to do a recursive copy of a directory (similar to cp -r ), directory can contain files (binaries or text files) and directories.

I now see 

exit status 1: cp: ./*.json: No such file or directory

Code:

//ExecuteCommandWithOuput will execute cmd passed as argument and display output.

func ExecuteCommandWithOuput(cmd *exec.Cmd) {

var out bytes.Buffer

var stderr bytes.Buffer

cmd.Stdout = &out

cmd.Stderr = &stderr

err := cmd.Run()

if err != nil {

fmt.Println(fmt.Sprint(err) + ": " + stderr.String())

return

}

fmt.Println("Result: " + out.String())

}

[Deepak Jain]

I appended pwd command output to source directory, i still get same error.

pwd := util.ExecuteCommandWithOuput(exec.Command("pwd"))

fmt.Println("pwd", pwd)

util.ExecuteCommandWithOuput(exec.Command("cp", "-r", pwd+"/./*.json", artifact.dir))

Output:

Cmd:[cp -r /Users/userId/sd101 /./*.json myartifact] 

exit status 1: cp: /Users/userId/sd101

/./*.json: No such file or directory

I believed that above change should have worked. 

Any suggestions ?

Appreciate your time.

[Dave Cheney]

The problem is expanding shell meta characters like *, ? and ~ is a property of the _shell_, as Dan mentioned above. You are executing a command directly so the shell is not involved and cannot expand *.json into a list of files ending with .json.

A cheap solution to this might be something like

util.ExecuteCommandWithOuput(exec.Command("sh", "-c", cp", "-r", *.json", artifact.dir))

Which will pass your command to sh to run, and thus expand (untested, but should be close).

[Konstantin Khomoutov]

On Mon, 16 Jan 2017 13:35:07 -0800 (PST)
Deepak Jain <deep...@gmail.com> wrote:

> util.ExecuteCommandWithOuput(exec.Command("cp", "-r", "./*.json",
> artifact. dir))
>
> func ExecuteCommandWithOuput(cmd *exec.Cmd) {
> output, err := cmd.Output()
> if err != nil {
> log.Print("Error executing ", cmd.Args, err)
> }
> fmt.Print(string(output))
> }
>
> Output
>
> 2017/01/16 13:26:35 Error executing [cp -r ./*.json myartifact] exit
> status 1
>
> Questions
> 1. How do i get details of complete error message on failure of cp
> command ? I did have err != nill and Print err

The os/exec.Command() function constructs an value of type os/exec.Cmd
which is a struct type describing the details of an external process to
execute. Processes on Unix-like OSes (and Windows) have notions of the
standard streams [2]. The "stderr" is where (well-written) programs
write their error messages when they encounter a problem preventing
them from completing their intended task. Values of the os/exec.Cmd
have three fields, Stdin, Stdout and Stderr, which control the standard
streams of the process to be executed, and as it happens by default
these fields are initialized to nil which means the spawned process
will have its standard streams connected to the so-called "null device"
(/dev/null on Unices and NUL on Windows), and so its error output will
go to nowhere.

There are different approaches at handling error output of the external
processes.

1. You can connect its Stderr to the Stderr of your running process
which is available as os.Stderr, like this:

cmd := exec.Command(`cp ...`)
cmd.Stderr = os.Stderr

and then whatever that `cp` process outputs will end up on the
standard error stream of your host process.

2. You can "collect" that output and do something with it.

One approach is to exploit the fact pointers to values of
type bytes.Buffer implement the io.Writer interface, so you can do

var errmsg bytes.Buffer
cmd := exec.Command(`cp ...`)
cmd.Stderr = &errmsg
err := cmd.Run()
if err != nil || errmsg.Len() > 0 {
log.Fatalf("Failed to copy: %s\n", errmsg.String())
}

3. The CombinedOutput() method of the os/exec.Cmd type provides a way
to intelligently collect the outputs--to both the standard output
stream and the standard error stream--of the process being executed
and present them to you as a combined chunk of bytes.

Such collection is done in an intelligent way -- by keeping only
a manageable "header" and "trailer" parts of the output to deal with
cases where a process spews enormous amounts of data, and you don't
intend to process it all, and would be okay with just decorating
those collected bits with some error message prefix and so on.

> 2. Does exec.Command not support copy of files and recursive copy of
> directories ?

exec.Command() has nothing to do with copying anything.
As its name suggests, it's a tool to execute external processes,
and these processes can do anything. The `cp` program is for
copying/linking filesystem entities but you could execute FireFox or
Apache just as easily.

> 3. Any suggestions how i implement copy of files and recursive copy
> of directories ?

[...]

Answered over there at SO [1].
Here's my answer copied from there:

---------------->8----------------
The problem

To explain: the so-called "wildcards" are expanded by the shell in
which you typically execute command-line commands. That is, when you
invoke `cp -r ./*.json dir/`, the shell kicks in, expands *.json by
itself—producing a list of names of the files matching that pattern and
located in the current directory and pass the cp command a list of such
names.

So if you have, say, 10 matching files, the actuall call to cp will end
up looking like

cp -r file1.json file2.json ... dir/

When you pass call `cp ...` directly—without the shell kicking in and
expanding that *.json "fileglob" for you, the cp command receives the
name of a file "*.json" verbatim and attempts to open it. Since the
file named exactly "*.json" supposedly does not exist, `cp` fails and
exits.

The solutions

The first (admittedly lame) solution is to pass a call to cp "through"
a shell. That is, turn your call to cp into a shell script and pass it
to a shell.

The simplest way to do this is to use something like

exec.Command(`/bin/sh -c 'cp -r ./*.json manifest'`)

This will call a shell `/bin/sh` and pass it the script to execute via
its -c command-line option.

Another solution is to roll copying yourself using the standard Go
library: the functions of the path/filepath package provide support for
both expanding fileglobs as the shell would do it and iterating over
the entries of a given directory. Using either of these approaches you
can build the list of files to copy and/or iterate over them.

Then you can use the function os.OpenFile() to open the source and
destination files, and io.Copy() to copy the contents between them.
---------------->8----------------

Please, if you ask your question using several venues, make sure to
cross-reference between them.

1. http://stackoverflow.com/a/41690953/720999
2. https://en.wikipedia.org/wiki/Standard_streams

[roger peppe]

You don't really need to use the external cp command:
https://play.golang.org/p/F1YHzQL4UN

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Deepak Jain]

Thanks for pointing to https://github.com/juju/utils/blob/master/fs/copy.go

I was testing Copy function that recursively copies directories. I am able to copy first level of files and it creates directories. It fails to copy the contents within those directories. 

func Copy(src, dst string) error {...}

Error

$ triggerCopyCommand.go

lstat sd101-jvm/target/*.jar: no such file or directory ---> This is OK. As it does not exists.

lstat sd101-python/sd101/*: no such file or directory  ---> This exists in the directory from which Copy is invoked.

lstat sd101-r/sd101/*: no such file or directory   ---> This exists.

lstat sd101-shell/sd101/*: no such file or directory ---> This exists.

$ ls sd101-python/sd101/ 

__init__.py prepare setup.py train

$ 

$ ls sd101-r/sd101/

helloWorld.R

$ ls sd101-shell/sd101/

ls: sd101-shell/sd101/: No such file or directory

$ 

Any suggestions ?

Can this code be replicated https://github.com/juju/utils/blob/master/fs/copy.go ? Or do you recommend using it as a library ?

[Deepak Jain]

I fixed the error by invoking Copy command with abs source path.

srcPath, _ := 

e := util.Copy(srcPath, dst)

If i could modify into a single line

util.Copy(filepath.Abs(filepath.Dir(src)), dst)

Throws error as filepath.Abs(filepath.Dir(src)) returns a tuple.  

In scala i can use _1, _2 to access elements within a tuple. any suggestions in Go?

[Matt Harden]

In Go, don't try to combine those actions into a single line. Also you shouldn't ignore the error from filepath.Abs.

[roger peppe]

On 17 January 2017 at 18:50, Deepak Jain <deep...@gmail.com> wrote:

Thanks for pointing to https://github.com/juju/utils/blob/master/fs/copy.go

I was testing Copy function that recursively copies directories. I am able to copy first level of files and it creates directories. It fails to copy the contents within those directories. 

func Copy(src, dst string) error {...}

Error

$ triggerCopyCommand.go

lstat sd101-jvm/target/*.jar: no such file or directory ---> This is OK. As it does not exists.

lstat sd101-python/sd101/*: no such file or directory  ---> This exists in the directory from which Copy is invoked.

lstat sd101-r/sd101/*: no such file or directory   ---> This exists.

lstat sd101-shell/sd101/*: no such file or directory ---> This exists.

$ ls sd101-python/sd101/ 

__init__.py prepare setup.py train

$ 

$ ls sd101-r/sd101/

helloWorld.R

$ ls sd101-shell/sd101/

ls: sd101-shell/sd101/: No such file or directory

$ 

Any suggestions ?

It works fine for me. Can you provide a concrete code example that fails for you?

 

Can this code be replicated https://github.com/juju/utils/blob/master/fs/copy.go ? Or do you recommend using it as a library ?

That code is LGPL, so it's fine to copy it and use it as long as you keep its license.

But it's fine to use it as a library too.

  cheers,

    rog.

[roger peppe]

On 17 January 2017 at 19:14, Deepak Jain <deep...@gmail.com> wrote:

I fixed the error by invoking Copy command with abs source path.

srcPath, _ := 

e := util.Copy(srcPath, dst)

If i could modify into a single line

util.Copy(filepath.Abs(filepath.Dir(src)), dst)

Throws error as filepath.Abs(filepath.Dir(src)) returns a tuple.  

It should not be necessary to use an absolute path.

 

In scala i can use _1, _2 to access elements within a tuple. any suggestions in Go?

As Matt says, it's not good practice to ignore errors in Go. It's not always possible

to obtain the current directory and you really want to know if it's failed to do so.

 

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

[ÐΞ€ρ@Ҝ (๏̯͡๏)]

Thanks for time in responding. 

Appreciate.

--

Deepak