subprocess stdout truncated

[Han-Wen Nienhuys]

Hi,

I am having a problem grabbing the stdout of a shell command: I am
running an exec.Cmd as follows,

{/bin/sh [/bin/sh -c echo aaaa..1500 bytes.. a | cat] }

stdout := &bytes.Buffer{}
cmd.Stdout = stdout
err := cmd.Run()
result := stdout.String()

the result I get back is 1024 bytes, rather than 1501.

Any ideas?

I'm think it's not strictly a problem with the exec package or Go:
I've seen this with my own os.StartProcess() code, and recently in
unrelated python code that used subprocess.Popen().communicate().

The full code is at

https://github.com/hanwen/termite/blob/master/termite/task.go#L27

failing test:

https://github.com/hanwen/termite/blob/master/termite/worker_test.go#L245

--
Han-Wen Nienhuys
Google Engineering Belo Horizonte
han...@google.com

[David Symonds]

On Sat, Aug 20, 2011 at 1:46 PM, Han-Wen Nienhuys <han...@google.com> wrote:

> I am having a problem grabbing the stdout of a shell command: I am
> running an exec.Cmd as follows,
>
> {/bin/sh [/bin/sh -c echo aaaa..1500 bytes.. a | cat] }
>
>        stdout := &bytes.Buffer{}
>        cmd.Stdout = stdout
>        err := cmd.Run()
>        result := stdout.String()
>
> the result I get back is 1024 bytes, rather than 1501.
>
> Any ideas?

That seems like it should work. You might find it easier to replace
those four lines with:
result, err := cmd.Output() // result will be []byte, not string
That's almost exactly what you've got there.

The 1024 looks suspiciously like there's some buffering that's not
being flushed.

Dave.

[Han-Wen Nienhuys]

I forgot to mention that I am also redirecting stderr. Unfortunately,
I need both stdout and stderr, and I need them separately, so
cmd.Output() doesn't work for me.

[Han-Wen Nienhuys]

On Sat, Aug 20, 2011 at 4:53 PM, Han-Wen Nienhuys <han...@google.com> wrote:
>> The 1024 looks suspiciously like there's some buffering that's not
>> being flushed.

Also, which part system is using a 1024 byte buffer? Pipes have a 4k
(pagesize) minimum, and io.Copy has a 32k buffer?

The same problem also happens if I remove the " | cat" part of the command.

[Gustavo Niemeyer]

> I am having a problem grabbing the stdout of a shell command: I am
> running an exec.Cmd as follows,

FWIW, the logic you mention seems to work fine here:

http://paste.ubuntu.com/671244/

Prints 1501.

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/plus
http://niemeyer.net/twitter
http://niemeyer.net/blog

-- I never filed a patent.

[Archos]

On Aug 20, 8:53 pm, Han-Wen Nienhuys <han...@google.com> wrote:
> I forgot to mention that I am also redirecting stderr.  Unfortunately,
> I need both stdout and stderr, and I need them separately, so
> cmd.Output() doesn't work for me.

It works if it's used so:

var cmdName string
var cmdArgs []string

cmd := exec.Command(cmdName, cmdArgs...)
cmd.Env = os.Environ()
cmd.Stdin = os.Stdin

var stderr bytes.Buffer
cmd.Stderr = &stderr

out, err := cmd.Output()

[Archos]

Well, this is the necessary code to get separately both Stderr and
Stdout:

---

var stderr bytes.Buffer
cmd.Stderr = &stderr

out, err := cmd.Output()

---

cmd.Output() returns the Stdout in []byte, so it is only necessary to
add a buffer for Stderr before of call to Output()

http://golang.org/src/pkg/exec/exec.go?s=7190:7231#L284

[Han-Wen Nienhuys]

On Sat, Aug 20, 2011 at 5:03 PM, Han-Wen Nienhuys <han...@google.com> wrote:
> On Sat, Aug 20, 2011 at 4:53 PM, Han-Wen Nienhuys <han...@google.com> wrote:
>>> The 1024 looks suspiciously like there's some buffering that's not
>>> being flushed.
>
> Also, which part system is using a 1024 byte buffer? Pipes have a 4k
> (pagesize) minimum, and io.Copy has a 32k buffer?
>
> The same problem also happens if I remove the " | cat" part of the command.

PEBKAC

I had forgotten a * when I copied over the response for printing,
causing me to trim not only the response for printing but also RPC
response. The 1024 set me on a wild goose chase for buffering
problems.

With great ':=' comes great responsibility.

Sorry for the confusion and thanks everyone,