SDK aws S3 Download to stdout

[Walter Garcia]

Hello all.

Im new using AWS SDK in go and Im need download file object from S3 to Stdout and not create a file.
For example:
my_download_s3_program > file.txt
or
my_download_s3_program | tar
etc

So, I can download to file, but I need stream on the fly to stdout, it's possible?

This example write a file:
----------------------------------------------------------------------
            file, err := os.Create("download_file.txt")
            if err != nil {
                fmt.Println("Failed to create file", err)
                os.Exit(4)
            }
            defer file.Close()

            key := path.Join(prefix, filename)

            downloader := s3manager.NewDownloader(session.New(&aws.Config{Region: aws.String("us-west-2")}))
            numBytes, err := downloader.Download(f,
                &s3.GetObjectInput{
                    Bucket: aws.String(bucket),
                    Key:    aws.String(key),
                })
            if err != nil {
                fmt.Println("Failed to download file", err)
                os.Exit(5)
            }

            fmt.Println("Downloaded file", file.Name(), numBytes, "bytes")

----------------------------------------------------------------------

How can modify this example to stream to stdout?

Well, thanks in advance.

[Dave Cheney]

Try using os.Stdout as the download target.

[Joshua Boelter]

I've been tinkering w/ the AWS Go SDK to get used to it ... this is one method using the GetObject API; see the full gist for a working example

https://gist.github.com/jboelter/6f5bd598673eb0e606f10660495fc175

s3Svc := s3.New(awsSession)

result, err := s3Svc.GetObject(&s3.GetObjectInput{

Bucket: aws.String(bucket),

Key:    aws.String(key),

})

if err != nil {

fmt.Fprintln(os.Stderr, err)

os.Exit(1)

}

n, err := io.Copy(os.Stdout, result.Body)

result.Body.Close()

[Dave Cheney]

numBytes, err := downloader.Download(os.Stdout, &s3.GetObjectInput{ ... })

Should do it.

[Walter Garcia]

I had already tried this option but does not work
this is the error

Failed to download file write /dev/stdout: illegal seek

Thanks in advance

[Walter Garcia]

Excellent , this work fine

I attach the final sample code

            s3Svc := s3.New(session.New(&aws.Config{Region: aws.String("us-west-2")}))

            result, err := s3Svc.GetObject(&s3.GetObjectInput{
                Bucket: aws.String(bucket),
                Key:    aws.String(key),
            })

            if err != nil {
                fmt.Fprintln(os.Stderr, err)
                os.Exit(1)
            }

            io.Copy(os.Stdout, result.Body)
            result.Body.Close()

Thanks so much

[Joshua Boelter]

If you want to use the s3manager Downloader to gain parallel/chunked downloads, you can use a temp buffer like so.  The seek error your getting is because it's using a io.WriterAt interface for io and pipes can't seek when you chain commands together (at least that I know of).

https://gist.github.com/jboelter/ecfb08d6a18440ac16d93b5183aad207

buff := &aws.WriteAtBuffer{}

s3dl := s3manager.NewDownloader(awsSession)

n, err := s3dl.Download(buff, &s3.GetObjectInput{

Bucket: aws.String(bucket),

Key:    aws.String(key),

})

if err != nil {

fmt.Fprintln(os.Stderr, err)

os.Exit(1)

}

n2, err := io.Copy(os.Stdout, bytes.NewReader(buff.Bytes()))