Google API client: downloading files from Cloud Storage

[Ian Rose]

Howdy -

I'm just getting started using the google-api-go-client with Cloud Storage and am able to list files in a bucket, get metadata on a single file, etc. but I can't figure out how to download a file.  As I would expect, I can't simply GET the MediaLink that is returned with each file's metadata (it gives me a "Login Required" error).  But I don't see any functions in v1beta2/storage-gen.go for requesting/downloading file contents.  I do see the reverse (uploading file contents) however...

Thanks very much!

- Ian

Apologies if this is the wrong forum for this question (please direct me to the right place!)...

[Kyle Lemons]

Can you provide source code?  You should be able to make an authenticated GET request using your oauth token.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ian Rose]

Here is an example program.  Note that I am running on Google Compute Engine, so I can make use of service accounts, which means that I don't manipulate the access-token directly at all.  Although I could make a generic REST call to the JSON API to accomplish what I want, but I'd of course rather use the client library just for convenience.

package main

import (

"code.google.com/p/goauth2/compute/serviceaccount"

"code.google.com/p/google-api-go-client/storage/v1beta2"

"fmt"

"os"

)

func main() {

client, err := serviceaccount.NewClient(&serviceaccount.Options{})

if err != nil {

fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n", err)

os.Exit(1)

}

filename := "5105650963054592/2014-01-27-15-01-29-01-04-146bc100000000-12972000000000-11fbc100000000.json"

bucket := "fs-staging-cooked-events"

service, _ := storage.New(client)

obj, err := service.Objects.Get(bucket, filename).Do()

if err != nil {

fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)

os.Exit(1)

}

fmt.Printf("Found %s/%s...\n", bucket, filename)

fmt.Printf("MediaLink: %s\n", obj.MediaLink)

fmt.Printf("SelfLink: %s\n", obj.SelfLink)

[Carlos Castillo]

Are you using an oauth2 authenticated http.Client to make the request? The one created on line 1 of the main function should be authenticated if the API calls work.

eg, did you do:

client.Get(obj.MediaLink)

or: 

http.Get(obj.MediaLink) 

[Ian Rose]

Yes, as you can see in my example code, I am using an authenticated http.Client obtained from serviceaccount.NewClient().  And I know that this client works because I can use it to do things like list the objects in a bucket.

If I try client.Get(obj.MediaLink), then I receive a 404 from the server.  To test things out, I tried using this same client to hit one of my local servers and from this I was able to confirm that the request includes a "Authorization: Bearer <token>" header, as expected.  However I'm wondering if there are some other headers that I need to set in my request?

[Carlos Castillo]

Just covering the bases, since your posted code didn't show how it performed the actual download (your code doesn't show the line that you claim "fails"). BTW, your posted code ignores the error from storage.New().

[Ian Rose]

Good point - I probably copy-and-pasted the storage.New line and forgot to add back the err variable.  I'll fix that.

In other news, I just tried with a different bucket & filename that are much shorter than the "real ones" (as shown in my sample code) and doing client.Get worked!  So this might be user error on my part, or maybe there is some weird interaction with long file names?  Stay tuned...

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/juguXl-ss2Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Ian Rose]

I believe I have narrowed the problem down to the fact that I am trying to download files from within a folder.  As a test case, I created a new bucket "test987" and uploaded two json files, one without an enclosing folder and one to a folder named "foo".  So the structure is like:

test987 bucket:

  - outside.json

  - foo/

    - inside.json

I can fetch metadata (including the MediaLink) on both outside.json and foo/inside.json.  However if I think client.Get() the MediaLink in each case, it works on outside.json but returns a 404 on foo/inside.json.  Here is my updated program:

package main

import (

"code.google.com/p/goauth2/compute/serviceaccount"

"code.google.com/p/google-api-go-client/storage/v1beta2"

"fmt"

"io/ioutil"

"os"

)

func main() {

client, err := serviceaccount.NewClient(&serviceaccount.Options{})

if err != nil {

fmt.Fprintf(os.Stderr, "Failed to create service account client: %v\n", err)

os.Exit(1)

}

filename := "foo/inside.json"

bucket := "test987"

service, err := storage.New(client)

if err != nil {

fmt.Fprintf(os.Stderr, "Failed to create GCS client client: %v\n", err)

os.Exit(1)

}

obj, err := service.Objects.Get(bucket, filename).Do()

if err != nil {

fmt.Fprintf(os.Stderr, "Failed to GET %s/%s: %v\n", bucket, filename, err)

os.Exit(1)

}

fmt.Printf("Found %s/%s...\n", bucket, filename)

fmt.Printf("MediaLink: %s\n", obj.MediaLink)

url := obj.MediaLink

fmt.Printf("Fetching %s\n", url)

rsp, err := client.Get(url)

if err != nil {

fmt.Fprintf(os.Stderr, "Failed to direct fetch file: %v\n", err)

os.Exit(1)

}

fmt.Printf("Status code: %d\n", rsp.StatusCode)

fmt.Printf("Reading %d bytes of response body...\n", rsp.ContentLength)

content, err := ioutil.ReadAll(rsp.Body)

if err != nil {

fmt.Fprintf(os.Stderr, "Failed to ready body of fetch-file response: %v\n", err)

os.Exit(1)

}

fmt.Printf("Read %d bytes from response body...\n", len(content))

fmt.Println("")

fmt.Println(string(content))

}

And here is the output when filename is set to "foo/inside.json":

Found test987/foo/inside.json...

MediaLink: https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media

Fetching https://www.googleapis.com/storage/v1beta2/b/test987/o/foo%2Finside.json?generation=1390881110333000&alt=media

Status code: 404

Reading -1 bytes of response body...

Read 9 bytes from response body...

Not Found

And here is the output when filename is set to "outside.json":

Found test987/outside.json...

MediaLink: https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media

Fetching https://www.googleapis.com/storage/v1beta2/b/test987/o/outside.json?generation=1390881123389000&alt=media

Status code: 200

Reading 798 bytes of response body...

Read 798 bytes from response body...

{

  "albums": [

    (...content truncated for brevity...)

  ]

}

[Kyle Lemons]

Shot in the dark; if that %2F is a /, does it work?

[Ian Rose]

On the contrary - it has to be %2F and in fact therein lays the problem; net/url (as used by an http.Client) "normalizes" escape sequences which means that it will actually replace the %2F in the URL with a '/' and thus the request fails.  The solution is to make use of the Opaque property.

Full discussion here:

https://plus.google.com/111198546322070924637/posts/Tzw3QZqEQZk

- Ian