Tutorial or sample code for Golang storage package (Cloud Storage)?

[Ralf Rottmann]

I wonder whether anybody is aware of a good tutorial / sample code for using the storage package (Google Cloud Storage library)? The Godocs are pretty comprehensive but – as most of the package docs – totally lack examples even for the most basic use cases. I'd love to learn from an example, where e.g. a file is put to Cloud Storage and read back. Any help is greatly appreciated.

--

rottmann.net | Google+ | Twitter | Facebook | LinkedIn

[Glenn Lewis]

Have you seen this?  http://blog.golang.org/go-and-google-cloud-platform

Using the google-api-go-client is probably the easiest way to access all the RESTful APIs.

Once you get the hang of the google-api-go-client, things get easier: https://code.google.com/p/google-api-go-client/wiki/GettingStarted

Then, one of the greatest resources for playing with Google APIs is the Google API Explorer: https://developers.google.com/apis-explorer/#p/

In the explorer, you can fully play around with the APIs and see how they work.

For example, you can browse through the API, and let's say you want to perform a "storage.objects.list" operation in the API explorer...

You can go to: https://developers.google.com/apis-explorer/#p/storage/v1beta2/storage.objects.list and make sure you click the "Authorize via OAuth2" and then you can execute calls directly in the web browser by filling in the fields and clicking "Execute".

You can also use Cloud Storage without the google-api-go-client from Go with this: https://developers.google.com/storage/docs/reference-uris

Here is an example of its use: https://github.com/bradfitz/camlistore/blob/master/pkg/googlestorage/googlestorage.go

And of course, general Google Cloud Storage training: https://developers.google.com/storage/

I hope that helps.

-- Glenn

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ralf Rottmann]

Thanks Glenn,

Now I've got something to read for the reminder of this Sunday here in Germany. :-)

Do you happen to know, whether the google-api-go-client can be used without requiring a user's Google account? What if I have a backend, that should just enable user to post photos, without the need for having a Google account. How would the OAuth part work then? Would I use "my credentials" or some sort of "service credentials".

It's possibly all in the docs but in case you know from the top of your head, I'd appreciate your response.

[Glenn Lewis]

Yes, I believe that the App Engine app would use its own credentials to read/write from/to the Google Cloud Storage API without requiring any need for the end user to have a Google account.

Your app has a "Client ID" and "Client Secret" that are used to enable it to access GCS whether or not the end user has gone through its OAuth2 dance.

Only when you are accessing Google services on *behalf* of an end user (as if the user were interacting with the Google services directly themselves) do you need to issue the scopes that request specific authorizations that the user must agree to... and only *then* can you use those services on behalf of the user.

So, for example, if your app wanted to create a G+ post on my behalf, your app must perform the OAuth2 dance and request the scope that enables it to make G+ posts, and then I have to agree to that authorization before you can call that API for me from within your app.

However, if your app is simply writing to its GCS buckets and doesn't require any particular end-user ownership of those files, then there is no need for any OAuth2 dance and no need for scopes, and your app simply uses its client ID/secret combo to access GCS.

I hope that makes sense.

Please let me know if it doesn't.

-- Glenn

--

[pfi...@gmail.com]

Hey Glenn,

Sorry to bring up an old thread but this is the best resource I've found for GO/GCS documentation so far :p.

I've been trying to do a simple photo upload to GCS via Golang server so I'm most curious about your last line: "However, if your app is simply writing to its GCS buckets and doesn't require any particular end-user ownership of those files, then there is no need for any OAuth2 dance and no need for scopes, and your app simply uses its client ID/secret combo to access GCS."

I've been trying to do something like: 

client, err := serviceaccount.NewClient(context, "https://www.googleapis.com/auth/storage")

service, err := storage.New(client)

file, _, err := r.FormFile("photo")

objectsService := storage.NewObjectsService(service)

storageObject, err := objectsService.Insert("testphotouploader.appspot.com", &storage.Object{Name: "test"}).Media(file).Do()

I'm getting an error on the first line: API error 9 (app_identity_service: UNKNOWN_SCOPE)

So it sounds like I shouldn't be worried about finding the correct scope to append but rather there is a completely different way to get an authorized http client that I am unable to find in any docs/samples scattered throughout the interwebs. I was kind of assuming this would "just work" because it is my app engine account that is connected to the GCS but I suppose that was a bit naive. I have my clientID/private key but I can not for the life of me find any examples of how to use them.

Any insight you could share or documentation/samples out there would be extremely helpful. 

Thanks a lot

-Pat

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Glenn Lewis]

Hi Pat,

Brian Dorsey gave a great talk at Gophercon 2014 on how to get started on the Google Cloud Platform with Go, which included talking to Google Cloud Storage.

Here are his instructions:

Lab 1 - Making API calls (Cloud Storage)

• Start by getting this running: https://github.com/bensonk/gophercon-samples/blob/master/storage.go

• Clone the repo, or download the file

• Edit the file to include your project specific info

• Client id

• Client secret

• Your project ID

• A bucket name and object name of your choosing

• run storage.go

• Troubleshooting

• If you’re having trouble with the client secrets, consider creating a new service account. Go to the developers console, APIs & Auth → Credentials → CREATE NEW CLIENT ID → Installed Application → other. And try those credentials.

• Modify it to do something different

• List the bucket

• Save object to a new file, rather than stdout

• Change the ACLs to make the object public

Also, you may find these links helpful:

https://github.com/GoogleCloudPlatform/storage-getting-started-go

https://github.com/GoogleCloudPlatform/storage-getting-started-go/blob/master/storage-sample.go

In place of "v1beta2", you should use "v1", which is located here:

https://code.google.com/p/google-api-go-client/source/browse/storage/v1/storage-gen.go

Also, I have posted a simple app on this list in the past that demonstrates saving an image to Google Cloud Storage:

$ cat app.go

package main

import (

"net/http"

"os"

"appengine"

"code.google.com/p/goauth2/appengine/serviceaccount"

storage "code.google.com/p/google-api-go-client/storage/v1"

)

func init() {

http.HandleFunc("/", handler)

}

func handler(w http.ResponseWriter, r *http.Request) {

if r.URL.Path != "/" {

http.NotFound(w, r)

return

}

c := appengine.NewContext(r)

client, err := serviceaccount.NewClient(c, storage.DevstorageRead_writeScope)

if err != nil {

c.Errorf("failed to create service account client: %v", err)

return

}

c.Infof("GML: client.Transport=%#v", client.Transport)

service, err := storage.New(client)

if err != nil {

c.Errorf("failed to create storage service: %v", err)

return

}

// now use the service methods documented here:

//   http://godoc.org/code.google.com/p/google-api-go-client/storage/v1

bucket := "gmlewis-test.appspot.com"

name := "gopher.png"

b, err := os.Open(name)

if err != nil {

c.Errorf("unable to open %v: %v", name, err)

return

}

defer b.Close()

obj, err := service.Objects.Insert(bucket, nil).Name(name).Media(b).Do()

if err != nil {

c.Errorf("unable to insert %v into cloud storage: %v", err)

return

}

// You can also set the "*Object" instead of "nil" to set the ACLs.

// or you can change the ACLs after you get the "obj" back.

c.Infof("inserted object %#v into bucket %v, name %v", obj, bucket, name)

}

I hope this helps.

-- Glenn

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

[Kun Li]

Hi Pat:

Try using "https://www.googleapis.com/auth/devstorage.read_write"
instead of "https://www.googleapis.com/auth/storage" as the parameter
to pass into newClient.
----------------------------------------------------------------------------------
Yours Sincerely
Kun

gplus.to/kunli

>>> email to google-appengin...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>

> --
> You received this message because you are subscribed to the Google Groups
> "google-appengine-go" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to google-appengin...@googlegroups.com.

[pfi...@gmail.com]

Thank you Glenn and Kun,

The issue was using /storage instead of /devstorage and then I had to enable the Cloud Storage JSON API along with the Cloud Storage API. Working great now! Thanks again

-Pat

>>> email to google-appengine-go+unsub...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "google-appengine-go" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to google-appengine-go+unsub...@googlegroups.com.

[Prateek Malhotra]

The API provides the different scopes for you: https://code.google.com/p/google-api-go-client/source/browse/storage/v1/storage-gen.go#43

I've written a wrapper around communication with the storage API and would like to share a couple tips/tricks:

• Fetching objects using the JSON API is broken in Go due to a bug in the net/http package, you have to be very wary on how you select your object names or use a different endpoint altogether (e.g. storage.googleapis.com/<bucket>/<object> for fetching objects using the same authorized client you'd pass in to storage.New()) another helpful tidbit for this: http://play.golang.org/p/TkKRROJTfb (Update to the current API version). You can break apart MediaLink's and use this method to fetch objects.
  
• You don't have to "emulate" GCS locally on the devserver if you don't want to. You can authorize your client locally for access to the actual API. 'jcgr...@google.com (Joe Gregorio)' wrote a "get_refresh_token.py" script to do a local web server oauth2 flow and store a JWT  locally which you can rig your auth client in Go to read in if its on the dev server
• From what I can tell you are limited to 10MB uploads since the default Go client for GCS does not do multi-part uploads and AppEngine restricts you to a 10MB max POST data size on a URL fetch API call
• Watch your timeouts! You have 60 seconds to do a url fetch call (when you manually increase the timeout to its max), so if you plan to list an entire "folder" with millions of objects, think about doing it in batches and using the NextPageToken to "paginate" through

My usage has been mostly Get (occasionally with Generation IDs), Put (occasionally with Generation ID Matches), ListDir, Delete, DeleteDir, & Compose. I hope this information is helpful!

Thank you,

Prateek

[Robin Andersson]

I am still a bit confused about the Service Account Authentication on the development server. When I try to do any action on the scope storage.DevstorageFull_controlScope it just says: error: googleapi: Error 401: Invalid Credentials

The only values that see I can configure are:

- ProjectID

- Bucketname

Do I need to setup the project and/or bucket somewhere for this to work on the development server? And should I setup a separate development project to differ between production and dev?

Trying to write an example app engine app for Service Account Auth based on the Google oauth2 example: https://github.com/robinwassen/go-cloud-storage-app-engine-example

[Holger Knauer]

I had ran into the same problem (I think).
Maybe you can reuse my solution.

[Robin Andersson]

Thank you Holger, I really appreciate it. :)

[Prateek Malhotra]

Someone over at Google sneaked in some command line arguments for the dev server that aren't well documented, but I would think they're intended to be used to do what we all found different ways of achieving (I haven't tested this myself yet, but if someone does or has, let us know the results!):

# App Identity

appidentity_group = parser.add_argument_group('Application Identity')

appidentity_group.add_argument(

'--appidentity_email_address',

help='email address associated with a service account that has a '

'downloadable key. May be None for no local application identity.')

appidentity_group.add_argument(

'--appidentity_private_key_path',

help='path to private key file associated with service account '

'(.pem format). Must be set if appidentity_email_address is set.')

[Holger Knauer]

I tried those arguments, but without success.

[Holger Knauer]

Looks like the former package at code.google.com/p/goauth2 is being redone right now. The new version is at https://github.com/golang/oauth2

Have a look at http://godoc.org/github.com/golang/oauth2/google and the ServiceAccountsJSON example on that page. This is really the easiest way to interact with Google APIs on the App Engine dev server now.
Keep in mind that you need to set JWTConfig.Client and JWTConfig.Transport to a client and transport created via the urlfetch package on appengine, though.