Problem with Cloud Storage ACLs and Golang's file package

[Ralf Rottmann]

We've added a bucket and set the permission for ANY user to Read.
The docs state, that any item created / added into a bucket, will inherit the bucket's ACL and permissions.
We then create a file in the bucket using Golang's file package.

However, when trying to access the file with an anonymous, not-at-all authenticated user, we get a permission denied error.

The item is only accessible to those users when we use Google Developer Console and set it to public.

Any idea? Are the docs just wrong?
--
--
rottmann.net | Google+ <https://plus.google.com/106658337982640166595> |
Twitter <http://twitter.com/ralf> | Facebook<http://www.facebook.com/ralfrottmann>|
LinkedIn <http://de.linkedin.com/in/rottmann>

[Glenn Lewis]

To help debug this, can you please send me the output of:

gsutil acl get gs://your-apps-bucket/path/file

twice... once for a file that works correctly (is visible), and once for a file that does not work?

Also, can you send the bucket's ACL and permissions?

I'm hoping that "gsutil acl get gs://your-apps-bucket" will do the trick.

Finally, can you show the values found on your Cloud storage panel here:

https://code.google.com/apis/console/b/0/?noredirect#project:<your-project-number>:storage

To get you up and running, I would recommend that you set the ACLs using the RESTful API immediately upon file creation while we try to debug this.

-- Glenn

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ralf Rottmann]

I'm on the iPad, it's 00:42 here in Germany. We need gsutil for iOS :-) I will provide the output tomorrow. Meanwhile, do you happen to have a go-api-client storage sample that leverages a service client (oauth) versus a real authenticated user?

[Glenn Lewis]

No, I don't, but it is on my list of things to do. Maybe someone else on this list has a working example they can share.

[Erik Troan]

I don't have a fully baked example, but this is cut and paste from working code:

import(

    storage "code.google.com/p/google-api-go-client/storage/v1beta2"

    "code.google.com/p/goauth2/appengine/serviceaccount"

)

webClient, err := serviceaccount.NewClient(c, arrayOfScopes []string)

storeClient, err := storage.New(webClient)

err := storeClient.Objects.Delete(bucket, objectName).Do()

---

As you can see, once you're in appengine there really isn't much to it. Make sure you've enabled the JSON api and that your appengine projects has the right permissions to the bucket (user permission for projectId@appspot.gserviceaccount.com)

I hope this helps.

Erik

[ralf.r...@grandcentrix.net]

So, Erik, are you saying I don't need any of the OAuth2 dance?

{

Wir suchen Verstärkung für unser Team: http://du.bist.ein.guru?

}

grandcentrix GmbH

Schanzenstrasse 6-20
51063 Köln, Deutschland

phone: +49 221 677 860 0

fax:    +49 221 677 860 99

Amtsgericht Köln | HRB  70119 | Geschäftsführer: R. Rottmann, M. Willnow | USt.-IdNr.: DE266333969

[Ralf Rottmann]

Sorry for double-posting. Google multi-account foo. :-)

Erik: Are you saying that I can totally forget about the entire OAuth2 dance when using a service account?

On Friday, April 4, 2014 12:25:40 AM UTC+2, Ralf Rottmann wrote:

--

rottmann.net | Google+ | Twitter | Facebook | LinkedIn

[Erik Troan]

That's excactly what I'm saying. This does use the gooauth2 library, which has specific support for appengine, and appengine has specific support for generating the magic you need for your project to authenticate to other google services.

No dancing required.

Erik

--

[Tim Greiser]

On Friday, April 4, 2014 6:00:18 AM UTC-7, Erik Troan wrote:

I don't have a fully baked example, but this is cut and paste from working code:

import(

    storage "code.google.com/p/google-api-go-client/storage/v1beta2"

    "code.google.com/p/goauth2/appengine/serviceaccount"

)

webClient, err := serviceaccount.NewClient(c, arrayOfScopes []string)

storeClient, err := storage.New(webClient)

err := storeClient.Objects.Delete(bucket, objectName).Do()

---

As you can see, once you're in appengine there really isn't much to it. Make sure you've enabled the JSON api and that your appengine projects has the right permissions to the bucket (user permission for projectId@appspot.gserviceaccount.com)

Any suggestions for trying to debug what credentials are being used? I've tried a bunch of variations on this, but I always get "googleapi: Error 401: Invalid Credentials".

To clarify:
projectId@appspot.gserviceaccount.com

This is the same project ID that is in my site URL? <project-id>.appspot.com

Thanks,

Tim.

[Glenn Lewis]

There are several steps involved... particularly regarding setting up the credentials...

At Gophercon 2014, Brian Dorsey put together a great Google Cloud Platform Workshop and has graciously made the presentation available here:

http://bit.ly/gcp-gophercon-slides

(or presentation view here: https://docs.google.com/a/google.com/presentation/d/1801ZRYkWrtyZeSZ6mHnZf9x4FPIvkncv3hnTxigCyHg/present#slide=id.g12c783db6_10 )

It includes setting up an example app in Go that accesses GCS from Go on App Engine.

The actual examples are located here: http://bit.ly/gcp-gophercon

This should help you get started.

Thank you, Brian!

-- Glenn

--

[Tim Greiser]

On Wednesday, April 30, 2014 10:53:13 PM UTC-7, Glenn Lewis wrote:

The actual examples are located here: http://bit.ly/gcp-gophercon

This should help you get started.

I followed along with the examples here: http://bit.ly/gcp-gophercon

Lab 0:

  Did this for a brand new project, and also enabled "URL Shortener API".

Lab 3:

  Deployed https://github.com/GoogleCloudPlatform/appengine-goshorten to my new project. The only thing I changed was the application identifier in app.yaml

When I run it locally in a dev server, I still get:

  error getting history: googleapi: Error 401: Invalid Credentials

When I deploy and run it on appengine I get:

  error getting history: googleapi: Error 403: User Rate Limit Exceeded. Please sign up

Here is the test application: https://cloud-platform-test.appspot.com/

Tim.

[Tim Greiser]

On Thursday, May 1, 2014 1:57:40 PM UTC-7, Tim Greiser wrote:

When I deploy and run it on appengine I get:

  error getting history: googleapi: Error 403: User Rate Limit Exceeded. Please sign up

Eventually on the deployed version this just started working. Not sure if my testing triggered a rate limit response that blocked me temporarily, but I got this false positive error for about 24 hours and then it just went away without me changing anything.

I dug a little deeper on the dev server issue and it appears that the underlying call to appengine.AccessToken isn't intended to work on dev (it would be nice if AccessToken() returned an error to this effect). The only place I found this referenced is at: https://code.google.com/p/googleappengine/issues/detail?id=9165

Tim.

[Ralf Rottmann]

Glenn, where would you cache the token on App Engine? Brian's talk uses a local file (cache.json) which would not work on App Engine.

[Glenn Lewis]

I'm sorry, I don't understand the question.

If you are using the serviceaccount that makes it easier to talk to GCS, it stores the token within its transport and refreshes the token for you when it expires so that you don't have to be concerned about the token at all:

https://code.google.com/p/goauth2/source/browse/appengine/serviceaccount/serviceaccount.go

-- Glenn

[Matthew Zimmerman]

I use something like this and set it as the TokenCache in the
oauth.Config struct

http://play.golang.org/p/GwfnKWxUWj

(it doesn't compile in play)

[Matthew Zimmerman]

Even better:

https://code.google.com/p/goauth2/source/browse/appengine/serviceaccount/cache.go

[Holger Knauer]

I'm in the same situation as Tim right now: Access to GCS using the packages

code.google.com/p/goauth2/appengine/serviceaccount and
code.google.com/p/google-api-go-client/storage/v1beta2

works just fine on the live server. It doesn't so on the dev server, though, because the token returned by
appengine.AccessToken(c, storage.DevstorageFull_controlScope)
just contains:"InvalidToken:https://www.googleapis.com/auth/devstorage.full_control:93.5629999638"

I took a blind shot and tried what might happen if I created serviceaccount credentials and key in the developer console and give them as "--appidentity_email_address" and "--appidentity_private_key_path" to dev_appserver.py.
In fact, calling appengine.AccessToken() now produces this error:

API error 1003 (app_identity_service: UNKNOWN_ERROR): Error getting access token. Response code: 400, Content: {
  "error" : "invalid_grant"
}


So the question remains: How to get access to GCS from the dev_server? Any of you guys figured out a way how to do it?

[Erik Troan]

I explicitly check if I'm running on devserver, and if so load keys from a local PEM file and use that keys for authentication. A bit of a pita but works fine.

Erik

--

[Holger Knauer]

Thanks for the heads up, that's what I figured as well.

To make the whole thing more comfortable I went ahead and developed a drop-in replacement for code.google.com/p/goauth2/appengine/serviceaccount to be used on the devserver, basically taking the original package as a template and combining it with bits and pieces from code.google.com/p/goauth2/oauth/jwt, see the attached file. With that I can now simply do:

c := appengine.NewContext(r)
var client *http.Client
if appengine.IsDevAppServer() {
    client, _ = devserviceaccount.NewClient(c, "path-to/client-secret.json", "path-to/privatekey.pem", storage.DevstorageFull_controlScope)
} else {
    client, _ = serviceaccount.NewClient(c, storage.DevstorageFull_controlScope)
}
service, err := storage.New(client)

As second and third parameter devserviceaccount.NewClient() takes the path to the json file and private key that can be downloaded when creating service-account credentials in the developer console. The downloaded key must have been converted to PKCS1 (PEM) format. Apart from that, devserviceaccount behaves identically to serviceaccount, i.e. stores the token in memcache and refreshes it automatically once expired.

Hope this helps whoever happens to be in the same boat I was yesterday.

[hamish...@gmail.com]

I spun my wheels on this for a while with bigquery, but just found an easier way that allows you to use the pem and service account with dev_appserver.py locally

Step 1. Convert the PKCS12 format to PKCS1 format using the following command:

cat /path/to/xxxx-privatekey.p12 | openssl pkcs12 -nodes -nocerts -passin pass:notasecret | openssl rsa > /path/to/secret.pem

Step 2. Run the dev_appserver command as per below, replacing the relevant CAPS:

dev_appserver.py PROJECT --appidentity_email_address XX...@developer.gserviceaccount.com --appidentity_private_key_path /PATH-TO/FORMATTED-PKCS1.PEM

Then you can simply create a service as normal, e.g. 

config := google.NewAppEngineConfig(c, []string{

   "https://www.googleapis.com/auth/bigquery",

})

client := http.Client{Transport: config.NewTransport()}

bqs, err := bigquery.New(&client)

if err != nil {

c.Errorf("BQ (error): %v", err)

}

I got the tip from this thread on stackoverflow: http://stackoverflow.com/questions/20349189/unable-to-access-bigquery-from-local-app-engine-development-server

[Holger Knauer]

I had actually tried this before (see post above) but couldn't get it to work. Thanks to your post I just tried again and can confirm it works:
appengine.SignBytes() and appengine.AccessToken() (the latter being used internally in the transport you get from config.NewTransport() as shown in your code above.) both work on dev server now if -appidentity_email_address and -appidentity_private_key_path are set.

Funny thing is that I used the exact same shell command as in May, still had it as comment in my dev startup script. Maybe something changed in the dev server scripts since then (I'm on Win 64)?