Can't write to local GCS file (python)

[Mike Lucente]

I'm getting "Access Denied" trying to open a GCS file for writing. I'm using these instructions: https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/read-write-to-cloud-storage. Googling not helping much. Thanks in advance.

 

Here's my code and the error:

        bucket_name = os.environ.get('BUCKET_NAME', app_identity.get_default_gcs_bucket_name())

        filename = '/' + bucket_name + '/' + 'myfile'

        print filename

        gcs_file = gcs.open(filename,

                            'w',

                            content_type='text/plain')

ForbiddenError: Expect status [201] from Google Storage. But got status 403.

Path: '/app_default_bucket/myfile'.

Request headers: {'accept-encoding': 'gzip, *', 'content-type': 'text/plain', 'x-goog-api-version': '2', 'x-goog-resumable': 'start'}.

Response headers: {'server': 'UploadServer', 'vary': 'Origin', 'content-length': '210', 'x-guploader-uploadid': 'AEnB2UrEb9QG7lLMhN-Fu_kjPLbVwSuKfufgxIufy37ENZq0bK97IMaPdIuxA0ao4pTfuGAhZyi5hWzG5eqvBKEFk0nwheghJbkH4MlUSgFdKVzkFgt6rO8', 'content-type': 'application/xml; charset=UTF-8', 'alt-svc': 'quic=":443"; ma=2592000; v="36,35,34"', 'date': 'Sat, 03 Dec 2016 17:33:41 GMT'}.

Body: "<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Caller does not have storage.objects.create access to bucket app_default_bucket.</Details></Error>".

Extra info: None.

[Jason Collins]

App Engine accesses the Cloud Storage bucket using a service account, so likely that service account doesn't have write access to that bucket.

If you are accessing the default bucket (it appears you are, though the error message doesn't look like it), then the service account usually has access unless someone has revoked/changed that permission. You can repair your configuration using the Admin API repair method: https://cloud.google.com/appengine/docs/admin-api/reference/rest/v1/apps/repair   You can use the form at the bottom of the page to invoke it.

[Adam (Cloud Platform Support)]

Interestingly, your code works fine for me on the development server without any application default credentials set, or when setting GOOGLE_APPLICATION_CREDENTIALS to a service account key without any role set.

The name app_default_bucket would be the bucket name returned on the development server with app_identity.get_default_gcs_bucket_name(), but it appears that gcs.open() is calling the production GCS XML API. Could you share the full contents of the source file and the steps you're using to run the app?

[Mike Lucente]

I've tabled this effort for now (though it's still a problem). I was trying to save a file so that I could then do a "LOAD DATA INFILE" to improve performance. But I think that there's a bigger issue with latency of my cloud sql instance.

Please see the following: https://groups.google.com/forum/#!topic/google-appengine/07f3QCOT7qQ

Thanks.

[Adam (Cloud Platform Support)]

Fair enough. Using GCS to do a "LOAD DATA INFILE" would not work in any case, as Cloud SQL has no special connector for recognizing a file saved in Google Cloud Storage (Cloud Storage import and export is a separate managed service). "LOAD DATA INFILE" actually isn't supported at all, but "LOAD DATA LOCAL INFILE" can be used from clients that have access to the local files.

Regarding the Cloud SQL latency, are you using a 1st gen or 2nd gen instance?