Can't write any file into bucket only read after gcfuse

[Manuel Perrotta]

I deployed an instance of Wowza Streaming Engine on Google Cloud thank Made a bucket in Google Cloud Storage and mounted it all with GCFUSE. My bucket connected with success and I can see in it and list but I can't write any file with gsutil nor with any FTP even with RSA Key. I also tried gcsfuse -o allow_other but nothing change. What am I doing wrong please help

[Justin Reiners]

You might need storage admin or at least storage writer added to your json key in IAM. Only the user mounting gcsfuse can write as well. I have have my Apache user mount the share. That solved my access issues.

On Sat, Jun 9, 2018, 10:51 AM Manuel Perrotta <manuelp...@gmail.com> wrote:

I deployed an instance of Wowza Streaming Engine on Google Cloud thank Made a bucket in Google Cloud Storage and mounted it all with GCFUSE. My bucket connected with success and I can see in it and list but I can't write any file with gsutil nor with any FTP even with RSA Key. I also tried gcsfuse -o allow_other but nothing change. What am I doing wrong please help

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/55f679d7-7fa7-400b-9875-059a880c453b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Navi Aujla (Google Cloud Support)]

Hello Manuel 

In addition to Justin's response. 

It is possible that the GCE instance is not running with scope "storage-full" configured. For example: If you have created GCE instance with default Cloud API access scopes, it set the GCE instance storage access scope to read only. In that case, you can change the access scope of the instance for Storage to "Full" using the instructions provided here. 

However, there are other methods of setting up credentials for Cloud Storage FUSE and using those, you can write to the Cloud storage bucket provided the service-account or user have appropriate cloud storage role to write to the bucket. For more information, check this documentation link  which included information on setting up credentials for the Cloud Storage FUSE. 

I hope it helps. 

On Saturday, June 9, 2018 at 1:23:31 PM UTC-4, Justin Reiners wrote:

You might need storage admin or at least storage writer added to your json key in IAM. Only the user mounting gcsfuse can write as well. I have have my Apache user mount the share. That solved my access issues.

On Sat, Jun 9, 2018, 10:51 AM Manuel Perrotta <manuelp...@gmail.com> wrote:

I deployed an instance of Wowza Streaming Engine on Google Cloud thank Made a bucket in Google Cloud Storage and mounted it all with GCFUSE. My bucket connected with success and I can see in it and list but I can't write any file with gsutil nor with any FTP even with RSA Key. I also tried gcsfuse -o allow_other but nothing change. What am I doing wrong please help

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.

[Manuel Perrotta]

I made an user with rsa key and placed in the VM Instance as image attached is it not enough? Using that use I can login and see the bucket also with ftp. If I access the bucket through the Goggle Console Panel and I Upload through the web interface it works. The problem is ony when I try to upload through gsutil or through FTP. I have undreds of viedo files every day and doing it manually through the web interface on Google Cloud Platform web console is not an option I need direct access via FTP and command line. 

Here on the web console It works

[Manuel Perrotta]

I can write under my own directory cut I cannot access the directory where is the content stored by default

On Saturday, 9 June 2018 19:57:00 UTC+2, Navi Aujla (Google Cloud Support) wrote:

Hello Manuel 

In addition to Justin's response. 

It is possible that the GCE instance is not running with scope "storage-full" configured. For example: If you have created GCE instance with default Cloud API access scopes, it set the GCE instance storage access scope to read only. In that case, you can change the access scope of the instance for Storage to "Full" using the instructions provided here. 

However, there are other methods of setting up credentials for Cloud Storage FUSE and using those, you can write to the Cloud storage bucket provided the service-account or user have appropriate cloud storage role to write to the bucket. For more information, check this documentation link  which included information on setting up credentials for the Cloud Storage FUSE. 

I hope it helps. 

On Saturday, June 9, 2018 at 1:23:31 PM UTC-4, Justin Reiners wrote:

You might need storage admin or at least storage writer added to your json key in IAM. Only the user mounting gcsfuse can write as well. I have have my Apache user mount the share. That solved my access issues.

On Sat, Jun 9, 2018, 10:51 AM Manuel Perrotta <manuelp...@gmail.com> wrote:

I deployed an instance of Wowza Streaming Engine on Google Cloud thank Made a bucket in Google Cloud Storage and mounted it all with GCFUSE. My bucket connected with success and I can see in it and list but I can't write any file with gsutil nor with any FTP even with RSA Key. I also tried gcsfuse -o allow_other but nothing change. What am I doing wrong please help

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

[Fady (Google Cloud Platform)]

Hello Manuel,

Before answering your question, I would like to touch upon authentication without much details. When accessing your project, you are authenticated using your email account. This is not enough to create or edit a resource. But since you created the project (probably), you are able to make such changes because your role is defined in IAM as an owner. But what if you want to add your partner to the project to create or edit a resource, but not to access everything else?  You would have to add his “ Google Account” in IAM and assign him that specific role. For example, if you need your partner to have full access to Cloud storage and nothing else. You may give him this role only. Hence, he can edit buckets but can not create instances. For further information about this subject you may check this document.

On the other hand, what if we want to give a resource (instance) access to another resource (bucket) on the platform? This can be achieved through service accounts. Service accounts like users can be authorized to access other resources. So if you need your instance to have full access to the Storage bucket, and elaborating on Justin’s and Navi’s suggestions, you need to either create a service account, give it a Cloud Storage role like “storage admin”, authorize it in the instance (using a key), or alternatively use the default service account which has an editor access (no changes to IAM roles but may have to give the “storage admin” role per this document ). For simplicity, you may choose the latter per point 2-a in this document, but there is a catch. The instance itself at creation time and by default does not give the full access to storage (if you chose the default scopes). Therefore, and per Navi’s suggestion and after checking the screenshots provided, it seems your instance is still using default scopes. In order to change this you need to:

1. SSH ( with private key) to the instance using the exact username you added to the metadata (public key).
2. Run “gcloud auth list” and make sure the default service account is active. If you authorized yourself (your email), revoke your account by using "gcloud auth revoke [your email]” . If the default service account is active, You should have a star next to the default service account.
3. Shutdown the GCE instance
4. In the Console (UI) locate the instance, and edit.
5. Scroll down to “ Cloud API access scopes” and change “default” to “set access for each api” 
6. Change  “storage” from “read only” to “full”. Alternatively you may change the “default scope” to “ Allow full access to all Cloud APIs” (too much permissions for the instance) 
7. Save and start instance  (changing scopes you may check this document)
8. SSH (with private key) using the exact username you added to the metadata (public key).
9. In order to avoid Linux permission issues, remount the bucket using gcsfuse as a sub-directory in your user's home directory [/home/user/somefolder]  (you may delete the old one if you wish) (points 3-6 in this document)
10. Now you can use any SFTP client given that you are using the same user (same private key) you used before.

Using the above, you should be able to copy files through the SFTP client to the mounted folder directly. However, if using gsutil at the same time to copy folders, you may have issues viewing those folders in the mounted folder to the bucket as per this document. Therefore I suggest using one of the methods and not both. Furthermore, using gsutil commands, you do not need to mount a folder via gcsfuse on the instance (you won’t be able to use an SFTP client though). A workaround would be using gsutil commands directly from your local machine given that you are authenticated to the Cloud SDK (gcloud), but that is a different subject. For further information, you may check this document.