gcsfuse mounts a Google Storage Bucket successfully, but the contents aren't there!

[James Lampert]

I have a Google Storage Bucket. For argument's sake, I'll call it "foo-test" (not the real bucket name).

I have another bucket. Call it "foo-01" (again, not the real name).

On the advice of a Google engineer on StackOverflow, I copied the contents of "foo-test" into "foo-01" with

gsutil cp -r gs://foo-test/* gs://foo-01

It worked. Or so I thought. If I go to the Google Storage Browser, the contents of the old bucket are present in the new one.

But my Google Compute instances can't see any of it!

When I mount the old bucket with

gcsfuse -o allow_other -file-mode=777 -dir-mode=777 foo-test /var/foo

and then I do

ls /var/foo

I get the contents of the bucket.

But when I mount the new bucket with

gcsfuse -o allow_other -file-mode=777 -dir-mode=777 foo-01 /var/foo

and then I do

ls /var/foo

I get nothing at all!

[James Lampert]

I ran a couple of further tests (one suggested by somebody else with Google, on Stack Overflow).

First, I did the "gcsfuse" manually, with the --debug_gcs flag. Then, I tried sticking something into the bucket FROM THE INSTANCE. Here is the result (real names redacted):

foo@bar-vm:/# stat -fc%t:%T "/var/foo"

ef53:ext2/ext3

foo@bar-vm:/# gcsfuse --debug_gcs -o allow_other -file-mode=777 -dir-mode=777 foo-01 /var/foo

Using mount point: /var/foo

Opening GCS connection...

Opening bucket...

Mounting file system...

File system has been successfully mounted.

foo@bar-vm:/# ls /var/foo

foo@bar-vm:/# edtf /var/foo/foo31july2018.txt

bash: edtf: command not found

foo@bar-vm:/# nano /var/foo/foo31july2018.txt

foo@bar-vm:/# ls /var/foo

foo31july2018.txt

foo@bar-vm:/# 

Notice that the bucket DID mount, and I WAS able to put a file into it. But that file is the ONLY thing that shows up when looking at the bucket from the instance. Contrast this with what I see from the Storage Browser (sensitive information blurred out):

[James Lampert]

The problem was with Access Control Lists (ACLs).

The "gsutil cp" command wasn't preserving them, even with a "-p" option, that is supposed to preserve the ACLs of the objects being copied.

What DID work was mounting both buckets in an instance, and then using the instance command line to copy the contents of the buckets, i.e.,

cp -r -p /var/foo-test/* /var/foo

(-r for recursive, -p for preserve mode, ownerships, timestamps.)

Not very fast, but thankfully, there wasn't all that much to copy.

[Navi Aujla (Google Cloud Support)]

Hello James, 

I ran a quick test and was not able to replicate the behavior you had experience with "gsutil cp". Here are the steps I took after log in to the GCE instance as "root" user:

1. Mounted bucket-1 on GCE VM instance on mount point "/bucket1"
2. Mounted bucket-2 on same GCE VM instance on mount point "/bucket2" 
3. Ran the following command to copy object from bucket-1 to bucket-2

gsutil cp -r gs://bucket-1/beach.jpeg gs://bucket-2
Copying gs://bucket-1/beach.jpeg [Content-Type=image/jpeg]... 
/ [1 files][ 2.1 MiB/ 2.1 MiB] 

 

gsutil cp -r gs://bucket-1/1.PNG gs://bucket-2
Copying gs://bucket-1/1.PNG [Content-Type=image/png]...
/ [1 files][ 29.6 KiB/ 29.6 KiB] 

 

Change directory, to the mount point for bucket-2
cd /bucket2 
ls -latr 
total 2179 
-rw-r--r--. 1 root root 30263 Aug 2 19:23 1.PNG 
-rw-r--r--. 1 root root 2199565 Aug 2 19:32 beach.jpeg 

 

I was able to list the copied files from bucket-1 to bucket-2 on the GCE VM instance.