Issue 337 in s3fs: FIle permissions 000

[s3...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 337 by david...@qedmf.net: FIle permissions 000
http://code.google.com/p/s3fs/issues/detail?id=337

Detailed description of observed behavior:

To be clear about what I'm doing: I am synching a directory from a source
host (not virtualized) up to an s3 bucket with s3cmd, then mounting that
same bucket on an ec2 instance using s3fs. The IAM user used for the synch
from the non virtualized host is the same used to mount the bucket on the
ec2 instance.


Mounting an s3 bucked with s3fs version 1.68 results in the files and
directores in the mount having perms 000:

d--------- 1 root root 0 Jan 1 1970 debian
---------- 1 root root 231 May 2 19:34 distributions

I am trying to serve the files in this mount under apache (www-data user)
and because of these permissions, apache is unable to read the files.
Mounting with allow_other does not change the behavior. If I manually
chmod/chown the files and directories, apache can then read them all, but
any new files added to the s3 bucket - by another host - show up with perms
000, and apache cannot read them. root is able to read all files in the
mount.

I have tried quite a few troubleshooting/work around steps. Most surprising
(to me) is that even if I use the allow_other, uid, gid, and umask options,
I still have trouble accessing the files on the mount. Even if I set gid
and uid to that of the www-data user and umask to 022, and confirm by
looking at the disk that the ownership and permissions appear open for the
www-data user, I still get issues reading the files:

$ id
uid=33(www-data) gid=33(www-data) groups=33(www-data)
$ ls -l
total 0
drwxr-xr-x 1 www-data www-data 0 Jan 1 1970 debian
drwxr-xr-x 1 www-data www-data 0 Jan 1 1970 mvn
drwxr-xr-x 1 www-data www-data 0 Jan 1 1970 rpm
$ ls -l debian/
ls: cannot open directory debian/: Operation not permitted

I have also tried mounting the bucket as the www-data user (i.e. not as the
root user), and I still get errors like "Operation not permitted" trying to
access the files on the mount.

I am mounting the bucket with an IAM user who has permissions to Get* List*
Delete* Put* on the bucket and the bucket/*, and has ListAllMyBuckets as
well. This same user is able to use s3cmd to synch from a local drive on a
source machine to the s3 bucket. Here's the policy with the bucket name
redacted:

{
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": [
"s3:List*",
"s3:Delete*",
"s3:Put*",
"s3:Get*"
],
"Resource": "arn:aws:s3:::${bucket}",
"Resource": "arn:aws:s3:::${bucket}/*"
}
]
}

I have also tried previous versions of s3fs from 1.60 forward. With
1.60-1.62 the mount happens, but the only files I can see are those in the
top level of the bucket. And by files, I mean not directories. Top level
directories are simply not shown in the mounted fs. With 1.63-1.67 result
in i/o errors when trying to read the mount, even as root, and question
marks in place of dashes for the permissions bits.

So, to reiterate, the only thing that seems to enable the www-data user to
read the files on the mount is to mount as root, with or without the
allow_other option, then manually chmod/chown the files and directories in
the mount. But then, new files synched up to the bucket show up with perms
000.

What steps will reproduce the problem - please be very specific and
detailed. (if the developers cannot reproduce the issue, then it is
unlikely a fix will be found)?

Use s3cmd to synch a directory on a non-virtualized host to an s3 bucket.
Mount the same s3 bucket on an ec2 instance using s3fs. Look at the file
system permissions on the mouted s3 bucket.

===================================================================
The following information is very important in order to help us to help
you. Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version):

Amazon Simple Storage Service File System 1.68

Version of fuse being used (pkg-config --modversion fuse):

2.8.6

System information (uname -a):

Linux www 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux

Distro (cat /etc/issue):

Ubuntu 12.04.2 LTS \n \l

s3fs command line used (if applicable):

/etc/fstab entry (if applicable):

(bucket name redacted)

s3fs#${bucket} /var/www/mnt fuse allow_other,url=https://s3.amazonaws.com 0
0

s3fs syslog messages (grep s3fs /var/log/syslog):

many of these:

May 2 20:01:40 www s3fs: init $Rev: 414 $


--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[s3...@googlecode.com]

Comment #1 on issue 337 by LaGho...@gmail.com: FIle permissions 000
http://code.google.com/p/s3fs/issues/detail?id=337

I have the same issue. I was using version 1.61, which work fine for me
(all files on the root of the bucket), but 1.68 give me "Permission denied"

For now I will keep using 1.61

[s3...@googlecode.com]

Comment #2 on issue 337 by djakelam...@gmail.com: FIle permissions 000
http://code.google.com/p/s3fs/issues/detail?id=337

I have the same issue. The folder had 777 permissions before mount. My
bucket successfully mounted using user and group as set in my fstab file.
However, the permissions for existing files are -------------- . I can
chmod and the issue is fixed, but it is part of a bootstrap process and I
would like to have the permissions set properly when the mounting initially
occurs.

[s3...@googlecode.com]

Comment #3 on issue 337 by ggta...@gmail.com: FIle permissions 000
http://code.google.com/p/s3fs/issues/detail?id=337

Hi, all

I think that this issue is two issue.
1) the object which is uploaded by other S3 client has no s3fs's meta
header, then these objects are displayed permission 000.
2) uid/gid mount option is not supported by s3fs yet.

For sub issue 1), you can change permission(owner/group/mode) by root user,
after that the object has s3fs's meta information.
Otherwise you probably use uid/gid mount option as davidoff did it.
But it does not work because of sub issue 2).

I chckeed in new codes as Revision 424, this rev supports uid/gid option
and it seems that it works good.
And I'll update s3fs version soon.

Please use it and check it.

Last,
Hi, djakelambert

It seems your problem is not either sub issue 1) or 2).
If your problem is not same as this issue, please post new issue.

Regards,

[s3...@googlecode.com]

Updates:
Status: Fixed

Comment #4 on issue 337 by ggta...@gmail.com: FIle permissions 000
http://code.google.com/p/s3fs/issues/detail?id=337

Hi, all

I uploaded new version v1.69.
This version solved this issue by that s3fs supported uid/gid mount option.

Please use new version.
If you find another bug, please post new issue.

Thanks,

[s3...@googlecode.com]

Comment #5 on issue 337 by Relay3...@gmail.com: FIle permissions 000
http://code.google.com/p/s3fs/issues/detail?id=337

I am having the same issue on 1.73

[Takeshi Nakatani]

Hi,

Please let us know which version do you use and more infomraion about your problem.

Thanks in advance.

[s3...@googlecode.com]

Comment #6 on issue 337 by rpar...@flyinghippo.com: FIle permissions 000
https://code.google.com/p/s3fs/issues/detail?id=337

I've tried version 1.77 and 1.78 and still get this same error.

s3cmd sync --acl-public . s3://... to sync files to s3

s3fs uid gid allow_other to mount.

All permissions show ------ (i.e. 000)

Once I mount, I AM able to cd into the directories and perform chmod, but
Apache does not serve files because it thinks that it can't access them.

Is there any real solution?

[s3...@googlecode.com]

Comment #7 on issue 337 by ggta...@gmail.com: FIle permissions 000
https://code.google.com/p/s3fs/issues/detail?id=337

HI, rparlee

If you can, please check the object headers, s3fs uses original header for
permission.(x-amz-meta-***)
I think your object does not any header(x-amz-meta-***), but gid/uid is
specified option, so these attributes is instead of headers.
But permission header could not be specified.
For solving it, you set permission header(x-amz-meta-mode) to each object.
(for example, It is made by command chmod/touch/etc..., please update the
object any attribute)

Regards,

[s3...@googlecode.com]

Comment #8 on issue 337 by anth...@anyperk.com: FIle permissions 000
https://code.google.com/p/s3fs/issues/detail?id=337

I'm having the same issue.

I'm installing s3fs and fuse via a chef recipe
(https://github.com/twilson63/s3fs-recipe) which installs s3fs 1.69 and
fuse 2.8.7.

This is my /etc/fstab line:
s3fs#<bucket> /mnt/<bucket> fuse allow_other,use_cache=/tmp 0 0

I'm mounting an s3 bucket across multiple servers, and that part works
great, permissions are set properly.

However, when uploading a file to S3 using the AWS S3 management tool (the
web interface) then the file shows up with no permissions. Obviously, S3
doesn't set any of s3fs metadata fields.

Is there a way for s3fs/fuse to set metadata when there aren't any?

[s3...@googlecode.com]

Comment #9 on issue 337 by ggta...@gmail.com: FIle permissions 000
https://code.google.com/p/s3fs/issues/detail?id=337

Hi, anthony

I think that S3 we console can upload objects with Metadata.
Then you can specify "x-amz-meta-***" key and its values.

Regards,